fix(provider/google): include usageMetadata in providerMetadata for non-streaming generate#16460
Open
R-Taneja wants to merge 1 commit into
Open
fix(provider/google): include usageMetadata in providerMetadata for non-streaming generate#16460R-Taneja wants to merge 1 commit into
R-Taneja wants to merge 1 commit into
Conversation
…on-streaming generate doStream attaches usageMetadata (incl. promptTokensDetails) to providerMetadata.google, and v6/v7 do so in both generate and stream, but v5's doGenerate omitted it. This left non-streaming consumers without the per-modality token breakdown. Mirror the streaming path so generate is consistent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TODO:
Background
doStreamattachesusageMetadata(includingpromptTokensDetails) toproviderMetadata.google, and v6/v7 do so in bothgenerateandstream. But on the v5 line,doGenerateomitted it — so non-streaming consumers got no per-modality token breakdown inproviderMetadata.This matters because
LanguageModelV2Usagehas norawfield (unlike v3/v4), soproviderMetadata.google.usageMetadatais the only place a consumer can recoverpromptTokensDetailsfor a non-streaming v5 request. The result was that per-modality usage (e.g. audio vs text input tokens, which Gemini prices differently) was unavailable forgenerateTextcalls.Change
Mirror the streaming path: add
usageMetadata: usageMetadata ?? nulltodoGenerate'sproviderMetadata.googleblock. TheusageMetadatavariable is already in scope. This brings v5doGeneratein line with its owndoStreamand with v6/v7.Testing
doGenerateexposesusageMetadata(incl.promptTokensDetails) inproviderMetadata.google-generative-ai-language-model.test.tspasses (100 tests); no snapshot changes (existing tests assert specificproviderMetadata.googlefields rather than the whole object).