fix(vertexai): include thinking tokens in ChatVertexAI output_tokens

[kurniawan (codechrl)]

Why

_get_usage_metadata_gemini (libs/vertexai/langchain_google_vertexai/chat_models.py) sets output_tokens from candidates_token_count alone:

output_tokens = raw_metadata.get("candidates_token_count", 0)
...
thought_tokens = raw_metadata.get("thoughts_token_count", 0)
...
if thought_tokens > 0:
    return UsageMetadata(
        input_tokens=input_tokens,
        output_tokens=output_tokens,       # thinking tokens not included
        total_tokens=total_tokens,         # but total_tokens does include them
        ...
    )

total_token_count from the raw Gemini API response already includes thinking tokens, so for any thinking-enabled Gemini call (gemini-2.5-flash/-pro) through ChatVertexAI, input_tokens + output_tokens != total_tokens — breaking the UsageMetadata contract — and anything pricing off output_tokens undercounts completion cost.

The sibling ChatGoogleGenerativeAI._response_to_result (libs/genai/langchain_google_genai/chat_models.py) in this same repo already handles this correctly:

output_tokens = (
    response.usage_metadata.candidates_token_count or 0
) + thought_tokens

What

Fold thought_tokens into output_tokens in _get_usage_metadata_gemini, mirroring the working genai implementation. output_token_details={"reasoning": ...} is unchanged — it already correctly reports the reasoning subset.

Tests

Added test_get_usage_metadata_gemini_includes_thought_tokens_in_output_tokens to tests/unit_tests/test_chat_models.py, asserting output_tokens folds in thinking tokens and input_tokens + output_tokens == total_tokens. Confirmed test-first: fails on main (5 == 25), passes with the fix. Full test_chat_models.py: 116 passed. ruff check/ruff format --check clean.

Areas worth a careful look

• _get_usage_metadata_gemini is used by both the non-streaming (_gemini_response_to_chat_result) and streaming (_gemini_chunk_to_generation_chunk) paths, so this fixes both at once.
• The non-thinking branch (thought_tokens == 0) is untouched — output_tokens there was already correct.

---

Disclaimer: this PR was prepared with the assistance of an AI agent (Claude Code). All code and test changes were reviewed by the author before submission.