New Feature Request : Incorporation of Token Count Feature for Gemini

[arunrajes]

Hi Team,

Request you to incorporate the token count feature inside gemini.R as it will help users to get a view of tokens (both input and output) for cost budgeting (informed decision making)

Documentation Reference: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/get-token-count

Thanks
Arun

[hadley]

You mean before you do the request? I can't quite imagine how that would fit into the existing ellmer API.

[arunrajes]

Hi @hadley ,

Assume we have prompt and then some data (may be a pdf or an image file) which are passed as inputs to Gemini. As an end user, we would be interested to know how much input tokens are being consumed . This helps us in optimizing the prompt and the type of input which we can pass (obviously having control over the cost ). The Output token in gemini (max can be 8124 and configurable).

This is very similar to computing token cost
#203 (comment)

Sharing you python sample code from gemini documentation (vertex ai)

from google import genai
from google.genai.types import HttpOptions, Part

client = genai.Client(http_options=HttpOptions(api_version="v1"))

contents = [
Part.from_uri(
file_uri="gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
mime_type="video/mp4",
),
"Provide a description of the video.",
]

response = client.models.count_tokens(
model="gemini-2.0-flash-001",
contents=contents,
)
print(response)

Example output:
total_tokens=16252 cached_content_token_count=None

Thanks
Arun

[hadley]

I understand how this could be done, but it's very far from the existing ellmer interface and I don't think many other proividers provide a way to determine the number of tokens apart from actually doing the request.

[cpsievert]

Anthropic also supports it https://docs.anthropic.com/en/docs/build-with-claude/token-counting

Last time I looked, OpenAI didn't appear to have API for this, but in Python there is {tiktoken}, which at least gives a rough estimate of the input tokens for a given model.

[arunrajes]

Hi @cpsievert ,

Running this code shall give the count of input and output tokens
chat_CoT_ellmer$tokens()
input output
[1,] 0 0
[2,] 31771 474

Thanks
Arun

[hadley]

I think for now this is out of scope for ellmer — there's no obvious way to add it to the API and there's no API for OpenAI. We might revisit this in the future, but we won't plan to work on it in the short term.