[opentelemetry-instrumentation-genai-langchain] Record streaming telemetry for the LangChain instrumentation - #482
Open
sfc-gh-zeningchen wants to merge 3 commits into
Conversation
A streamed call was indistinguishable from a synchronous one. The callback-based handler never routes chunks through util-genai's stream wrappers, so gen_ai.request.stream and gen_ai.response.time_to_first_chunk were never set and the gen_ai.client.operation.time_to_first_chunk and time_per_output_chunk histograms never recorded a point. The wrappers own that bookkeeping for instrumentations that receive an SDK stream to proxy, which a callback handler never does. Add GenAIInvocation.record_stream_chunk() as the public entry point for reporting chunk arrival without a stream, and implement on_llm_new_token to call it once per output chunk. Streaming is inferred from a token arriving rather than from the stream invocation param, which LangChain derives from the model's streaming field and leaves False for a plain .stream() call. LangChain's contentless end-of-stream marker is skipped so it does not contribute a spurious near-zero inter-chunk gap.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds support for streaming timing telemetry to the callback-based LangChain instrumentation by introducing a util-level API (GenAIInvocation.record_stream_chunk()) that can be invoked from token callbacks, enabling gen_ai.request.stream, gen_ai.response.time_to_first_chunk, and the streaming timing histograms to be emitted for LangChain .stream() / .astream().
Changes:
- Add
GenAIInvocation.record_stream_chunk()inopentelemetry-util-genaito record per-chunk arrival timing without wrapping an SDK stream. - Implement
on_llm_new_tokenin the LangChain callback handler to callrecord_stream_chunk()for each real output chunk (skipping LangChain’s end-of-stream marker). - Add unit tests in both util-genai and langchain packages to validate streamed vs non-streamed behavior and marker handling.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| util/opentelemetry-util-genai/src/opentelemetry/util/genai/_invocation.py | Adds record_stream_chunk() to mark streaming and record per-chunk timing. |
| util/opentelemetry-util-genai/tests/test_handler_metrics.py | Adds direct tests for record_stream_chunk() and its metric/span effects. |
| util/opentelemetry-util-genai/.changelog/461.added | Towncrier fragment documenting the new util entry point. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/utils.py | Adds is_stream_end_marker() helper for filtering LangChain’s contentless terminal marker. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/callback_handler.py | Implements on_llm_new_token to report streaming chunk arrivals. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/test_streaming.py | Adds streaming/non-streaming tests and an end-of-stream marker regression test. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/.changelog/461.fixed | Towncrier fragment documenting the LangChain streaming telemetry fix. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Pull request dashboard statusWaiting on reviewers · refreshed 2026-08-26 00:18 UTC Review the latest changes. Status above doesn't look right?
|
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Streamed LangChain calls never set
gen_ai.request.streamorgen_ai.response.time_to_first_chunk, and never recorded thetime_to_first_chunk/time_per_output_chunkhistograms added in #269,because the shared stream wrappers are the only producers of those values and a
callback-based instrumentation has no stream to wrap. This adds
GenAIInvocation.record_stream_chunk()toopentelemetry-util-genaias theentry point for reporting chunk arrival without a stream, and implements
on_llm_new_tokento call it once per output chunk. The capability lives in theutil rather than the instrumentation, per the direction on #8.
Fixes #481
Type of change
How has this been tested?
New
tests/test_streaming.pyin the langchain package covers.stream(),.astream(), a non-streamed.invoke()asserting no stream attributes ormetric points appear, and a parametrized case separating LangChain's
end-of-stream marker from a genuinely content-free chunk. These patch the
model's
_stream/_astreamrather than replaying a cassette, because theassertions are about chunk timing, which a cassette does not reproduce. Two
tests in
test_handler_metrics.pycoverrecord_stream_chunk()directly.util-genaisuite: 332 passedlangchainsuite: 239 passedlangchain==0.3.21,langchain-core0.3.86, wherechunk_positiondoes not exist) and atlatest (
langchain==1.3.16,langchain-core1.6.0)tox -e precommit(ruff, ruff-format) andpyrightcleanChecklist