[opentelemetry-instrumentation-genai-smolagents] instrument agent runs - #403
Conversation
a143831 to
b34031c
Compare
Pull request dashboard statusWaiting on the author · refreshed 2026-08-25 20:50 UTC Respond to 1 review item (e.g. link a commit, explain why not, ask a follow-up):
Status above doesn't look right?
|
e274572 to
3be1988
Compare
3be1988 to
b708def
Compare
There was a problem hiding this comment.
Pull request overview
This PR extends the opentelemetry-instrumentation-genai-smolagents package to instrument smolagents.MultiStepAgent.run() as a GenAI invoke_agent operation, including support for both streaming and non-streaming runs and recording the GenAI client operation duration metric. It also adds coverage via unit tests, conformance scenarios, and VCR-backed fixtures, and updates package documentation/changelog accordingly.
Changes:
- Add
invoke_agentspan + duration metric instrumentation forMultiStepAgent.run()(streaming + non-streaming) via new wrapper logic inpatch.py. - Add comprehensive tests for agent runs (stream lifecycle, error paths, managed agents, content-capture modes) plus a new conformance
AgentScenarioand VCR cassette/config. - Update smolagents package README and add a towncrier fragment documenting the new
invoke_agentinstrumentation.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_utils.py | Adds agent/model/tool fakes plus a shared span lifecycle recorder used by agent-run tests. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_models.py | Removes the local lifecycle recorder (moved to shared test utilities/fixtures). |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_instrumentor.py | Expands instrument/uninstrument and rollback tests to include MultiStepAgent.run patching and third-party wrapper interaction. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_conformance.py | Registers the new AgentScenario in the conformance test runner. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_agents.py | New test module covering invoke_agent spans, streaming behavior, content capture, error scenarios, and metrics. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/requirements.oldest.txt | Adds a test-only OpenAI dependency for VCR-backed agent tests in the oldest env. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/requirements.latest.txt | Switches to smolagents[openai] for latest-env agent/VCR coverage. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/conftest.py | Enables VCR plugin, adds VCR scrubbing config, and provides a shared lifecycle fixture plus an additional content-capture fixture. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/conformance/agent.py | Adds a conformance scenario validating invoke_agent span + duration metric and presence of gen_ai.agent.name. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/cassettes/agent_with_image.yaml | Adds a VCR cassette for an agent run that includes an image input. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/patch.py | Implements MultiStepAgent.run wrapping (invoke_agent) and supporting recording utilities/stream wrapper. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/_messages.py | Adds conversions specific to agent-run tasks/final answers and supports managed-agent definitions as tools. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/init.py | Wires agent-run wrapper into the instrumentor’s patch/unpatch lifecycle. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/README.rst | Updates docs to reflect new agent-run span coverage and adjusts known-gaps list. |
| instrumentation/opentelemetry-instrumentation-genai-smolagents/.changelog/403.added | Documents the new invoke_agent span support for MultiStepAgent.run(). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| self._finish_once(error) | ||
| raise | ||
|
|
||
| def close(self) -> None: |
There was a problem hiding this comment.
A run(stream=True) generator that isn't drained never finishes the span, and its context stays attached - so every later span on that thread nests under the leaked invoke_agent. GC closes the underlying generator, but nothing calls the wrapper's close().
breaking out of for step in agent.run(task, stream=True): is a normal way to use this API, so this is easy to hit. Both of these fail on this branch:
def _first_step_only(agent: CodeAgent) -> None:
for _ in agent.run("Test question", stream=True):
break
def test_abandoned_streaming_run_finalizes_the_span(
instrument_with_content, span_exporter
) -> None:
agent = CodeAgent(tools=[], model=FakeCodeModel(), max_steps=3)
_first_step_only(agent)
gc.collect()
assert (
len(
spans_by_operation(
span_exporter.get_finished_spans(), "invoke_agent"
)
)
== 1
)
def test_abandoned_streaming_run_does_not_leak_context(
instrument_with_content, span_exporter, tracer_provider
) -> None:
agent = CodeAgent(tools=[], model=FakeCodeModel(), max_steps=3)
_first_step_only(agent)
gc.collect()
with tracer_provider.get_tracer("test").start_as_current_span("unrelated"):
pass
(unrelated,) = [
span
for span in span_exporter.get_finished_spans()
if span.name == "unrelated"
]
assert unrelated.parent is NoneE AssertionError: assert 0 == 1
E + where 0 = len([])
E + where [] = spans_by_operation((), 'invoke_agent')
E assert SpanContext(trace_id=0xba5e..., span_id=0x68a7..., ...) is None
E + where SpanContext(...) = <ReadableSpan 'unrelated'>.parent
The context leak is the worse half: everything the process traces afterwards is silently reparented under a span that never ends. _ModelStreamWrapper has the same hole, but a partially consumed run generator is much more reachable. Finalizing on __del__ here (or in SyncStreamWrapper) would close it.
| steps = agent.memory.steps | ||
| last_step = steps[-1] if steps else None | ||
| finish_reason = ( | ||
| "length" |
There was a problem hiding this comment.
it's not technically length, but AgentMaxStepsError, wdyt about using "max_steps" as the constant?
| finish_reason = ( | ||
| "length" | ||
| if isinstance(last_step, ActionStep) | ||
| and isinstance(last_step.error, AgentMaxStepsError) |
There was a problem hiding this comment.
if last_step.error, but of a different type we should report "error"
| try: | ||
| yield | ||
| except Exception: # pylint: disable=broad-except | ||
| _logger.debug("Failed to record %s", what, exc_info=True) |
There was a problem hiding this comment.
why is this needed? we should not throw from instrumentation code and should not try to catch exceptions from instrumentation code. None of the instrumentations do
Description
Adds invoke_agent instrumentation and the duration metric for smolagents
MultiStepAgent.run, streaming and non-streaming. The span records the agent name and description, the model, tool and managed-agent definitions, the finish reason, and the task and final answer when content capture is enabled.Known gaps:
CodeAgentwith the defaultexecutor_type="local"writes code and runs it in a worker thread. OpenTelemetry context isn't copied into the thread, so a managed agent called from that code cannot see the parent span.Part of #141
Type of change
How has this been tested?
Checklist
See CONTRIBUTING.md
for the style guide, changelog guidance, and more.