From 33e142cadc4885f8b29124f72ac6691ea34eefb2 Mon Sep 17 00:00:00 2001 From: Mahmoud Mabrouk Date: Sun, 12 Jul 2026 18:18:49 +0200 Subject: [PATCH] docs(agent-workflows): plan usage and cost telemetry --- .../projects/agent-usage-telemetry/README.md | 32 ++ .../projects/agent-usage-telemetry/context.md | 71 +++++ .../agent-usage-telemetry/interface-design.md | 278 ++++++++++++++++++ .../projects/agent-usage-telemetry/plan.md | 132 +++++++++ .../projects/agent-usage-telemetry/qa.md | 83 ++++++ .../agent-usage-telemetry/research.md | 180 ++++++++++++ .../projects/agent-usage-telemetry/status.md | 53 ++++ .../agent-usage-telemetry/trace-inventory.md | 84 ++++++ 8 files changed, 913 insertions(+) create mode 100644 docs/design/agent-workflows/projects/agent-usage-telemetry/README.md create mode 100644 docs/design/agent-workflows/projects/agent-usage-telemetry/context.md create mode 100644 docs/design/agent-workflows/projects/agent-usage-telemetry/interface-design.md create mode 100644 docs/design/agent-workflows/projects/agent-usage-telemetry/plan.md create mode 100644 docs/design/agent-workflows/projects/agent-usage-telemetry/qa.md create mode 100644 docs/design/agent-workflows/projects/agent-usage-telemetry/research.md create mode 100644 docs/design/agent-workflows/projects/agent-usage-telemetry/status.md create mode 100644 docs/design/agent-workflows/projects/agent-usage-telemetry/trace-inventory.md diff --git a/docs/design/agent-workflows/projects/agent-usage-telemetry/README.md b/docs/design/agent-workflows/projects/agent-usage-telemetry/README.md new file mode 100644 index 0000000000..862402a2ad --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-usage-telemetry/README.md @@ -0,0 +1,32 @@ +# Agent usage telemetry + +This project defines how agent harnesses report token usage, context utilization, and monetary +cost from the harness boundary through the runner and Python service into Agenta tracing. + +The immediate symptom is inaccurate cost in traces for Pi and Claude. The underlying problem is +broader: the current four-number usage object conflates billable token usage, ACP context-window +occupancy, and cost from sources with different authority. It also loses cache detail and places +the same totals on several span levels without an attribution rule. + +The proposal starts with the pre-production runner and runner-to-service contracts. It also +documents a service-to-API semantic-convention proposal for review. That later boundary requires +CTO approval before implementation. + +## Files + +- [context.md](context.md): problem, scope, goals, and constraints. +- [research.md](research.md): current path, information-loss points, existing API behavior, and + broader tracing gaps. +- [trace-inventory.md](trace-inventory.md): end-to-end inventory of runner trace fields, adapter + handling, known gaps, and this project's scope. +- [interface-design.md](interface-design.md): proposed canonical usage model, source mappings, + aggregation rules, and semantic-convention proposal. +- [plan.md](plan.md): phased implementation and review sequence. +- [qa.md](qa.md): contract, trace, API, and live verification matrix. +- [status.md](status.md): current state, decisions, approvals, and blockers. + +## Recommended reading order + +Read `context.md`, then `research.md`, `trace-inventory.md`, `interface-design.md`, and `plan.md`. Reviewers deciding +the service-to-API boundary should also read the semantic-convention section in +`interface-design.md` and the approval gate in `status.md`. diff --git a/docs/design/agent-workflows/projects/agent-usage-telemetry/context.md b/docs/design/agent-workflows/projects/agent-usage-telemetry/context.md new file mode 100644 index 0000000000..70b14f70ff --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-usage-telemetry/context.md @@ -0,0 +1,71 @@ +# Context + +## Problem + +Agent traces show token totals and monetary cost for Pi and Claude runs, but those numbers do not +have one stable meaning today. + +The runner reduces every harness to: + +```json +{"input": 10, "output": 5, "total": 15, "cost": 0.001} +``` + +That shape cannot say whether input includes cached tokens, whether cost came from the provider or +a local model catalog, which currency applies, or whether an ACP number describes billed usage or +current context occupancy. Pi and Claude both expose more information than this contract retains. + +Tracing adds a second problem. The runner places usage on leaf LLM spans and repeats run totals on +agent and workflow spans to bridge separately exported OTLP batches. Agenta ingestion treats +values as incremental and rolls children into parents, so repeated totals can be counted more than +once. The API also recalculates supported-model costs with LiteLLM even when a producer supplied a +cost. + +## Boundary constraints + +- The harness-to-runner and runner-to-Python-service interfaces are pre-production. We can change + them while defining the correct model. +- The service-to-API tracing boundary is more stable. A semantic-convention or ingestion change + needs CTO approval. +- Existing trace storage is JSON-path based and can retain additional numeric attributes. The + hard part is agreeing on meaning, normalization, rollup, and compatibility. +- A tracing failure must not fail an agent run. Missing telemetry must remain distinguishable from + reported zero. + +## Goals + +1. Preserve the usage and cost facts Pi, Claude, and future harnesses actually report. +2. Separate billed token usage, monetary cost, and context-window utilization by semantic role. +3. Define inclusive input/output totals and cache/reasoning subcategories consistently with the + supported OpenTelemetry GenAI conventions. +4. Preserve cost currency, provenance, and reported-versus-estimated status. +5. Define field-level reconciliation when final responses, stream updates, and Pi writeback each + provide part of the result. +6. Attribute incremental usage exactly once in a trace and derive parent cumulative totals without + double counting. +7. Propose an Agenta cost semantic convention, cache-aware fallback behavior, UI implications, and + documentation updates for CTO review. +8. Cover partial, cancelled, resumed, multi-turn, and cost-only runs instead of only successful + single-turn requests. + +## Non-goals + +- Reconstruct provider invoices when the provider or harness reports no usage. +- Pretend ACP exposes per-model-call detail that it does not expose. +- Add a new pricing service in the runner. +- Change the service-to-API contract before approval. +- Redesign the observability UI in the first implementation phase. + +## Success criteria + +- Cache-heavy Pi and Claude fixtures retain inclusive input, cache-read, cache-creation, output, + total, and cost provenance through the runner-to-service result. +- ACP `used` and `size` appear only as context-utilization fields. +- Streamed provisional context never overrides final billed usage. +- Every monetary value carries a currency and source, including zero. +- A trace attributes billable usage to one level only and produces the expected cumulative total. +- The API proposal states when reported cost wins, when estimation is allowed, and how every token + and cost bucket rolls up. +- Public semantic-convention and cost-tracking documentation is updated in the implementation that + changes the stable tracing boundary. + diff --git a/docs/design/agent-workflows/projects/agent-usage-telemetry/interface-design.md b/docs/design/agent-workflows/projects/agent-usage-telemetry/interface-design.md new file mode 100644 index 0000000000..bda99f19f2 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-usage-telemetry/interface-design.md @@ -0,0 +1,278 @@ +# Interface design + +## Design rules + +The contract groups fields by semantic role: + +- `usage`: consumed/generated units and monetary charges for this run; +- `context`: a point-in-time runtime gauge, not billing data; +- tracing metadata: how a fact maps onto a span and whether it is incremental or cumulative. + +Missing and zero are different. Every number must be finite and non-negative. Producers preserve +reported totals instead of silently replacing them with arithmetic when a provider's semantics do +not match the canonical invariant. + +## Proposed runner-to-service result + +```jsonc +{ + "usage": { + "tokens": { + "input": 12000, // inclusive: uncached + cacheRead + cacheCreation + "output": 800, // inclusive: includes reasoning when reported that way + "total": 12800, + "details": { + "uncachedInput": 1000, + "cacheReadInput": 10500, + "cacheCreationInput": 500, + "reasoningOutput": 300 + } + }, + "costs": { + "total": { + "amount": 0.021, + "currency": "USD" + }, + "details": { + "uncachedInput": 0.003, + "output": 0.012, + "cacheReadInput": 0.001, + "cacheCreationInput": 0.005 + }, + "provenance": { + "kind": "reported", // reported | calculated | estimated + "source": "provider", // provider | harness | agenta + "sourceName": "anthropic" + } + }, + "status": "final" + }, + "context": { + "usedTokens": 12800, + "windowTokens": 200000, + "source": "acp", + "observedAt": "2026-07-12T12:00:00Z" + } +} +``` + +The final naming should follow the existing wire's camelCase convention. `usage.tokens` and +`usage.costs` are run-level cumulative facts. They do not claim per-LLM-call fidelity. + +### Token invariants + +- `input` follows current OTel GenAI meaning and includes every input token. +- Cache-read and cache-creation counts are subcategories of `input`, not additive siblings after + normalization. +- `output` includes every output token. Reasoning is a subcategory when the provider reports it. +- `total`, when canonical, equals `input + output`. +- `details.uncachedInput + cacheReadInput + cacheCreationInput` should equal `input` when all + three mutually exclusive raw buckets are known. A missing detail stays missing. +- Never calculate a missing cache bucket by subtraction unless the source contract guarantees all + other buckets are complete. + +### Cost invariants + +- Currency is required whenever any monetary amount exists. Version one supports USD but keeps + currency explicit. +- `reported` means a provider or provider SDK supplied the amount. +- `calculated` means the harness calculated it from a model catalog and observed token buckets. +- `estimated` means Agenta calculated it after ingestion as a fallback. +- Zero is a valid reported amount. Absence means unknown. +- A total may exist without component costs. Do not invent a prompt/output split. +- `uncachedInput`, `cacheReadInput`, `cacheCreationInput`, and `output` are mutually exclusive + cost components when all are known. Their sum equals `total`. +- Compatibility `prompt` cost is inclusive input-side cost. It equals uncached input plus cache + read plus cache creation and must not be added to those details again. +- Cost provenance is metadata, not a numeric metric bucket. + +### Context invariants + +- `usedTokens` and `windowTokens` are gauges observed at one time. +- They never populate `usage.tokens.total`. +- A newer context snapshot replaces an older snapshot. Usage snapshots instead follow their + declared run/turn scope and aggregation rules. + +## Harness normalization + +| Source | Canonical mapping | Authority | +|---|---|---| +| Pi message `input`, `cacheRead`, `cacheWrite` | sum to inclusive input; retain exclusive details | final per Pi model call | +| Pi `output`, optional reasoning detail | inclusive output plus detail | final per Pi model call | +| Pi `cost.*` | cost components/total, USD, calculated by harness unless provider reporting is proven | calculated | +| Claude `PromptResponse.usage` | normalize input + cached-read + cached-write, output, and explicit total | final run tokens | +| Claude ACP `usage_update.used/size` | context used/window only | latest gauge | +| Claude ACP `cost.amount/currency` | run cost snapshot | provider-reported via SDK | + +Future adapters must document whether their input/output fields are inclusive before mapping them. + +## Internal observation type + +Normalization operates on observations before producing the aggregate result: + +```jsonc +{ + "scope": "model_call", + "temporality": "delta", + "status": "provisional", + "resource": { + "provider": "anthropic", + "model": "claude-sonnet-4-6", + "responseModel": "claude-sonnet-4-6", + "operation": "chat" + }, + "tokens": { /* normalized token facts */ }, + "costs": { /* amount, currency, provenance */ }, + "context": { /* only for gauge observations */ }, + "observedAt": "2026-07-12T12:00:00Z" +} +``` + +Resource identity is required before Agenta estimates a component. Multi-model runs aggregate +reported monetary values only when currencies match. Estimation remains model-scoped and sums the +components after pricing each model separately. The final run result may expose one aggregate plus +optional `breakdown` entries keyed by provider, response model, operation, and currency. It must +not collapse heterogeneous observations before estimation. + +Cost provenance is attached to each breakdown entry. The aggregate may carry one provenance only +when every component has the same kind and source. Otherwise its provenance is `mixed` and the +breakdown is required. + +Token status and validation quality survive on the final result. If a provider total conflicts +with normalized inclusive input plus output, retain the reported total as `reportedTotal`, emit +the normalized arithmetic total as `total`, and mark `validation.status="mismatch"`. Do not send +the inconsistent provider number as the canonical rollup total. + +## Reconciliation reducer + +Replace whole-object precedence with a field-level reducer: + +1. Validate every observation and retain its source, scope, temporality, and final/provisional + status internally. +2. Use final Pi message/writeback or final Claude prompt response for token usage. +3. Use the latest final provider-reported cost. Fall back to harness-calculated cost. Do not + estimate in the runner. +4. Keep the latest ACP context gauge separately. +5. Accumulate incremental Pi model-call observations once. Do not sum repeated cumulative ACP + snapshots. +6. Preserve partial usage on cancellation and error when the harness reported it. +7. Emit exactly one final run-level usage record. Provisional context uses a distinct event type. + +Recommended neutral events: + +```jsonc +{"type": "context_usage", "usedTokens": 12000, "windowTokens": 200000} +{"type": "usage", "scope": "run", "status": "final", "usage": { /* shape above */ }} +``` + +The Vercel adapter may initially project inclusive input/output/total to its established usage +metadata. It must not project `context_usage` as billed usage. + +## Exact result and event placement + +`AgentRunResult.usage` holds the final run aggregate and optional resource breakdown. A sibling +`AgentRunResult.context` holds the last context gauge for diagnostics. The terminal result always +contains the final usage when known, including on partial/error completion. + +Streaming uses `context_usage` for provisional gauges and at most one `usage` event with +`status=final` for the settled run aggregate. The terminal result repeats the final aggregate as +the authoritative transport result. Stream adapters de-duplicate the event/result pair rather +than choosing whichever arrived first. + +## Trace attribution + +### Incremental owner + +Billable usage belongs exactly once: + +- Pi with per-call instrumentation: each leaf LLM span owns its incremental usage. +- ACP without per-call usage: the synthetic run LLM span owns the run usage and is explicitly a + run-level approximation. +- Agent and workflow parents expose derived cumulative values only. They do not repeat the same + values as incremental metrics. + +Separate OTLP batches currently prevent the API from deriving a parent total in one ingest pass. +The implementation must choose one reviewed bridge: + +1. export a clearly marked cumulative summary on the remote parent; +2. teach ingestion to reconcile later-arriving children before cumulative query results; +3. stop duplicating and accept that an isolated parent batch has no total until trace-level query + aggregation. + +The current approach, repeating `gen_ai.usage.*` on every level, is not acceptable because those +attributes are interpreted as incremental. + +## Service-to-API semantic-convention proposal + +This section is a proposal pending CTO approval. + +### Attribute classification + +The implementation must pin one supported OTel GenAI semantic-convention version and maintain one +authoritative mapping table. Every emitted field is classified as `otel_standard`, +`compatibility_alias`, or `agenta_extension`. + +Input and output usage are current OTel GenAI fields. Cache and reasoning dotted fields are +already supported by Agenta's Logfire adapter and appear in current OTel GenAI registries, but +their exact status must be verified against the pinned version before the design calls them +standard. Until then, treat them as supported compatibility fields: + +```text +gen_ai.usage.input_tokens +gen_ai.usage.output_tokens +gen_ai.usage.cache_read.input_tokens +gen_ai.usage.cache_creation.input_tokens +gen_ai.usage.reasoning.output_tokens +``` + +Keep legacy prompt/completion and total attributes only as documented compatibility aliases at +ingestion. Pin the supported OTel GenAI vocabulary/version in one authoritative mapping table +instead of letting `semconv.py` and individual adapters drift. + +### Agenta cost extension + +OTel currently has no standard GenAI monetary cost attribute. Emit the established Agenta metric +namespace: + +```text +ag.metrics.unit.costs.total +ag.metrics.unit.costs.input +ag.metrics.unit.costs.output +ag.metrics.unit.costs.cache_read_input +ag.metrics.unit.costs.cache_creation_input + +ag.meta.cost.kind = reported | calculated | estimated +ag.meta.cost.source = provider | harness | agenta +ag.meta.cost.source_name = anthropic | pi | litellm | ... +ag.meta.cost.currency = USD +``` + +Compatibility can continue exposing prompt/completion cost, with these definitions: + +- `prompt` is the total input-side cost, including uncached input, cache reads, and cache creation; +- `completion` is total output-side cost; +- detailed buckets are subcomponents and must not be added again to `prompt` or `total`. + +The API must preserve producer-supplied cost. LiteLLM estimation runs only when no reported or +harness-calculated total exists. Estimated values carry `kind=estimated`, `source=agenta`, and the +pricing source/version when available. + +### Agenta token extension and rollups + +Ingestion should normalize standard OTel fields into canonical Agenta token metrics with inclusive +input/output plus optional cache/reasoning subcategories. Rollups must retain arbitrary approved +numeric buckets or use a schema table, rather than hardcoding three keys. + +The semantic-convention documentation must state whether each subcategory is included in its +parent and define total arithmetic. This prevents cache and reasoning double counting. + +## Compatibility and versioning + +- Change the pre-production `/run` result and event schema directly, with updated goldens. +- Keep a temporary parser for the old flat four-field result only if deployed runner/service skew + can occur during rollout. +- Do not silently reinterpret old `input` as inclusive. Version or normalize based on the wire + shape. +- Keep service-to-API additions backward compatible. Existing prompt/completion/total queries + continue to work while detailed paths become available. +- Update public docs in the same PR that changes API semantics. diff --git a/docs/design/agent-workflows/projects/agent-usage-telemetry/plan.md b/docs/design/agent-workflows/projects/agent-usage-telemetry/plan.md new file mode 100644 index 0000000000..fe7ef7faef --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-usage-telemetry/plan.md @@ -0,0 +1,132 @@ +# Plan + +## Strategy + +Establish the correct pre-production contract first. Normalize Pi and Claude into that contract, +then carry it through the Python service. Treat service-to-API semantic-convention changes as a +separate approval-gated phase. + +## Phase 0: approve semantics and attribution + +Before implementation, review and decide: + +1. the canonical inclusive token rules and cache/reasoning subcategories; +2. cost provenance names and whether Pi's catalog-derived cost is `calculated`; +3. final versus provisional event vocabulary; +4. the one-owner rule for incremental span usage; +5. the cross-batch parent-summary strategy; +6. whether temporary old-runner/new-service compatibility is necessary. + +Exit criterion: runner and service owners approve `interface-design.md`. CTO approval is not +required for the pre-production boundary, but the service-to-API proposal remains gated. + +## Phase 1: canonical runner usage model + +1. Replace flat `AgentUsage` with typed token, cost, provenance, and context objects. +2. Define internal observations with source, scope, temporality, and final/provisional status. +3. Implement finite, non-negative validation that preserves zero and absence distinctly. +4. Implement one field-level reducer. Remove whole-object Pi-writeback precedence. +5. Separate `context_usage` from final billed `usage` events. +6. Return partial reported usage on cancellation and error where available. + +Tests: + +- reducer precedence and incomplete observations; +- zero, absent, negative, non-finite, and cost-only cases; +- incremental observations versus repeated cumulative snapshots; +- terminal usage after provisional context updates; +- partial/error runs. + +## Phase 2: Pi and Claude normalization + +### Pi + +1. Normalize uncached input plus cache-read/cache-write into inclusive input. +2. Preserve cache details, output, total, and any reasoning detail. +3. Preserve Pi cost components and mark provenance accurately. +4. Make usage writeback serialize the rich final observation. +5. Merge writeback field by field instead of replacing other sources. + +### Claude and ACP + +1. Read every final `PromptResponse.usage` field, including cached read/write and explicit total. +2. Map ACP `used/size` only to context utilization. +3. Preserve `cost.amount` and `cost.currency` as a cost snapshot. +4. Ensure repeated cumulative ACP updates replace rather than sum. +5. Document the synthetic run-level LLM span's lack of per-call fidelity. + +Tests use captured or package-faithful fixtures for cache-heavy, multi-tool, resumed, cancelled, +and zero-cost runs on local and Daytona paths. + +## Phase 3: runner-to-service contract + +1. Update `AgentRunResult`, `AgentEvent`, Python `WireAgentUsage`, and typed `AgentResult` usage. +2. Update the SDK catalog schema, generated artifacts if applicable, wire goldens, and drift tests. +3. Decide and document the Vercel compatibility projection. Never expose context occupancy as + token usage. +4. Update service workflow tracing projection to consume the rich shape without truthy-total + gating. +5. Update living agent-workflows interface docs: + - `interfaces/cross-service/runner-to-harness.md`; + - `interfaces/cross-service/service-to-agent-runner.md`; + - `interfaces/cross-service/service-and-runner-trace-export.md`. + +Exit criterion: the rich result survives runner, streaming and non-streaming transports, Python +parsing, and service handling without information loss. + +## Phase 4: prepare trace projection for approval + +1. Build pure, tested projection functions from canonical usage to the candidate OTel and Agenta + attributes without changing exported production spans. +2. Produce before/after fixture traces for leaf attribution, parent summaries, separate OTLP + batches, cache-heavy usage, and multi-model cost. +3. Classify every candidate field as OTel standard, compatibility alias, or Agenta extension. +4. Present cross-batch summary alternatives and their stored/queryed outcomes. +5. Add the proposed trace-tree expectations to the CTO approval packet. + +No export ownership, parent metric, cost attribute, adapter, or API behavior changes in this phase. + +## Phase 5: service-to-API semantic convention (CTO approval required) + +Prepare the approval packet from `interface-design.md`, including before/after traces and these +decisions: + +1. standard OTel token mapping and compatibility aliases; +2. Agenta cost extension, currency, provenance, and component definitions; +3. reported/calculated cost precedence over platform estimates; +4. cache-aware fallback calculation; +5. schema-driven cumulative rollups for cache and reasoning buckets; +6. parent/child attribution and late/separate-batch behavior; +7. query/UI compatibility and reported-versus-estimated presentation. + +After approval: + +1. change runner/service span projection so incremental usage has one owner; +2. implement the approved cross-batch parent-summary strategy; +3. consolidate GenAI mappings into an authoritative versioned table; +4. preserve explicit producer cost and calculate only missing cost; +5. extend cost calculation for normalized, model-scoped cache buckets when pricing supports them; +6. extend token/cost cumulative rollups; +7. add query presets and UI labels where approved; +8. update the public semantic-convention, cost, and API tracing docs in the same implementation PR. + +## Phase 6: end-to-end QA and rollout + +1. Replay deterministic Pi and Claude transcripts through runner and Python service tests. +2. Run live cache-heavy Pi and Claude turns against a provider account and compare provider/harness + raw usage, runner result, stored leaf metrics, and trace totals. +3. Verify streaming and non-streaming parity, local and Daytona parity, continuation/resume, and + cancelled/error usage. +4. Verify old trace queries still resolve prompt/completion/total compatibility fields. +5. Add observability for rejected usage observations, missing cost, estimation fallback, and trace + export degradation without logging secrets or prompts. + +## PR decomposition + +1. Runner internal usage model and reducer. +2. Pi and Claude/ACP adapters plus fixtures. +3. Runner-to-service wire, Python DTO, streaming projection, and living interface docs. +4. Pure trace-projection fixtures and CTO approval packet. +5. Approval-gated trace attribution, API semconv, rollup, cost fallback, UI/query, and public docs. + +Keep implementation PRs narrow. The design PR contains no implementation. diff --git a/docs/design/agent-workflows/projects/agent-usage-telemetry/qa.md b/docs/design/agent-workflows/projects/agent-usage-telemetry/qa.md new file mode 100644 index 0000000000..56c9ed5f07 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-usage-telemetry/qa.md @@ -0,0 +1,83 @@ +# QA plan + +## Contract fixtures + +| Case | Required assertions | +|---|---| +| Pi cache hit | inclusive input equals uncached + read + creation; details and calculated cost survive | +| Pi no cache | missing or zero cache details remain distinguishable according to source | +| Claude cache hit | all PromptResponse buckets survive; explicit total validates | +| Claude ACP update | `used/size` become context only; amount/currency become cost only | +| Repeated ACP snapshots | latest cumulative snapshot replaces; values are not summed | +| Cost-only result | reported zero/nonzero cost survives without token total | +| Cancel/error after usage | partial usage survives with partial/final status | +| Multi-turn continuation | run/turn scope prevents previous cumulative totals from being charged again | +| Unknown model | reported cost survives; estimation remains absent or explicitly unavailable | +| Invalid values | negative/non-finite values are rejected from canonical usage and diagnosed safely | +| Reported total mismatch | normalized total is canonical; raw total and mismatch status survive | +| Multi-model run | breakdown retains provider/model; each estimate is priced before aggregation | +| Mixed provenance | aggregate is marked mixed and retains component provenance | + +## Wire and service tests + +- TypeScript `AgentRunResult` and event goldens use the rich shape. +- Python wire schema and catalog export match TypeScript. +- Streaming and terminal results select the same final usage. +- Vercel metadata receives inclusive input/output/total only; context snapshots never appear as + billed usage. +- Workflow tracing accepts cost-only and reported-zero usage. +- Temporary compatibility parsing, if approved, is covered in both skew directions. + +## Trace-tree tests + +For each supported path, assert the full stored tree rather than individual raw attributes: + +- local Pi with multiple model calls; +- Daytona Pi with usage writeback; +- Claude over ACP with tools; +- streaming and one-shot requests; +- continuation/resume; +- cancellation/error after provider usage. + +Each test proves: + +1. incremental billable usage appears exactly once; +2. parent cumulative totals equal the sum of incremental owners; +3. cache and reasoning are subcategories and do not inflate inclusive totals; +4. context used/window are gauges, not token totals; +5. reported or calculated cost retains amount, currency, and provenance; +6. no API fallback overwrites an explicit cost after the approved API phase. +7. cost components sum once and inclusive prompt cost is not added to its subcomponents. + +## API semantic-convention tests + +- Standard GenAI input/output/cache/reasoning fields map to canonical Agenta metrics. +- Legacy prompt/completion aliases remain compatible without duplicate counting. +- Direct `ag.metrics.unit.costs.*` values survive ingestion. +- LiteLLM estimation runs only when cost is absent. +- Cache-aware estimation applies the correct model rates and labels the result estimated. +- Numeric-vector rollup retains every approved bucket. +- Separate or late OTLP batches do not duplicate parent and child usage. +- Unknown currencies and models fail safely without inventing USD values. +- Stored paths remain queryable through the spans and analytics APIs. + +## Live reconciliation + +For one cache-heavy Pi run and one cache-heavy Claude run, capture: + +1. raw harness/provider usage; +2. runner final result; +3. Python service parsed result; +4. raw OTLP leaf and parent attributes; +5. stored incremental and cumulative Agenta metrics; +6. observability UI total. + +Compare every bucket and cost source in one worksheet. The UI total must equal the chosen reported +or calculated source, and the trace total must not multiply when parent spans are present. + +## Documentation verification + +- Semantic-convention examples state inclusive and subcategory arithmetic. +- Cost docs distinguish reported, harness-calculated, and Agenta-estimated values. +- API docs identify producer-supplied incremental metrics versus server-derived cumulative metrics. +- Agent-workflows living interface docs match the implemented wire shape. diff --git a/docs/design/agent-workflows/projects/agent-usage-telemetry/research.md b/docs/design/agent-workflows/projects/agent-usage-telemetry/research.md new file mode 100644 index 0000000000..57aa7e825c --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-usage-telemetry/research.md @@ -0,0 +1,180 @@ +# Research + +## Executive finding + +The first correctness failure is the runner contract, not the API price table. Pi and Claude +already expose cache-aware usage, but the runner collapses it into four ambiguous scalars. ACP +context occupancy is then stored as token usage. The Python service and Vercel projection preserve +only that reduced shape. + +The API has independent problems that matter once the upstream data is correct. It accepts direct +`ag.metrics.unit.costs.*` attributes, then overwrites them with a LiteLLM calculation. Its tree +rollups retain only prompt, completion, and total. Identical metrics on a parent and child are both +treated as incremental. + +## Current path + +```text +Pi message usage -------------------+ + +--> runner AgentUsage --> /run result --> Python AgentResult +Claude PromptResponse.usage --------+ | | +ACP usage_update.used/cost ----------+ | +--> workflow span + +--> runner OTLP spans + +runner and service OTLP spans --> API adapters --> Agenta attributes --> cost calculation + |--> tree rollup --> query/UI +``` + +## Harness and runner findings + +### The neutral contract loses information + +`services/runner/src/protocol.ts` defines both the streamed `usage` event and final `AgentUsage` +as input, output, total, and cost. It cannot represent cache reads, cache creation, reasoning, +currency, provenance, context occupancy, or whether a value is provisional or final. + +### Pi has richer facts + +Pi assistant messages expose: + +- uncached input tokens; +- output tokens; +- cache-read tokens; +- cache-write tokens; +- a total token count; +- a cost breakdown and total calculated from Pi's model catalog. + +`services/runner/src/tracing/otel.ts` preserves cache fields on each Pi LLM span, but its run +accumulator keeps only input, output, total, and cost. The Pi extension writes that lossy object to +`.agenta-usage.json`, and the outer runner prefers the file wholesale over other sources. + +Pi therefore helps with token and cost tracking, but its cost must carry honest provenance. Unless +the provider itself returned the monetary amount, call it harness-calculated rather than billed. + +### Claude over ACP exposes two different concepts + +The installed Claude ACP adapter returns final `PromptResponse.usage` with input, output, +cached-read, cached-write, and total tokens. It separately streams `usage_update` with: + +- `used`: current context occupancy; +- `size`: context-window capacity; +- `cost.amount` and `cost.currency`: the Claude SDK's run cost when available. + +The runner currently assigns `used` to `usage.total`, drops `size` and currency, then reads only +input and output from the final prompt response. It discards both cache buckets and the explicit +total. This makes provisional context data compete with final billed usage. + +### Source reconciliation is all-or-nothing + +The current precedence is Pi writeback, otherwise final prompt tokens plus stream cost. A partial +writeback suppresses complementary fields from another source. Reconciliation needs field-level +precedence and provenance. + +### ACP tracing has lower call-level fidelity + +Local Pi instrumentation produces one LLM span per assistant/model call. The ACP tracer creates +one synthetic LLM span for the full prompt run because ACP does not expose equivalent per-call +lifecycle and usage detail. The design should preserve correct run totals without presenting them +as per-call measurements. + +## Runner-to-service findings + +- `WireAgentUsage` in `sdks/python/agenta/sdk/agents/wire_models.py` closes the schema to the same + four fields. Golden wire fixtures and generated catalog tests pin it. +- Runtime `AgentResult.usage` becomes an untyped dictionary, so new fields can cross permissively + but have no semantic or validation protection. +- `record_usage` in `sdks/python/agenta/sdk/agents/tracing.py` projects only input, output, total, + and bare cost onto the workflow span. It drops cost-only usage because it gates on a truthy + token total. +- The Vercel stream adapter allowlists the same four fields. A streamed ACP context snapshot can + win over a corrected terminal result because both use the same `usage` event type. +- Failed runs can incur tokens and cost but return no final usage on the shared error path. + +Changing this boundary requires updates to the TypeScript protocol, Python wire model and DTO, +catalog schema, result and event goldens, Vercel projection decision, and transport tests. + +## Service-to-API and semantic-convention findings + +### Span attributes are the transport + +Yes, cache reads currently cross the service-to-API boundary as OTLP span attributes. The runner +uses `gen_ai.usage.cache_read.input_tokens` and +`gen_ai.usage.cache_creation.input_tokens`. The Logfire/GenAI adapter maps them to Agenta token +metrics. + +This is only transport mapping. It does not define the complete Agenta metric semantics, cost +breakdown, attribution, or rollup behavior. + +### Existing Agenta attributes can carry cost today + +The default Agenta adapter accepts `ag.*` attributes. The span builder rewrites +`ag.metrics.unit.costs.*` to `ag.metrics.costs.incremental.*`. A producer can therefore send a +reported total through the existing OTLP transport without inventing a standard GenAI cost key. + +However, the API unconditionally runs `calculate_costs` for supported span types and replaces the +entire incremental cost dictionary with a LiteLLM prompt/completion estimate. This contradicts +the public cost guide, which says calculation is a fallback when cost is absent. + +### Current Agenta rollups are closed + +Cost and token rollups hardcode prompt, completion, and total. Cache and reasoning keys may survive +initial extraction but disappear from cumulative metrics. A correct extension needs schema-driven +or generic numeric-vector rollup with explicit invariants. + +### OpenTelemetry meanings constrain normalization + +The supported OpenTelemetry GenAI vocabulary treats: + +- `gen_ai.usage.input_tokens` as inclusive input tokens; +- cache-read and cache-creation input tokens as input subcategories; +- `gen_ai.usage.output_tokens` as inclusive output tokens; +- reasoning output tokens as an output subcategory. + +Adding cache tokens again to an already normalized `input_tokens` value would double count them. +Pi and Claude raw fields instead expose mutually exclusive uncached input and cache buckets, so +the runner must normalize them before emitting OTel attributes. + +The current OTel registry does not define a monetary cost span attribute. `gen_ai.usage.cost` +should not be presented as standard. Agenta needs a documented extension for cost, currency, and +provenance. + +### Parent rollups can double count + +Pi places usage on leaf LLM spans and repeats run totals on the agent span. The ACP tracer stamps +the final total on its synthetic LLM span and its agent span. The Python service repeats totals on +the workflow span because its OTLP batch is separate. The API treats all these values as +incremental and adds parent values to child cumulative values. + +Incremental usage must have one owner. Parent summaries need an explicitly cumulative/summary +representation or must be omitted when leaf spans are present. + +## Storage and UI findings + +- Trace storage and analytics largely support arbitrary nested numeric paths. +- Observability presets and cells expose only prompt/completion/total cost and token totals. +- The UI does not label reported versus estimated cost. +- Adding cache, reasoning, or cost provenance to stored attributes does not automatically make + them discoverable or understandable in the trace UI. + +## Documentation gaps + +The public semantic-convention page lists only prompt/completion/total. It does not define cache, +reasoning, inclusive totals, currency, provenance, or attribution. The cost guide says explicit +cost is preserved, while the API currently overwrites it. The API tracing guide describes +`ag.metrics` as server-computed even though producers can supply incremental metrics. + +Any stable-boundary implementation must update: + +- `docs/docs/observability/trace-with-opentelemetry/03-semantic-conventions.mdx`; +- `docs/docs/observability/trace-with-python-sdk/06-track-costs.mdx`; +- `docs/docs/reference/api-guide/10-tracing.mdx`; +- the agent-workflows interface inventory for runner-to-harness, service-to-runner, and trace + export. + +## Related work + +`docs/design/agent-workflows/projects/otel-run-recorder-refactor/` proposes separating the ACP run +recorder from OTel span emission. The usage normalization reducer belongs in the protocol-neutral +recorder side of that boundary. This project defines its data semantics and can land before or as +part of that refactor without depending on a file move. + diff --git a/docs/design/agent-workflows/projects/agent-usage-telemetry/status.md b/docs/design/agent-workflows/projects/agent-usage-telemetry/status.md new file mode 100644 index 0000000000..148fdacb62 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-usage-telemetry/status.md @@ -0,0 +1,53 @@ +# Status + +**State:** design ready for review + +**Date:** 2026-07-12 + +## Completed + +- Traced Pi, Claude ACP, runner, Python service, OTLP ingestion, cost calculation, tree rollup, + storage/query, observability UI, tests, and public docs. +- Confirmed the first information loss occurs in the pre-production runner usage contract. +- Confirmed ACP context occupancy is currently conflated with billed token totals. +- Confirmed existing Agenta OTLP attributes can transport direct monetary metrics, but API cost + calculation overwrites them. +- Confirmed cache/reasoning subcategories need inclusive-parent semantics and explicit rollup rules. +- Defined a runner-first implementation sequence and a separate approval-gated API semconv phase. + +## Proposed decisions + +1. Runner work comes first. The canonical result separates usage, cost, provenance, and context. +2. Input/output totals follow current OTel GenAI inclusive semantics. Cache and reasoning are + subcategories. +3. ACP `used/size` is context utilization only. +4. Reported provider cost outranks harness-calculated cost; the runner does not estimate. +5. Incremental billable usage has one span owner. Parent totals are derived or explicitly marked + summaries. +6. Monetary cost uses a documented Agenta extension because OTel has no standard GenAI cost + attribute. +7. Public semantic-convention, cost-tracking, and API tracing docs change with the stable API + implementation. + +## Approval gates + +- Runner and service owners: approve the pre-production wire shape and normalization rules before + phases 1 through 3. Phase 4 only prepares projection fixtures and the approval packet. +- CTO: approve exported span attribution, cross-batch summaries, service-to-API semantic + convention, ingestion precedence, rollup, and public docs before phase 5. + +## Open decisions for review + +1. Which cross-batch parent-summary strategy should replace repeated incremental usage? +2. Can runner and service versions skew during deployment, requiring temporary parsing of the old + flat usage shape? +3. Should the first Vercel projection expose cache details or only inclusive input/output/total? +4. Should API rollups be generic over numeric metric keys or governed by a versioned bucket schema? +5. Should the first UI change label only reported versus estimated total, or also expose cache and + reasoning breakdowns? + +## Next action + +Review `interface-design.md`, especially token inclusivity, cost provenance, span attribution, and +the approval-gated Agenta semantic-convention proposal. No implementation starts until the design +PR is approved. diff --git a/docs/design/agent-workflows/projects/agent-usage-telemetry/trace-inventory.md b/docs/design/agent-workflows/projects/agent-usage-telemetry/trace-inventory.md new file mode 100644 index 0000000000..d6396ac70f --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-usage-telemetry/trace-inventory.md @@ -0,0 +1,84 @@ +# Trace inventory + +This inventory broadens the research beyond cost. It follows runner-emitted trace information +through Agenta ingestion and records which gaps this project fixes. + +## Span structure and identity + +| Concern | Runner source | API handling | Finding | Scope | +|---|---|---|---|---| +| trace and parent ids | incoming W3C `traceparent`; runner span context | native OTel tree | separate batches complicate cumulative rollup | usage attribution design | +| agent span | `invoke_agent`, AGENT | Agenta agent node | usage is repeated as incremental summary | fix after CTO approval | +| turn span | Pi emits real turns; ACP emits one synthetic turn | chain node | ACP cannot express each internal model round | document limitation | +| LLM span | Pi per call; ACP synthetic per run | chat node | fidelity differs by harness | usage normalization and docs | +| tool span | tool call/update events | tool node | ordering and orphan handling belong to recorder refactor | defer | +| session identity | session/conversation attributes | session metadata | present, but not part of usage reconciliation | verify regression only | + +## Model request and response + +| Field | Runner behavior | API destination | Gap and disposition | +|---|---|---|---| +| provider/system | Pi message or selected connection | provider metadata | preserve on every model-scoped usage observation | +| requested model | run config | `ag.meta.request.model` | keep distinct from response model | +| response model | assistant message when available | `ag.meta.response.model` | required for model-scoped pricing | +| response id | Pi assistant message when available | response metadata | preserve; ACP may not expose it | +| finish reasons | assistant stop reason | response metadata | partial/error usage must not be dropped | +| request parameters | limited runner config | request metadata | inventory only | + +## Content and policy + +| Field | Runner behavior | Gap and disposition | +|---|---|---| +| input/output messages | captured on LLM and agent spans | keep existing capture behavior | +| tool input/output | captured on tool spans | unrelated ordering gaps remain in recorder-refactor project | +| content capture policy | `telemetry.capture.content.enabled` | suppress content without suppressing numeric usage | +| redaction | runner redacts events and errors | do not retain raw provider payload by default | + +## Status, errors, and timing + +| Field | Runner behavior | Gap and disposition | +|---|---|---| +| span status and exception | error paths mark spans | partial usage disappears on some failures; fix upstream | +| stop reason | run and assistant result | preserve alongside usage status | +| start/end and duration | native span timing | no cost-specific change | +| export failure | tracing degrades best effort | weak diagnostics are a related follow-up | + +## Usage, context, and cost + +| Field | Current state | Target | +|---|---|---| +| inclusive input/output | ambiguous and harness-dependent | normalize to pinned OTel meaning | +| cache read/creation | lost in run result/rollup | preserve as input subcategories | +| reasoning | not carried by neutral usage | preserve as output subcategory when reported | +| total tokens | arithmetic, provider total, or ACP context used | canonical input + output plus mismatch metadata | +| context used/window | conflated with billed total; window dropped | separate gauge object/event | +| cost amount | bare scalar | currency, provenance, model scope, and status | +| cost components | discarded | retain mutually exclusive components | +| reported versus estimated | indistinguishable | explicit kind/source and fallback-only estimation | +| aggregation | repeated across leaf and parents | one incremental owner plus approved strategy | + +## Adapter and semantic-convention classification + +The API has overlapping mappings in the generic semconv, Logfire/GenAI, OpenInference, +OpenLLMmetry, Vercel AI, and direct Agenta adapters. They do not support the same detail or naming. + +Phase 4 must produce a table with one row per emitted field: + +```text +wire fact | runner attribute | classification/version | adapter mapping | canonical ag path | +aggregation | compatibility alias | documentation link +``` + +At minimum it covers operation name, span kind, provider, requested and response model, response +id, finish reason, messages, tool identity and arguments, session identity, errors/status, +content-capture policy, all token buckets, context gauges, cost components, currency, provenance, +and estimation metadata. + +## Scope decision + +This project fixes usage normalization, context separation, cost provenance, model-scoped +attribution, incremental ownership, and the related semantic convention. It verifies that model, +identity, status, timing, content policy, and tool fields do not regress. + +It defers generic tool-event ordering, exporter concurrency/cache lifetime, export-failure +diagnostics, and the trace-recorder file refactor unless they block correct usage attribution.