Skip to content

Add observability for attempts abandoned before their first chunk #238

Description

@peteski22

Problem

When a streaming (or non-streaming) attempt fails before its first chunk, the runner abandons it and — if a successor exists — fails over to the next entry in the plan. There is currently no metric for how often this happens; only WARNING logs (Streaming attempt failed ..., All upstream attempts failed).

That makes it hard for operators to spot pathological failover/timeout waste: an abandoned attempt may still have incurred upstream cost, and a spike in abandonments (e.g. from a too-tight first-chunk timeout, or a flaky primary provider) is invisible in the metrics today. metrics.py has request/token/cost counters, but nothing per-attempt.

Proposal

  • Add a Prometheus counter for abandoned attempts, labelled by provider, model, reason (timeout | build_error | upstream_error), and position, incremented in the streaming runner's on_attempt_failed and the non-streaming attempt loop.
  • Optionally record an input-token estimate per abandoned attempt as a rough waste signal (output tokens are unknown — no usage chunk arrives before the first chunk).
  • Standalone deployments with pricing configured could additionally surface an informational cost estimate from the bundled pricing dataset.

Only the upstream provider's own billing is ground truth; these signals would be relative/trend indicators to catch pathological waste, not exact accounting.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions