Problem
When a streaming (or non-streaming) attempt fails before its first chunk, the runner abandons it and — if a successor exists — fails over to the next entry in the plan. There is currently no metric for how often this happens; only WARNING logs (Streaming attempt failed ..., All upstream attempts failed).
That makes it hard for operators to spot pathological failover/timeout waste: an abandoned attempt may still have incurred upstream cost, and a spike in abandonments (e.g. from a too-tight first-chunk timeout, or a flaky primary provider) is invisible in the metrics today. metrics.py has request/token/cost counters, but nothing per-attempt.
Proposal
- Add a Prometheus counter for abandoned attempts, labelled by
provider, model, reason (timeout | build_error | upstream_error), and position, incremented in the streaming runner's on_attempt_failed and the non-streaming attempt loop.
- Optionally record an input-token estimate per abandoned attempt as a rough waste signal (output tokens are unknown — no usage chunk arrives before the first chunk).
- Standalone deployments with pricing configured could additionally surface an informational cost estimate from the bundled pricing dataset.
Only the upstream provider's own billing is ground truth; these signals would be relative/trend indicators to catch pathological waste, not exact accounting.
Problem
When a streaming (or non-streaming) attempt fails before its first chunk, the runner abandons it and — if a successor exists — fails over to the next entry in the plan. There is currently no metric for how often this happens; only WARNING logs (
Streaming attempt failed ...,All upstream attempts failed).That makes it hard for operators to spot pathological failover/timeout waste: an abandoned attempt may still have incurred upstream cost, and a spike in abandonments (e.g. from a too-tight first-chunk timeout, or a flaky primary provider) is invisible in the metrics today.
metrics.pyhas request/token/cost counters, but nothing per-attempt.Proposal
provider,model,reason(timeout|build_error|upstream_error), andposition, incremented in the streaming runner'son_attempt_failedand the non-streaming attempt loop.Only the upstream provider's own billing is ground truth; these signals would be relative/trend indicators to catch pathological waste, not exact accounting.