Skip to content

[online evals] expose evaluator result metrics over GraphQL #15531

Description

@mikeldking

Part of #11650. Promotes the backend half of the "eval results observability — computed distributions, time-series of eval results" bullet on the epic. Pairs with the Evaluator Details View design (stat cards, pass-rate trend, output distribution, recent evaluations).

Current state

  • ProjectEvaluator (src/phoenix/server/api/types/Evaluator.py:1145) exposes configuration only — there are zero aggregate/metric fields on any evaluator GraphQL type (no counts, pass rate, avg score, distributions, or time series).
  • Results land as SpanAnnotation / ProjectSessionAnnotation rows with no evaluator FK. The only linkage is name == ProjectEvaluatorCriteria.name plus identifier == "online:" + sha256(config)[:16] (src/phoenix/server/online_eval/derivation.py:79). Editing an evaluator changes the fingerprint, so historical annotations from a prior config carry a different identifier.
  • Errors/attempts are recorded on eval_work_units / eval_session_work_units (src/phoenix/db/models.py:3693, :3763) but terminal rows are reaped (TTL drain), so error rates over long windows are lossy today.
  • All the machinery to mirror already exists at the project level: Project.spanAnnotationSummary (Project.py:1209AnnotationSummary label fractions), spanAnnotationScoreTimeSeries (Project.py:2105), spanAnnotationMetricsTimeSeries (Project.py:2219, takes annotationName), shared builders _annotation_score_time_series / _annotation_metrics_time_series.

Scope

Add evaluator-scoped result metrics to ProjectEvaluator, filtered by the annotation linkage above (all fields take an optional TimeRange):

  • evaluationCount — number of results in range
  • annotationSummary — label fractions + mean score, reusing the AnnotationSummary shape; pass rate is derivable client-side from the output config's labels/optimization direction
  • annotationLabelDistribution — label → count (the design's true/false/error distribution bars); include an error bucket sourced from work-unit ERROR rows
  • annotationScoreTimeSeries — evaluator-scoped variant of spanAnnotationScoreTimeSeries, honoring the evaluation target (span/trace/session)
  • errorCount — from eval_work_units.status = 'ERROR' (document the retention caveat, or extend retention for terminal error rows)
  • A paginated connection of recent results (annotation + evaluated artifact ref, ordered by time desc) to back the "Recent evaluations" table
  • Decide and document fingerprint semantics: do metrics span config versions (match on name + identifier LIKE 'online:%') or only the current fingerprint? Recommendation: span versions for counts/series, so editing an evaluator doesn't zero its history
  • Dataloaders for the aggregate fields so the evaluators table can eventually render metric columns without N+1

Out of scope

Refs: #11650, #15307 (details page shell), internal_docs/specs/online-evals.md § Triage failures in the UI

Metadata

Metadata

Assignees

Type

No type

Projects

Status
Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions