Part of #11650. Promotes the backend half of the "eval results observability — computed distributions, time-series of eval results" bullet on the epic. Pairs with the Evaluator Details View design (stat cards, pass-rate trend, output distribution, recent evaluations).
Current state
ProjectEvaluator (src/phoenix/server/api/types/Evaluator.py:1145) exposes configuration only — there are zero aggregate/metric fields on any evaluator GraphQL type (no counts, pass rate, avg score, distributions, or time series).
- Results land as
SpanAnnotation / ProjectSessionAnnotation rows with no evaluator FK. The only linkage is name == ProjectEvaluatorCriteria.name plus identifier == "online:" + sha256(config)[:16] (src/phoenix/server/online_eval/derivation.py:79). Editing an evaluator changes the fingerprint, so historical annotations from a prior config carry a different identifier.
- Errors/attempts are recorded on
eval_work_units / eval_session_work_units (src/phoenix/db/models.py:3693, :3763) but terminal rows are reaped (TTL drain), so error rates over long windows are lossy today.
- All the machinery to mirror already exists at the project level:
Project.spanAnnotationSummary (Project.py:1209 → AnnotationSummary label fractions), spanAnnotationScoreTimeSeries (Project.py:2105), spanAnnotationMetricsTimeSeries (Project.py:2219, takes annotationName), shared builders _annotation_score_time_series / _annotation_metrics_time_series.
Scope
Add evaluator-scoped result metrics to ProjectEvaluator, filtered by the annotation linkage above (all fields take an optional TimeRange):
Out of scope
Refs: #11650, #15307 (details page shell), internal_docs/specs/online-evals.md § Triage failures in the UI
Part of #11650. Promotes the backend half of the "eval results observability — computed distributions, time-series of eval results" bullet on the epic. Pairs with the Evaluator Details View design (stat cards, pass-rate trend, output distribution, recent evaluations).
Current state
ProjectEvaluator(src/phoenix/server/api/types/Evaluator.py:1145) exposes configuration only — there are zero aggregate/metric fields on any evaluator GraphQL type (no counts, pass rate, avg score, distributions, or time series).SpanAnnotation/ProjectSessionAnnotationrows with no evaluator FK. The only linkage isname == ProjectEvaluatorCriteria.nameplusidentifier == "online:" + sha256(config)[:16](src/phoenix/server/online_eval/derivation.py:79). Editing an evaluator changes the fingerprint, so historical annotations from a prior config carry a different identifier.eval_work_units/eval_session_work_units(src/phoenix/db/models.py:3693,:3763) but terminal rows are reaped (TTL drain), so error rates over long windows are lossy today.Project.spanAnnotationSummary(Project.py:1209→AnnotationSummarylabel fractions),spanAnnotationScoreTimeSeries(Project.py:2105),spanAnnotationMetricsTimeSeries(Project.py:2219, takesannotationName), shared builders_annotation_score_time_series/_annotation_metrics_time_series.Scope
Add evaluator-scoped result metrics to
ProjectEvaluator, filtered by the annotation linkage above (all fields take an optionalTimeRange):evaluationCount— number of results in rangeannotationSummary— label fractions + mean score, reusing theAnnotationSummaryshape; pass rate is derivable client-side from the output config's labels/optimization directionannotationLabelDistribution— label → count (the design's true/false/error distribution bars); include an error bucket sourced from work-unitERRORrowsannotationScoreTimeSeries— evaluator-scoped variant ofspanAnnotationScoreTimeSeries, honoring the evaluation target (span/trace/session)errorCount— fromeval_work_units.status = 'ERROR'(document the retention caveat, or extend retention for terminal error rows)name+identifier LIKE 'online:%') or only the current fingerprint? Recommendation: span versions for counts/series, so editing an evaluator doesn't zero its historyOut of scope
[needs-ticket]bullet)Refs: #11650, #15307 (details page shell),
internal_docs/specs/online-evals.md§ Triage failures in the UI