Project evaluator traces all land in one shared `evaluators` project instead of a per-evaluator project

[mikeldking]

Summary

Every project evaluator writes its execution traces into a single shared project literally named evaluators. Dataset evaluators already do the right thing — each one gets its own dedicated project — so the two evaluator surfaces are inconsistent, and the project-evaluator side is the wrong one.

Each project evaluator should get its own trace project, the way dataset evaluators do.

Current behavior

Dataset evaluators — one project per evaluator. src/phoenix/server/api/mutations/evaluator_mutations.py:271

project_name = f"dataset-evaluator-{token_hex(12)}"
...
description=f"Traces for dataset evaluator: {dataset_evaluator_name} on dataset: {dataset_name}"

Project evaluators — one shared project for all of them. src/phoenix/server/online_eval/tracing.py:87 resolves a single project by the constant name from src/phoenix/config.py:4081:

EVALUATORS_PROJECT_NAME = "evaluators"

async def evaluators_project_id(session, dialect) -> int:
    """Resolve the evaluator-trace project, creating it if it does not exist."""
    ... {"name": EVALUATORS_PROJECT_NAME, "description": "Traces from evaluator executions"} ...

Every project evaluator on every project shares that one bucket. The only per-evaluator distinction is a span attribute (PROJECT_EVALUATOR_NAME_ATTRIBUTE).

Why this is wrong

1. No isolation between evaluators or between projects. Traces from unrelated evaluators attached to unrelated projects are interleaved in one project. Retention, permissions, and cost all become all-or-nothing across every evaluator in the deployment.

2. The Traces tab reports the shared project's stats, not the evaluator's. On the Metrics/Traces tab for a single evaluator, the Project Info panel shows the shared project's name (evaluators), ID, and total trace count. Reproduced locally: a deployment with two evaluators — correctnesss on project default and correctness-copy on project assistant_agent — shows Total Traces 181 on the correctness-copy tab, which is the count for both evaluators across both projects, not for correctness-copy.

3. It creates a user-visible footgun that the codebase then has to defend against. Because evaluators is a real project with a well-known name, users can attach evaluators to it and cause feedback loops. Three separate places exist only to guard that:

• src/phoenix/server/api/mutations/evaluator_mutations.py:335 — refuses criteria creation on it
• src/phoenix/db/helpers.py:426 — exclude_criteria_targeting_evaluator_traces, applied to both sweep loads
• src/phoenix/server/api/types/Evaluator.py:1294 — reports such criteria as NOT_SCHEDULABLE with reason TARGETS_EVALUATOR_TRACES

A per-evaluator project scheme with generated names (as dataset evaluators use) makes this far less likely to be hit by hand.

4. Name collision with user projects. evaluators is an ordinary, plausible project name. A user who already has a project called evaluators has their project silently adopted as the evaluator-trace sink.

Expected behavior

Each project evaluator gets its own trace project at creation, mirroring the dataset evaluator pattern — a generated name and a description naming the evaluator and the project it evaluates. ProjectEvaluator.traceProject already exists in the GraphQL schema, so the field to hang this off of is in place.

Notes for whoever picks this up

• Migration: existing traces live in the shared evaluators project. Decide whether to backfill by splitting on PROJECT_EVALUATOR_NAME_ATTRIBUTE, or leave history in place and only route new traces.
• The three TARGETS_EVALUATOR_TRACES guards above may be simplifiable or removable once trace projects are generated rather than well-known; that should be evaluated deliberately rather than dropped by default, since generated projects are still real projects a user could target.
• EVALUATORS_PROJECT_NAME in config.py becomes dead once nothing resolves a project by that constant.

[mikeldking]

I can put a server agent on this