Feature idea: an `openeval`/EvalPort export target for `evaling export` (interop with other eval frameworks)

[adhabnr-ux]

Hi — really nice project; the variant×model matrix + scorecard + --dry-run cost estimate combination is exactly what I've wanted from ad-hoc eval scripts. I have a possibly-useful feature idea rather than a bug, so feel free to close if it's out of scope for now.

The idea

I maintain EvalPort (Apache 2.0), an open interchange spec + Python/TS SDK for portable eval test cases, graders, and results — the idea being that a suite or result set produced by one framework can be read by another without hand-translation. It's pre-1.0 (currently 1.0.0-rc.4) but has 32 real framework adapters merged (e.g. mlflow-openeval-adapter, langsmith-openeval-adapter) and has landed in Inspect AI's community extensions list.

evaling already has an evaling export RUN --format json|csv|md|html command with what looks like a small, addable format registry (docs/cli.md). A --format openeval (or a standalone converter, if you'd rather keep the CLI provider-agnostic) that emits a spec-valid EvalPort ResultSet would let an evaling run be diffed/aggregated/re-graded by any other EvalPort-speaking tool without writing a bespoke parser for results.jsonl.

How the real fields line up

I read src/evaling/config/schema.py and EvalPort's schemas (spec/schemas/{suite,testcase,grader,resultset}.json) to check this isn't hand-wavy:

evaling	EvalPort
Case (id, vars, expected)	TestCase (id, input, expected_output)
CriterionSpec + ScorerSpec (criterion, scorer.type)	Grader (id, type, params)
a results.jsonl record (case_id, output, scores, latency_ms)	a ResultSet.results[] entry (test_case_id, actual_output, grader_results[], duration_ms)
run.json (id, started_at, finished_at)	ResultSet (run_id, started_at, completed_at)

Sketch, using the real field names on both sides:

# evaling: what's already in a scorecard (src/evaling/config/schema.py)
CriterionSpec(
    criterion="cites-the-signal",
    weight=1.0,
    scorer=ScorerSpec(type="exact"),
)

// EvalPort spec/schemas/grader.json + testcase.json shape
{
  "id": "gr_cites-the-signal",
  "type": "exact_match"
}

// one results.jsonl record -> one ResultSet.results[] entry
{
  "test_case_id": "midnight-watch",
  "actual_output": "Fraud — the charge is card not present at an unusual hour for a first-time merchant.",
  "grader_results": [
    {"grader_id": "gr_cites-the-signal", "type": "exact_match", "score": 1.0, "passed": true}
  ],
  "passed": true,
  "duration_ms": 1204,
  "metadata": {"variant": "concise", "model": "claude-sonnet-5", "cost_usd": 0.0021}
}

Two honest wrinkles, not glossed over:

1. The matrix vs. single-provider ResultSet. EvalConfig runs variant × model × case, but EvalPort's ResultSet carries one provider per run. The natural fix is one ResultSet per (variant, model) pair from a single evaling run, with variant carried in metadata (as sketched above) since the spec doesn't have a first-class variant concept — that'd be closer to an EvalPort spec gap than an evaling one.
2. Scorer coverage isn't 1:1. exact/contains/regex/json-schema/llm-judge map cleanly onto exact_match/contains/regex/json_schema/llm_judge. python maps reasonably onto EvalPort's custom grader type. not-contains and agreement (the judge-calibration scorer, which is genuinely evaling-specific) don't have a clean EvalPort equivalent yet — those would need to round-trip through metadata rather than a typed grader, at least for now.

Ask

Does this look worth a PR, and if so — would you rather it live as --format openeval inside evaling itself, or as a standalone evaling-side converter script/package (mirroring how EvalPort's own adapters are structured, e.g. the two linked above)? Happy to build either, on whichever side is a better fit for how you want evaling's dependency surface to look, and it'd obviously need real converter tests per your fuzzing/mutation-testing bar in CONTRIBUTING.md before it'd be worth merging.

— Sahi, independent contributor (not affiliated with this project)