Skip to content

eval_first.md

Latest commit

 

History

History
209 lines (142 loc) · 9.57 KB

eval_first.md

File metadata and controls

209 lines (142 loc) · 9.57 KB
Raw

Eval-First

Read this when you need to prevent silent prompt regressions in CI. Skip if your LLM output is evaluated only by humans and never gated in an automated pipeline.

If you change prompts by feel, you ship regressions by feel.

Concrete scenario: SummarizeArticle has been running in production for two weeks. Customer success notices that complaints about service outages keep getting tone: "analytical" instead of "negative" — so their "critical feedback" filter silently misses angry users. Someone tweaks the system prompt to emphasise negative sentiment. It fixes the outage article but now three neutral product-update articles get misclassified as "negative". You find out from a Slack thread.

That is the cost of prompt-by-feel. Evals are how you stop it.

ruby_llm-contract works best when you treat evals as the source of truth:

  1. Capture real failures from production (the outage article, verbatim).
  2. Turn them into eval cases (add_case "service outage complaint").
  3. Change the prompt.
  4. Re-run the same eval — plus all previously-passing cases.
  5. Merge only if the eval says quality improved or stayed safe on every case.

Core rule

Do not start with the prompt. Start with the eval.

Using the SummarizeArticle step from the README:

SummarizeArticle.define_eval("regression") do
  add_case "ruby release",
           input: "Ruby 3.4 shipped with frozen string literals...",
           expected: { tone: "analytical" }  # partial match

  add_case "critical review",
           input: "Mesh networking hardware failed under load...",
           expected: { tone: "negative" }
end

"Partial match" means expected is treated as a subset of parsed_output — each key in expected must equal the corresponding key in the output, but extra keys in the output are ignored. So the ruby release case passes whenever the model produces tone: "analytical", regardless of what shows up in tldr or takeaways. To assert on multiple keys, list them all in expected.

Only after the eval exists, touch the prompt-shaping building blocks of the Step: system, rule, and example (blocks inside prompt do), validate (class-level invariant checks that run after the model returns), or the prompt version itself. See Getting Started for the DSL surface.

Validate your oracle before you trust the score

An eval is exactly as good as its ground-truth. Green tells you the system does what you specified, never what you wanted. Two failure modes show up the moment a regression eval is taken seriously:

1. Wrong oracle (green-but-wrong). You hand-label the expected output — expected: { page_type: "Article" } — the classifier returns Article, eval passes 8/8 stable. Then a real audit catches that Article was the wrong target all along: it triggers a cascade of downstream Article-scoped checks that fail on the actual page. The eval gated the label, not the downstream effect. Fix: assert the target effect, not a convenient proxy. If page_type matters because it changes which audit rules run, the case should assert which rules fire (or don't), not the label string.

2. Unvalidated LLM-judge oracle. You add a cheap gpt-5-nano judge that scores "is this output self-contained?" and gate prompt changes on it. The unit test for the judge stubs the judge's verdict and checks the scoring math — it proves the aggregation works, never that the judge's real verdicts agree with reality. A miscalibrated judge gives you a confident percentage that doesn't match what a human reviewer would say. Fix: before the judge gates anything, validate it against a human-labeled golden set of real production samples (not synthetic, not edge-cases-you-imagined). Only then do its before/after numbers mean what you think. See LLM-judge pattern for the full build → calibrate → gate workflow in code.

Rules:

  • Assert the target effect, not a proxy label — pick the assertion by reading what downstream actually does with the output, not what's convenient to label.
  • Validate the oracle (human labels on real prod samples) before trusting any eval's score.
  • For semantic checks that span multiple inputs/locales, use an LLM-judge (one prompt, all cases) — not per-case regex that drifts.
  • Keep LLM-judges in the eval suite (on-demand or scheduled), never on the production request path.

Three eval kinds

1. smoke — wiring check, offline

SummarizeArticle.define_eval("smoke") do
  default_input "Ruby 3.4 shipped with frozen string literals..."
  sample_response({
    tldr: "...",
    takeaways: ["point one", "point two", "point three"],
    tone: "analytical"
  })
end

sample_response returns canned data. Zero API calls. Verifies schema + validates parse and the step wiring is intact. Not a quality signal.

2. regression — real quality measurement

Represent real traffic and known failures. Good sources: production logs, bad completions, incidents, QA edge cases, cases a human had to correct.

Every production failure becomes add_case. That's the flywheel.

3. ab — prompt iteration

Compare two prompt versions on the same eval:

diff = SummarizeArticleV2.compare_with(
  SummarizeArticleV1,
  eval: "regression",
  model: "gpt-4.1-mini"
)

diff.safe_to_switch?  # => true if no cases regressed

This is the cleanest eval-first move: same eval, same cases, two prompt versions, one answer.

What counts as eval-first

Good — eval exists before the prompt change:

SummarizeArticle.define_eval("regression") do
  add_case "short article", input: "...", expected: { tone: "neutral" }
end

# Prompt iteration happens afterward
diff = SummarizeArticleV2.compare_with(
  SummarizeArticleV1, eval: "regression", model: "gpt-4.1-mini"
)

Bad:

# Tweak prompt for an hour
# Maybe add an example
# Maybe tighten a rule
# Then eyeball one or two responses

That's prompt guessing, not eval-first.

sample_response: useful, but not the main thing

Good for: offline smoke tests, local development, testing evaluator wiring, checking schema + validate behavior with zero API calls.

Not enough for real prompt decisions. For those:

  • run_eval(..., context: { model: "..." }) with a real model, or pass an explicit adapter via context: { adapter: ... }. An adapter here is the layer that actually executes the LLM call — RubyLLM::Contract::Adapters::Test returns canned data, the default RubyLlm adapter hits the real provider. model: chooses the model name; adapter: chooses whether the call goes live or stays canned.
  • compare_with(...) for prompt A/B.

compare_with intentionally ignores sample_response — canned data would make both sides look the same.

Parallel eval runs

For larger datasets, run_eval accepts a concurrency: argument — cases run in parallel using a thread pool:

report = SummarizeArticle.run_eval("regression",
  context: { model: "gpt-4.1-mini" },
  concurrency: 8)

Same accepted by compare_models and optimize_retry_policy. Thread count is a ceiling — dataset order of results is preserved. Keep it low enough to respect the provider's rate limits.

Budgeting an eval before you run it

estimate_eval_cost gives you a cost projection without calling the LLM:

SummarizeArticle.estimate_eval_cost("regression",
  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }

Use it in CI to decide which models are worth running regression on, or to cap worst-case spend per build.

Team workflow

  1. Build one eval that matters — 10–30 cases representing real mistakes and important business paths.
  2. Gate CI — pass_eval("regression").with_context(model: "...").with_minimum_score(0.8). See Getting Started for the full matcher chain.
  3. Save a baseline — report.save_baseline! makes quality drift visible.
  4. Change prompts only through comparison — pass_eval(...).compared_with(SummarizeArticleV1) in CI so any regression blocks the merge.
  5. Feed production failures back — every miss in prod → new add_case, then fix. The eval gets stronger over time.

Few-shot examples fit naturally

Adding example input: ..., output: ... inside the prompt is still a prompt change. The eval-first way:

  1. Add examples to the prompt.
  2. Rerun the existing regression eval.
  3. compare_with against the old prompt.

Few-shot isn't the proof. The eval is.

Model selection comes after prompt stability

Don't optimize cost before you stabilize quality:

  1. Build regression.
  2. Improve the prompt with compare_with.
  3. Lock quality in CI.
  4. Then run compare_models (see Optimizing retry_policy).
comparison = SummarizeArticle.compare_models(
  "regression",
  candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" }]
)

comparison.best_for(min_score: 0.95)

Strong defaults for teams

  • smoke uses sample_response.
  • regression uses real model calls.
  • Every prompt change uses compare_with.
  • Every merge runs pass_eval.
  • Every production failure becomes a new add_case.

Short version

  1. Write define_eval before touching the prompt.
  2. Treat sample_response as smoke only.
  3. Use run_eval("name", context: { model: "..." }) for real quality measurement.
  4. Use compare_with for every serious prompt change.
  5. Gate merges with pass_eval.
  6. Feed every production miss back into the dataset.

Prompts stop being vibes and start being engineering.