|
| 1 | +# Agent Settings: Evals |
| 2 | + |
| 3 | + |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The Evals panel measures how well an agent performs by running it against a **benchmark** — a set of test questions — and scoring every response, so you can catch weak spots and track improvements as you refine the agent. The panel header reads: "Run this agent against a benchmark and score the responses," and the helper banner explains: "Measure how well your agent performs. Run it against a benchmark — a set of test questions — and every response gets scored, so you can catch weak spots and track improvements as you refine the agent." |
| 8 | + |
| 9 | +A benchmark is a graded test set: each item pairs an **input** (the question) with an optional **expected output** (the answer you want). These evaluation benchmarks are separate from an agent's [Datasets](datasets.md) — Datasets are the RAG documents that ground the agent's answers, while benchmarks are the test cases used to grade them. Benchmark items can be authored directly, uploaded from a CSV, or seeded from existing chat traces so real conversations become test cases. |
| 10 | + |
| 11 | +Running an evaluation is asynchronous: it dispatches a background job that sends every benchmark question to the agent and records each response as a **trace**. A run moves through `pending → in progress → completed`, and must reach **Completed** before it can be reviewed or exported. |
| 12 | + |
| 13 | +To reach this screen, open the **Edit Agent** modal, go to the **Analytics** tab, and select **Evals** from its sidebar (alongside Memory, History, and Audit). Evals is admin-gated: it is only available to users with permission to view the agent's analytics. Evaluation data is scoped to your organization and isolated — no other organization can read your benchmarks, runs, or scores. |
| 14 | + |
| 15 | +## Target Audience |
| 16 | + |
| 17 | +**Administrator** | **Agent Builder** |
| 18 | + |
| 19 | +## Running an Evaluation |
| 20 | + |
| 21 | +The panel lists every evaluation run for the agent in a table with **Evaluation**, **Status**, **Initiated By**, **Created**, and **Actions** columns. Above the table, a benchmark selector chooses which test set to run, **Manage benchmarks** opens benchmark authoring, and **New Evaluation** starts a run of the selected benchmark against the current agent. |
| 22 | + |
| 23 | +Each row's **Actions** menu (`⋯`) exposes: |
| 24 | + |
| 25 | +#### View results |
| 26 | +Opens the run detail — every trace with its input, expected output, actual output, and any scores. |
| 27 | + |
| 28 | +#### New review |
| 29 | +Starts an LLM-as-Judge review of the run (see below). |
| 30 | + |
| 31 | +#### Check status |
| 32 | +Polls the background task's current state (`pending`, `in progress`, `completed`, or `failed`). |
| 33 | + |
| 34 | +#### Export CSV |
| 35 | +Downloads the run's results as a spreadsheet — one row per item with `item_id`, `input`, `expected_output`, `actual_output`, `trace_id`, and one column per score. |
| 36 | + |
| 37 | +#### Delete |
| 38 | +Removes the run. This is destructive and is confirmed before it proceeds. |
| 39 | + |
| 40 | +## Reviews (LLM-as-Judge) |
| 41 | + |
| 42 | + |
| 43 | + |
| 44 | +Opening a completed run shows its header — benchmark name, when it started, and how many traces it produced — above a **Reviews** section. As the panel explains: "Reviews are LLM-as-Judge runs that score every trace in this experiment. Scores from completed reviews appear under each item below." |
| 45 | + |
| 46 | +A review grades every response in the run automatically. Selecting **New review** defines a **criteria** rubric in plain language (for example, "Evaluate the response on accuracy") and picks the judge model; a separate LLM then reads each item's input, expected output, and the agent's actual output and assigns a score. Each review shows its status, the judge model that ran it (for example, `gpt-5`), and when it finished. Completed review scores roll up under each benchmark item, so an item can carry several scores from different reviews. |
| 47 | + |
| 48 | + |
| 49 | + |
| 50 | +Expanding a review reveals its **Criteria** and the **scores produced** — one per benchmark item, each with a numeric value and the judge's written reasoning. In the example above, the *accuracy* review explains why a clear, correct answer earned a full score while an empty or unanswerable response received a lower, neutral one. Reviews are the automated grading path; individual human, numeric, boolean, or categorical annotations can also be attached to any trace, and reusable rubrics can be saved and referenced across runs. |
| 51 | + |
| 52 | +## Programmatic Access |
| 53 | + |
| 54 | +Everything in this panel is available over the platform API, so evaluations can run in CI or from an AI agent: create benchmarks and items (JSON, CSV upload, or from chat traces), start experiment runs, launch LLM-as-Judge reviews, read scores, and export CSV. The API is packaged as the `iblai-api-agent-eval` skill at [github.com/iblai/api](https://github.com/iblai/api). |
| 55 | + |
| 56 | +## Related |
| 57 | + |
| 58 | +- [Agent Settings: Analytics](analytics.md) — usage reports and dashboards for the agent |
| 59 | +- [Agent Settings: Chat History](history.md) — the raw conversation traces evaluations can be seeded from |
| 60 | +- [Agent Settings: Datasets](datasets.md) — the RAG documents that ground the agent (distinct from evaluation benchmarks) |
0 commit comments