docs: add Agent Settings: Grader; enrich Support, Evals, and LTI

New page — Agent Settings: Grader — covering the grading toggle (a
Grading tool attachment), the Grading setup sub-tab (what gets graded,
feedback detail, grading instructions), and the Rubric sub-tab (criteria
table, add/edit/delete modals, points total), plus the misconfiguration
and no-access states. Linked from the overview recap.

Enriched three existing pages with sub-screens that were documented but
not pictured: Evals gains a "Managing Benchmarks" section (benchmark
library, Q&A items, manual vs CSV entry, the Evaluate dialog and LLM
picker, manual scores are 0-1); Support gains the ticket detail pane and
reply thread; LTI Keys gains the key detail dialog; LTI Tools gains the
auth audience and deployment IDs fields plus the create dialog.

Each of the four pages now closes with a "Building X Into Your Own App"
section pointing at the corresponding iblai/vibe skill and SDK component.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: mamigot

os/agent-settings/evals.md

@@ -22,6 +22,8 @@ The panel lists every evaluation run for the agent in a table with **Evaluation*
 
 Each row's **Actions** menu (`⋯`) exposes:
 
+![Actions menu open on an evaluation run, listing View results, New review, Check status, Export CSV, and Delete](/images/docs/os/agent-settings/agent_settings_evals_actions.webp)
+
 #### View results
 Opens the run detail — every trace with its input, expected output, actual output, and any scores.
 
@@ -37,6 +39,36 @@ Downloads the run's results as a spreadsheet — one row per item with `item_id`
 #### Delete
 Removes the run. This is destructive and is confirmed before it proceeds.
 
+![Delete evaluation confirmation modal warning that the run cannot be recovered](/images/docs/os/agent-settings/agent_settings_evals_delete.webp)
+
+**View results**, **New review**, and **Export CSV** stay disabled until the run reaches **Completed**. The status badge is colored by state — green for COMPLETED, red for FAILED, blue for anything still in flight — and **Check status** refreshes a single row without reloading the table.
+
+## Managing Benchmarks
+
+**Manage benchmarks** opens the benchmark library. Benchmarks are **organization-wide**, not tied to one agent — the same test set can be run against several agents, which is what makes it possible to compare them on equal footing. The dialog lists every benchmark with a search box and a **New Benchmark** action that takes a name and a description.
+
+![Manage benchmarks dialog listing the organization's benchmarks with a search box and a New Benchmark button](/images/docs/os/agent-settings/agent_settings_evals_manage_benchmarks.webp)
+
+![New Benchmark dialog with name and description fields](/images/docs/os/agent-settings/agent_settings_evals_new_benchmark.webp)
+
+Opening a benchmark shows its **Q&A items** — the question/answer pairs the agent is tested against. Each item pairs an input with an optional expected output.
+
+![Benchmark items view listing the question and answer pairs in a benchmark, with an Add Q&A button](/images/docs/os/agent-settings/agent_settings_evals_benchmark_items.webp)
+
+**Add Q&A** offers two paths:
+
+#### Manual entry
+Type the question and, optionally, the expected answer directly into the form — the quickest way to add a handful of cases.
+
+![Add Q&A dialog on the manual tab, with fields for the question and its expected output](/images/docs/os/agent-settings/agent_settings_evals_add_qa_manual.webp)
+
+#### CSV upload
+Upload a UTF-8 CSV with a required `input` column and an optional `expected_output` column, up to 10 MB or 10,000 rows. Rows with a blank question are skipped.
+
+![Add Q&A dialog on the CSV tab, showing the file upload area and the required column format](/images/docs/os/agent-settings/agent_settings_evals_add_qa_csv.webp)
+
+The benchmark selector picks the first available benchmark automatically, and **New Evaluation** stays disabled until at least one benchmark exists — so creating a benchmark is the first step on a new agent.
+
 ## Reviews (LLM-as-Judge)
 
 ![Evaluation run detail showing raza-eval-2 marked Completed, 3 traces produced, a Reviews section with a completed accuracy review scored by gpt-5, and the benchmark items listed with their score counts](/images/docs/os/agent-settings/agent_settings_evals_experiment.webp)
@@ -45,14 +77,28 @@ Opening a completed run shows its header — benchmark name, when it started, an
 
 A review grades every response in the run automatically. Selecting **New review** defines a **criteria** rubric in plain language (for example, "Evaluate the response on accuracy") and picks the judge model; a separate LLM then reads each item's input, expected output, and the agent's actual output and assigns a score. Each review shows its status, the judge model that ran it (for example, `gpt-5`), and when it finished. Completed review scores roll up under each benchmark item, so an item can carry several scores from different reviews.
 
+The **Evaluate** dialog collects the criteria, a **score name** (the label the resulting scores are filed under, such as `accuracy`), a **max concurrency** limit for how many traces are judged in parallel, and the judge model.
+
+![Evaluate dialog with a criteria textarea, a score name field, a max concurrency field, and the LLM selector](/images/docs/os/agent-settings/agent_settings_evals_new_review.webp)
+
+The model is chosen provider-first: pick the provider, then one of its models. Any LLM configured for your organization can act as the judge — including a model different from the one the agent itself runs on, which is the usual setup for an independent second opinion.
+
+![LLM picker inside the Evaluate dialog, listing providers and the models available under the selected provider](/images/docs/os/agent-settings/agent_settings_evals_llm_picker.webp)
+
 ![Expanded accuracy review showing the criteria field and four produced scores, each with a numeric value and the judge's written reasoning for that benchmark item](/images/docs/os/agent-settings/agent_settings_evals_review.webp)
 
 Expanding a review reveals its **Criteria** and the **scores produced** — one per benchmark item, each with a numeric value and the judge's written reasoning. In the example above, the *accuracy* review explains why a clear, correct answer earned a full score while an empty or unanswerable response received a lower, neutral one. Reviews are the automated grading path; individual human, numeric, boolean, or categorical annotations can also be attached to any trace, and reusable rubrics can be saved and referenced across runs.
 
+Scores can also be added by hand from a trace. A manual score takes a name, a value between **0 and 1**, and an optional comment, and sits alongside the judge's scores on the same item. Score writes are ingested asynchronously, so a freshly added score appears immediately in the open panel and is confirmed on the next refresh.
+
 ## Programmatic Access
 
 Everything in this panel is available over the platform API, so evaluations can run in CI or from an AI agent: create benchmarks and items (JSON, CSV upload, or from chat traces), start experiment runs, launch LLM-as-Judge reviews, read scores, and export CSV. The API is packaged as the `iblai-api-agent-eval` skill at [github.com/iblai/api](https://github.com/iblai/api).
 
+## Building Evals Into Your Own App
+
+The Evals panel is one of the agent-settings tabs published in the ibl.ai SDK, so the same screen can be mounted in a product you build yourself. Install the [iblai/vibe](https://github.com/iblai/vibe) skills and mount `<AgentEvaluationTab>` from `@iblai/iblai-js/web-containers/next` inside an `AgentSettingsProvider`. It handles benchmarks, runs, reviews, scores, export, and delete on its own; the host supplies a `getLLMProviderDetails` function so the judge picker can show provider logos and display names. The reusable skill for this workflow is [`iblai-vibe-agent-evals`](https://github.com/iblai/vibe/tree/main/skills/iblai-vibe-agent-evals), and the SDK overview is at [Vibe SDK](/developer/applications/vibe).
+
 ## Related
 
 - [Agent Settings: Analytics](analytics.md) — usage reports and dashboards for the agent

os/agent-settings/grader.md

@@ -0,0 +1,90 @@
+# Agent Settings: Grader
+
+![Grader panel on the Grading setup sub-tab, showing the grading toggle in its helper banner, the Grading setup and Rubric sub-tabs, the What gets graded and Feedback shared with the person selectors, the grading instructions textarea, and a Save button](/images/docs/os/agent-settings/agent_settings_grader_setup.webp)
+
+## Overview
+
+The Grader panel is where you set up how an agent grades work against a rubric you define. The panel header reads "Grader" with the subtitle "Set up how your agent grades work against a rubric you define," and a helper banner explains: "Let your agent grade work. When this is on, the agent scores what people submit against the rubric you set up below and shares the result as feedback — great for essays, projects, and practice exercises."
+
+Grading has two halves, split across two sub-tabs. **Grading setup** decides *what* the agent grades and how much of the result it shares back. **Rubric** decides *how* it is scored: a table of criteria, each worth a number of points, with the overall score reported as points earned out of the total.
+
+Grading only runs when all three pieces are in place — the **Grading** tool is attached to the agent, a grading setup has been saved, and the rubric has at least one criterion. Until then the panel shows an amber warning naming what is still missing.
+
+To reach this screen, open the **Edit Agent** modal, keep the **Configurations** tab group selected, and click **Grader** in the left sidebar. The panel is permission-gated: a user without access to the grading endpoints sees a "No access to grading settings" notice in place of the sub-tabs.
+
+## Target Audience
+
+**Administrator** | **Instructor** | **Agent Builder**
+
+## Panel Reference
+
+#### Grading toggle
+The switch in the helper banner attaches or detaches the **Grading** tool on the agent — it is the single activation control. While it is off, both sub-tabs are hidden and the panel shows a hint instead. Turning it off does **not** discard anything: the grading setup and the rubric are kept, so re-enabling grading restores the configuration you already built.
+
+#### Grading setup / Rubric sub-tabs
+Two sub-tabs organize the panel. **Grading setup** holds the three configuration fields; **Rubric** holds the criteria table. The rubric shows a hint to save the setup first until a grading configuration exists.
+
+#### What gets graded
+A dropdown with two options. **A submission** grades a piece of work the person hands in, like an essay or an answer. **The conversation** grades how the whole conversation went.
+
+#### Feedback shared with the person
+A dropdown controlling how much detail the person receives: **Overall feedback only**, **Feedback per criterion**, or **Overall + per criterion**. The overall score is always calculated — this setting only governs how much of it is shared back.
+
+#### Grading instructions
+A required textarea holding the context the agent uses every time it grades. This is where calibration lives: the audience and grade level, how strictly to weigh mistakes, what earns full credit, and what to do when a score is borderline.
+
+#### Save
+Enabled once the grading instructions are non-empty and something on the form has changed. The first save creates the grading configuration; later saves update it.
+
+#### Rubric table
+Each row is one thing the agent looks for and how many points it is worth, with **Name**, **Criteria**, and **Points** columns and a per-row actions menu. A footer reports "Total possible points: N" and "Overall score = points earned ÷ N."
+
+![Rubric sub-tab showing a criteria table with Name, Criteria, and Points columns, an Add criterion button, and a footer with the total possible points and the overall-score formula](/images/docs/os/agent-settings/agent_settings_grader_rubric.webp)
+
+#### Criterion actions menu
+The `⋯` menu on each rubric row offers **Edit** and **Delete**.
+
+![Per-row actions menu on a rubric criterion, offering Edit and Delete](/images/docs/os/agent-settings/agent_settings_grader_rubric_actions.webp)
+
+#### Add / Edit criterion
+A modal with three fields: **Name** (required — a short label such as "Clarity"), **Criteria** (required — what earns the points), and **Points** (a positive number).
+
+![Add criterion modal with Name, Criteria, and Points fields](/images/docs/os/agent-settings/agent_settings_grader_criterion_add.webp)
+
+![Edit criterion modal pre-filled with an existing criterion's name, description, and points](/images/docs/os/agent-settings/agent_settings_grader_criterion_edit.webp)
+
+#### Delete criterion
+A confirmation modal shown before a criterion is removed. The last remaining criterion cannot be deleted while grading is set up — add a replacement first, since a rubric with no criteria would leave grading misconfigured.
+
+![Delete criterion confirmation modal](/images/docs/os/agent-settings/agent_settings_grader_criterion_delete.webp)
+
+## How to Use
+
+#### Step 1: Turn grading on
+Open the **Edit Agent** modal, keep **Configurations** selected, click **Grader** in the sidebar, and switch the toggle in the helper banner on. This attaches the **Grading** tool to the agent — the same state you would see in the Tools panel — and reveals the two sub-tabs.
+
+#### Step 2: Choose what gets graded
+On the **Grading setup** sub-tab, pick **A submission** if the agent should score work the person hands in, or **The conversation** if it should judge how the exchange went overall.
+
+#### Step 3: Decide how much feedback to share
+Set **Feedback shared with the person**. Start with **Overall feedback only** for a light touch, or **Overall + per criterion** when the person should see exactly where each point went.
+
+#### Step 4: Write the grading instructions
+Fill in the required **Grading instructions**. Be explicit about the audience, the calibration you want, and how to resolve borderline scores — this text is sent with every grading run. Then **Save** to create the configuration.
+
+#### Step 5: Build the rubric
+Switch to the **Rubric** sub-tab and use **Add criterion** to add each thing the agent should look for, with a description of what earns the points and a point value. The footer keeps a running total, and the overall score is points earned divided by that total.
+
+#### Step 6: Confirm nothing is missing
+If the amber misconfiguration warning is still showing, grading is on but either the setup has not been saved or the rubric is empty. Grading runs only once the tool is attached, the setup is saved, and at least one criterion exists.
+
+## Behavior Notes
+
+- **The toggle is a tool attachment.** Turning grading on adds the **Grading** tool to the agent; turning it off removes it. If the Grading tool is not available in your organization's tool catalogue, enabling the toggle reports an error.
+- **Your configuration survives a disable.** Detaching the tool deliberately leaves the grading setup and rubric intact, so rubric work is never lost across a disable/re-enable cycle. Re-attaching the tool provisions the configuration record again.
+- **An empty first run is normal.** On a brand-new agent there is no grading configuration yet; the panel treats that as the first-run state, not an error, and the first save creates it.
+- **Grading is scored out of the rubric total.** The overall score is always points earned ÷ total possible points, regardless of the feedback setting.
+
+## Building Grading Into Your Own App
+
+The Grader panel is one of the agent-settings tabs published in the ibl.ai SDK, so the same screen can be mounted in a product you build yourself. Install the [iblai/vibe](https://github.com/iblai/vibe) skills and mount `<AgentGraderTab>` from `@iblai/iblai-js/web-containers/next` inside an `AgentSettingsProvider`; every label is overridable through a `labels` prop. The reusable skill for this workflow is [`iblai-vibe-agent-grader`](https://github.com/iblai/vibe/tree/main/skills/iblai-vibe-agent-grader), and the SDK overview is at [Vibe SDK](/developer/applications/vibe).

os/agent-settings/lti-keys.md

@@ -32,7 +32,14 @@ Each key is a row with:
 - **PUBLIC KEY** — a truncated preview of the PEM-encoded public key (beginning `-----BEGIN PUBLIC KEY-----`).
 - **ACTIONS** — a menu per row. **Edit** opens the key detail, where you can rename the key and view its full PEM public key and its public JWK (the JSON Web Key form published at the JWKS endpoint). **Delete** removes the key after confirmation — note that a key cannot be deleted while an LTI tool still references it.
 
-The list is paginated when there are many keys.
+The list is paginated at 10 rows per page.
+
+#### Key detail
+Choosing **Edit** opens the key's detail dialog: the key name can be changed here, and the full **Public Key (PEM)** and **Public JWK** are shown with copy buttons. Only the public halves are exposed — the private key never leaves the platform, which is what lets the LMS verify a launch without ever holding a secret of yours.
+
+![LTI key detail dialog showing the key name field, the full PEM-encoded public key, and the public JWK, each with a copy button](/images/docs/os/agent-settings/agent_settings_lti_key_detail.webp)
+
+Deleting is blocked while a tool still references the key, and the reason is surfaced in the dialog rather than failing silently — re-point the tool at another key first.
 
 ## How to Use
 

os/agent-settings/lti-tools.md

@@ -31,7 +31,13 @@ Opens the tool registration form. Its fields hold values that come from the LTI
 - **Auth login URL** — the platform's OIDC authorization endpoint.
 - **Auth token URL** — the platform's OAuth2 token endpoint.
 - **Key set** — the platform's public keys, provided either as a **JWKS URL** (for example `https://lms.example.com/.well-known/jwks.json`) or as raw JWKS JSON.
+- **Auth audience** — optional, and usually left blank; set it only when the platform requires a specific audience value on the token request.
 - **Signing key** — one of the platform-wide keys from the **Keys** sub-tab, used to sign LTI messages for this platform. Create the key first if none exists.
+- **Deployment IDs** — the deployment identifiers the LTI platform assigns, entered one per line or comma-separated. A single registration can carry several deployments, which is how one LMS covers multiple sub-accounts or course shells.
+
+Every field except the auth audience is required. Title, issuer, client ID, and the two OIDC URLs come straight from the LMS registration — copy them rather than retyping, since a single character's difference makes launches fail verification.
+
+![Create LTI Tool dialog with fields for the title, issuer, client ID, auth login and token URLs, auth audience, the JWKS key set, the signing key selector, and deployment IDs](/images/docs/os/agent-settings/agent_settings_lti_tool_create.webp)
 
 #### Tools table
 Each registered platform is a row with: