Implement agent solvers claude_code() and codex_cli() - #218
Open
simonpcouch wants to merge 13 commits into
Open
Conversation
These solvers evaluate the Claude Code and Codex coding agents on a Task's dataset by bridging to Python Inspect's inspect_swe package via reticulate, so that sandboxing, model-call proxying, and CLI version management are all maintained upstream. The bridged eval runs without a Python scorer and its log is read back into ellmer Chats with the existing log-read machinery, meaning the same R scorer can grade both a homegrown harness and an off-the-shelf coding agent.
Scorer-less logs (e.g. the intermediate Inspect logs written by agent solvers) store an empty scores array, which log_header_to_overview() previously assumed was a nonempty data frame.
Adds a section on evaluating Claude Code with the same dataset and scorer as a homegrown harness, the core workflow these solvers exist for.
Some agent harnesses send more than one user message per generation (Codex sends its environment context separately from the prompt), and ellmer's get_tokens() requires user and assistant turns to alternate. Accumulating user-role content until an assistant message arrives yields chats that ellmer can compute usage for, without changing behavior for alternating transcripts.
Remap Inspect provider prefixes that ellmer names differently (google, vertex, bedrock) when constructing chats from model strings, and preflight that construction at solve time so unsupported providers fail before the sandboxed eval spends money. Coerce agent_args whole numbers to integers (Python int params reject floats), guard the coercion against integer overflow, and reject reserved arguments (epochs and log paths in ..., agent options with dedicated arguments in agent_args) with pointed errors. Wrap the Python imports with guidance for user-configured environments, require reticulate >= 1.41, report how many sample transcripts survived a failed eval, and avoid echoing the input as the result when an agent produces no assistant message. Adds direct tests for consecutive user-role message merging and parallel tool-result coalescing in turns_from_messages().
The fixture only exercises a sample's messages, output, model usage and the `output` field of its model event; attachments, sandbox events and the model event's tool definitions made up most of the 238K file. It's now build-ignored like the other Python Inspect logs, so `claude_code_log()` skips when it isn't installed.
simonpcouch
commented
Jul 27, 2026
simonpcouch
commented
Jul 27, 2026
The solvers now take `(model, ..., version, sandbox)`, with `...` routed by name to either the inspect_swe agent or Python Inspect's `eval()` — the only names the two signatures share are `model` and `sandbox`, both formals, so the split is unambiguous. `sandbox` also accepts a type/config pair, and the docs describe the workspace the agent lands in. Each agent now requires the SDK its bridge speaks (`anthropic` for Claude Code, `openai` for Codex) regardless of which provider serves the model; running Claude Code on a Gemini model failed on this before. Samples that die partway are given a response turn so that logging doesn't fail on a transcript with no assistant message.
The transcripts the solvers hand back were already ellmer Chats, reconstructed by guessing an ellmer provider from Inspect's `provider/model` string. Taking the chat itself removes the guess: the user's object is the template, so the log's provider, model and system prompt round-trip exactly, and the chat's `params()` are forwarded to Inspect's generation config. The solver also accepts `solver_chat` the way `generate()`'s does, so `$eval(solver_chat = )` swaps the model powering the agent. Chats with tools registered are rejected rather than silently stripped, since the agent brings its own.
simonpcouch
commented
Jul 27, 2026
Comment on lines
+11
to
+12
| claude_code_input <- | ||
| "What is the capital of France? Reply with just the city name." |
Collaborator
Author
There was a problem hiding this comment.
Just inline this in the tests so that it's easier to run them one-by-one.
|
|
||
| These solvers run the agent's command line interface in a Docker sandbox via Python Inspect (resolved automatically with reticulate—you'll need the reticulate package installed and a running Docker daemon). The agent's transcript is read back into ellmer Chat objects and logged as usual. | ||
|
|
||
| Each sample gets its own throwaway container, built from Inspect's default image unless you supply a `Dockerfile` or `compose.yaml`. For a dataset like `are`, whose questions are self-contained, that empty workspace is all the agent needs; to evaluate an agent on a repository, build it into the image and point the agent at it with `cwd`. |
Collaborator
Author
There was a problem hiding this comment.
Could we write out what cwd is an argument to? e.g. claude_code(cwd)?
Moves the Claude Code log path helper alongside the other example fixtures in helper-.R and inlines the sample input, so each test in test-solver-agent.R can be run on its own.
Inspect's plain display prints a status line per sample, which piles up over a long eval, and its interactive displays assume they own the terminal. Set display = "none" and register a hook that reports sample and event counts back to a cli progress bar instead. Also disables Inspect's control server, which nothing here attaches to and whose bind failures surface as warnings.
Inspect resolves the model itself, so an ellmer provider only works if
Inspect knows it by name and reads its credentials from the same place.
Several providers didn't translate (e.g. Vertex needs `google/vertex`,
Azure and OpenAI-compatible chats can't translate at all), so the model
string now comes from an allowlist and unsupported providers error early.
Also validates the chat before the Docker preflight rather than after,
runs that preflight for configured sandboxes like `c("docker",
"compose.yaml")`, and rejects `eval()` arguments that can drop samples
from the log, which the solver needs one of per input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New
claude_code()andcodex()solvers evaluate the Claude Code and Codex coding agents on a task's dataset, so the same dataset and R scorer can grade both your own agent harness and an off-the-shelf coding agent. The agents run in a Docker sandbox via Python Inspect's inspect_swe package (resolved automatically with reticulate); their transcripts are read back into ellmer Chat objects for scoring and logging.