Skip to content

Add TerminalBench 4.0 - #3366

Draft
lafrancef wants to merge 24 commits into
mainfrom
flafrance/terminal-bench-4
Draft

lafrancef wants to merge 24 commits into
mainfrom
flafrance/terminal-bench-4

Conversation

@lafrancef

Copy link
Copy Markdown

This is based on #3332

It adds TB4 through the Harbor integration and includes several sub-changes to make all tasks run and be observable.

WIP:

  • Some tasks still don't run healthily; implementation may have to change to make them run
  • Target harness is likely to change to align with AA methodology

Known limitations:

  • Some TB4 tasks require a GPU sandbox. Right now this is handled by allowing to pass different sandbox domains for CPU vs GPU. This is not general, but allows us to run the whole benchmark as one unit.

lafrancef and others added 24 commits September 12, 2026 23:32
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Run Compose-defined services through the sandbox API with explicit provider
capability validation and configurable networking, storage, and runtime
requirements. Preserve dependency and health-check ordering and clean up
collections safely, including short-lived services and shared providers.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit c5c15ee)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Serialize named service handles through ConnectableProvider and reconnect
without provisioning. Keep managed-volume cleanup with the creating server
and preserve provider close semantics for connected services.

Keep Compose-specific operations on the optional provider interfaces instead
of adding public forwarding methods to the sandbox facade.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit 91750e0)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Show a two-service Compose setup, Python execution, and cross-server
serialization with explicit lifecycle ownership. Link the guide from
the sandbox overview and describe configurable provider capabilities.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit d7af208)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Use docker image inspect as the default metadata command and document
pulling images for the sandbox platform before startup. Preserve image
startup and Docker-specific health and shell defaults without requiring
crane. Keep the inspection command configurable.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit 180b66a)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Read image manifests and configuration directly from registries, preserving platform selection and Docker health metadata. Use standalone official Compose for normalization without requiring a Docker daemon.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit 63b86b9)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Allow deployments to supply complete image defaults and skip registry lookup for those images.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit 3377b7c)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Remove the public metadata cache and image health, shell, and environment emulation. Skip registry access for explicit YAML entrypoints. Validate all 11 TB4 Compose gold tasks using their official images and verifiers.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit 0aebb36)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Remove image inspection, metadata options, and the ORAS dependency. Consume startup commands, working directories, and ports from YAML and reject missing commands before provisioning.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit e6555f7)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit 9286eed)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Mark the two synthetic basic-auth test URLs as false positives and clarify that x-sandbox is an optional Gym-specific extension.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit f97f4b7)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Use existing PyYAML and validate upstream-resolved inputs before provisioning. Document the supported input boundary and remove Compose executable configuration.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
(cherry picked from commit bb03de6)
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
…acts

Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants