Reference Oracle
A reference solution that demonstrates a valid way to complete the task. Verifier tests check outcomes rather than requiring agents to copy the Oracle.
Terminal Bench task foundry
Benchmarks, forged to order.
We build benchmark tasks for AI agents in the Terminal-Bench format, with reference solutions, Docker environments, verifier tests, and task-validation checks. Choose a category or commission a custom task set.
Illustrative task package · not a live run
01 Reference Oracle02 Docker test suite03 Validation suite
What we forge
Every task ships complete
A benchmark task is only as useful as its ground truth and its harness. We deliver all three pieces, verified end to end.
Docker test suites
Containerized environments and test suites designed for repeatable runs. Runtime dependencies and resource requirements are agreed for each delivery.
Validation suite
Checks on task instructions, reference-solution execution, and verifier behavior to help identify ambiguous or broken tasks before delivery.
The format
Terminal Bench, from prompt to verdict
A task pairs instructions and an execution environment with verifier tests. The Oracle is a reference solution used to validate the task; the verifier checks an agent’s results. The target Terminal-Bench or Harbor format version is agreed for each order.
1
Prompt
A natural-language task with clear goals and constraints.
2
Agent
Your agent works inside an isolated container.
3
Terminal
The agent uses tools and modifies the environment.
4
Verifier
Tests inspect the resulting files and behavior.
5
Result
The harness records the verifier’s result.
How an order becomes a benchmark
From spec to shipped task
01
Spec
Share the category and constraints you need to evaluate.
02
Forge
We author the tasks and their environments.
03
Oracle
We build and verify the reference solution.
04
Validate
Docker and validation suites prove solvability.
05
Deliver
Packaged, documented, ready for your harness.
Featured
Categories in the catalog
Explore illustrative task categories. Availability and delivery scope will be confirmed with your quote.
Terminal BenchExample only
Code debugging & fixes
Repository-level debugging tasks with failing tests and a reference fix.
Terminal BenchExample only
Repository feature implementation
Implement a well-scoped feature in an existing codebase against hidden tests.
Terminal BenchExample only
Shell & CLI automation
Terminal-native tasks: scripting, orchestration, and environment wrangling.
Need a benchmark we don't have yet?
We forge custom categories to your spec — reference Oracle, Docker suites, validation, the whole loop.