Catalog

Benchmark task categories

Explore task categories and example briefs. Confirmed stock, production status, and format versions will be published after review by our data team.

6 categories shown

Terminal BenchExample only

Code debugging & fixes

Repository-level debugging tasks with failing tests and a reference fix.

debuggingtestspython
Terminal BenchExample only

Repository feature implementation

Implement a well-scoped feature in an existing codebase against hidden tests.

featurestypescriptgo
Terminal BenchExample only

Shell & CLI automation

Terminal-native tasks: scripting, orchestration, and environment wrangling.

bashclilinux
Terminal BenchExample only

Data engineering & pipelines

Build and repair ETL/ELT pipelines with correctness checks against expected output.

sqlsparketl
Terminal BenchExample only

Web scraping & API integration

Extract, normalize, and integrate data from web pages and third-party APIs.

scrapinghttpapis
Terminal BenchExample only

Infrastructure & DevOps

CI, containers, and cloud-config tasks validated in ephemeral environments.

dockerci-cdiac