Can an AI Agent Design a Chip? Building a Benchmark That Says No to Shortcuts

01Why chip design

Correctness is precise and checkable. An instruction set defines exactly what every instruction does, including overflow, alignment and faults, so no grader has to judge prose.

Correctness is also not enough, and everything else is measurable. A correct design still has to close timing, fit an area budget and spend little energy on real workloads, and open tools can measure all of that. That yields a continuous score with real tradeoffs: a deeper pipeline may run faster but cost area; a cache may save memory stalls but burn energy.

And the work is long. A competent attempt models options, writes and verifies RTL, iterates on physical results and decides when to stop, over hours under a fixed budget. That tests planning and follow-through as much as RTL skill.

02The tasks

Table 1. Task catalog as of 8 October 2026. Private counts are at the calibration seed and include the held-out kernels.¹

Every task leaves the microarchitecture open. The contracts say what the hardware must compute, never how many pipeline stages, lanes or execution units to build. The GPU contract defines logical lanes, and the candidate decides how many physical lanes exist; the NoC contract prescribes no topology or routing algorithm. The NoC branch already has two very different fixtures: an input-queued crossbar (7,656 µm², 1.545 ns, reward 0.223) and a mesh (12,381 µm², 0.982 ns, reward 0.113).²

The planned list is organized by what the agent may change (only the design, also the ISA, or the whole architecture) and by size (block, cluster, SoC). All five merged tasks sit in the smallest cell: fixed ISA, single block.

03How an episode works

Each task is defined by three things. An architecture contract in XML owns state, memory interfaces, reset and termination. A separate ISA contract owns encodings and exact instruction effects. An independent Python reference, the Untimed Reference, executes programs and returns the expected architectural result with no notion of clocks.

<instruction mnemonic="ADD" opcode="0x01" format="r3" role="compute">
  <semantics>rd = u32(rs1 + rs2).</semantics>
</instruction>

An entry from the CPU ISA contract. ADD is opcode 0x01 in the three-register format, and its semantics say rd gets rs1 + rs2 wrapped to 32 bits. The Untimed Reference executes this line directly, and the agent's RTL must match it.

Encoding metadata is shared with assemblers and decoders, but the semantic text is never executed; the reference implements it independently.

The agent works in a container with no network access, the contracts read-only under /public and a writable /workspace. The image includes Python, C++, gem5, Icarus, Verilator, Yosys, OpenROAD and KLayout.

  Explore               Implement             Measure                 Submit
+----------------+    +----------------+    +--------------------+    +------------------+
| Timing Models  |--->| RTL + Model    |--->| public_check       |--->| submit           |
| model-only     |    | Spec + Graph   |    | Netlist, Lockstep, |    | Latest Accepted  |
| cycle checks   |    |                |    | Formal; optional   |    | -> Private       |
+----------------+    +----------------+    | Route, STA, Power  |    |    Grading       |
        ^                                   +---------+----------+    +------------------+
        |                  Iterate                    |
        +---------------------------------------------+

Figure 0. One episode, from contract to graded checkpoint.

The agent must first write an executable Candidate Timing Model in Python or C++ that predicts its microarchitecture's cycles and hardware counters. It can compare several on the public workloads before any RTL exists. Then it writes RTL and calls public_check. The harness synthesizes an immutable snapshot into a canonical netlist, simulates it in a trusted testbench, compares results with the reference, and replays the agent's timing model one clock edge at a time against the recorded RTL trace. Pins, cycle counts and counters must all agree, so an agent cannot just claim its design is fast.

With physical scoring configured, a passing design can go through placement, clock-tree synthesis, routing, parasitic extraction and sign-off timing on ASAP7 cells, then a workload replay on the routed netlist for energy. Finally the agent must call submit. At the deadline its containers are torn down and the latest accepted checkpoint goes to Private Grading with a seed it never saw. Unsubmitted work does not count, and there is no hidden best-of selection.

04Scoring

cost   = (T / 1 µs) × (E / 1 nJ) × (A / 10,000 µm²)
reward = 1 / (1 + cost)

T is mean execution time across weighted workloads, E is mean execution energy and A is placed area. The scales are units, not targets from a reference design, so we never had to freeze a "good" answer. With equal exponents, halving execution time exactly compensates for doubling area.

The submission declares its own clock period, and that one period drives simulation, the timing model, synthesis, routing, power replay and scored time. If routed timing fails at the declared period, the design fails; grading never falls back to a slower clock. Memory latency is stated in picoseconds (2,500 ps on the CPU task) and converted to whole cycles at the declared clock, so a faster clock adds wait states rather than making memory faster.

Table 2. The current workload_product_v5 calibrations. They show the pipeline works; they are not targets.⁴

05Making the score hard to game

Agents find the cheapest path to a passing check, so much of the project went into writing dishonest designs and making sure the grader catches them. Every row below is a cheat we built and ran through the real checker.

Table 3. Cheats and what catches them.⁵

Some details matter more than they look.

• Random programs are biased toward hazards: load-use, writes to r0, boundary immediates and store-load aliasing. About 20% plan a fault, and unreachable words are filled with illegal opcodes and stray jumps.
• Held-out kernels are normalized by the reference's retired-instruction count. Each CPU instance retires 10,000 to 16,000 instructions, and the size of the draw does not move the score.
• Retirement ports are simulation-only. The physical flow strips them and their exclusive logic, so honest tracing costs no area.
• Grading seeds are per campaign. A review found that every preset used one constant seed, so anyone who learned it could enumerate the private tier from the public generator. Seeds now derive from an operator secret, and a test scans every written file for both.
• One leak came from our own tooling. The dashboard served captured task sources, which would have exposed the hidden kernel generator. It is fixed and covered by a test.

06Teams vs one agent, with run traces

The benchmark runs in two modes with the same deadline, resources and sign-off rules. A single generalist can do everything. Or a team of seven roles, each with its own conversation and workspace, splits the work:

The team coordinates through versioned releases, each published with trusted evidence. A performance study needs at least two passing model checks from distinct models; an RTL candidate needs a passing public_check. Sign-off is derived, not declared: every required release must be current and its evidence must match the exact netlist being submitted. Roles activate only when their input releases exist, and an open change request from any role blocks submission until it is resolved or the VP waives it. The VP's prompt is blunt about the main risk: "an idle perf or microarch is the most common way a team stalls."

Results

Table 4. Real-model episodes of the same model, one per arm, 5-hour budget.⁶

On CPU, the team found a faster clock (−4.0%) and slightly lower energy at almost the same area, for a 3.6% higher reward. It paid for that with 6.3× the model responses and 5.5× the input tokens. Before crashing, the team had also passed a physical check on an unsubmitted three-line fetch-cache variant at 0.1858. It was graded on its 69-minute baseline because the VP had locked that in first, which is exactly what its prompt asks for.

For GPU SIMT there is no retained real-model comparison between a single agent and a team. Single-agent and team GPU SIMT episodes were scheduled in a long-horizon campaign, but our records keep only their routing measurements, without agent mode, score or transcript. Routing records cannot establish team or single-agent scores, so we report none. What survives is still informative about the designs themselves: those episodes produced routed designs from 13,379 to 18,976 µm², all larger than our 12,683 µm² fixture, and three of them were killed by the routing memory cap (Figure 6).

Run traces

We do not have full turn-by-turn transcripts of these real-model runs on hand, so the traces below come from the closest retained records: an audit of a team run, grader output, harness logs and scripted episodes run through the real harness. Each is labeled.

Trace 1: a handoff that failed (CPU team, earlier functional-only audit). A shorter 30-minute team run with no physical grading lost every case to one missed handoff. From the audit of its shared room:

seq 2872  Perf: the model emits fetch-valid during reset; the latest RTL uses an IDLE state to suppress it.
event     Perf completes its assignment.
seq 2915  VP asks for a model patch in chat, without reactivating Perf.
seq 2924  VP integrates the stale model with the newer RTL and submits.
grader    timing mismatch at cycle 0 tick: imem_req_valid rtl=0, model=1   (0/27 cases)

The right role found the bug and reported it in the room. It was lost because a chat message does not wake a finished specialist. The same run spent 19 yield calls and 16 unchanged polls waiting, about 469K input tokens.⁷ This is why sign-off is now derived from evidence that matches the exact submitted netlist, rather than from a role saying it is done.

Trace 2: the team idles during sign-off (CPU team pair). The whole team sat idle through a 48-minute physical check, about 16% of its wall time. The VP and workflow prompts now keep three design generations in flight: one in sign-off, one being built, one being modeled.

Trace 3: exploration before RTL (GPU SIMT, scripted agent). The scripted fixture measures two timing models on all 66 public cases before writing RTL:

   4.7 s  public_model_check /workspace/models/serial
 255.9 s  -> passed, 66 cases, model_id serialized-gpu-simt, cycles 36072
 256.9 s  public_model_check /workspace/models/parallel
 493.4 s  -> passed, 66 cases, model_id parallel-gpu-simt, cycles 31176
 495.7 s  public_check include_physical=False
1022.4 s  -> passed, 66/66, formal passed, cycles 36072
1022.5 s  submit -> accepted

Trace 4: the formal gate catches what simulation missed (GPU SIMT, scripted mutant). The mutant withdraws a stalled instruction fetch after one clock edge. It passes all 66 public cases, and later all 71 private ones. The bounded formal gate fails it on every check:

SBY [run] engine_0 (abc bmc3) returned FAIL
SBY [run] failed assertion ... gpu_formal_protocol_progress_protocol_terminal.sv:325 step 17
SBY [run] DONE (FAIL, rc=2)

The violated property, from the generated harness:

// Stalled requests keep their complete payload until accepted.
always @* if (live && past_imem_stall) begin
    assert (imem_req_valid);
    assert (imem_req_addr == past_imem_addr);
end

Final record: passed_cases = 71, formal.outcome = design_failed, score.value = 0.0. The two barrier checks also failed, at steps 243 and 123, on the property that every warp eventually reaches a terminal state. Each check took 25 to 29 seconds.⁸

After submitting, the scripted agent kept replying without tool calls, and the coordinator answered each reply with a continuation notice:

Clock: 13.5 of 30.0 min remain; deadline 2026-10-02T20:38:55Z.
Accepted submission (graded unless replaced): checkpoint 8208…, no public measurement of that checkpoint.
Observed trusted job durations: public_check median 513 s, max 513 s; public_model_check median 232 s, max 239 s.

A reply without tool calls never ends a timed episode; the agent is told how much time is left and what a check costs.

Trace 5: timing closure on the reference fixtures (harness, not an agent). The physical-design lesson came from our own fixtures. Declaring a design at its earlier achieved clock failed sign-off: the CPU at 1,294 ps by 2.65 ps of setup slack, the GPU at 1,406 ps by 40.9 ps. Timing repair stopped at zero estimated slack, and the estimate was off by −39 to +30 ps after global routing.

Repair now closes to 50 ps of estimated margin, sign-off still requires only nonnegative extracted slack, and the defaults declare periods about 5% above their first routes.

07Add a task

We would like the task list to grow faster than we can grow it ourselves. A good task has precise observable behavior, room for real microarchitectural choices, workloads that pull those choices in different directions, and adversarial cases.

tasks/<id>/
  task.toml                identity, contract paths, hooks, profiles, public and hidden files
  prompt.md                task-specific instructions captured into episodes
  arch/arch.xml            state, interfaces, memory, reset, termination, counters
  arch/isa.xml             encodings and exact effects
  reference/reference.py   independent Untimed Reference
  workloads/cases.py       seeded public cases
  workloads/heldout.py     private scored kernels (optional)
  dv/                      RTL contract, trusted execution and audit, model/RTL lockstep
  profiles/*.toml          memory service, workload weights, [scoring] once calibrated
  tests/                   task tests, mutants that must fail, coverage map
tests/fixtures/<id>_submission/   reference RTL, timing model, spec, resource graph

1. Contracts and reference. Write the architecture and ISA XML and an independent Python reference, then validate them:

sh uv sync --locked --extra shared uv run oai-bench validate tasks/<id> --seed 0 uv run oai-bench run-reference tasks/<id> --seed 0 --output work/runs/<id>-reference uv run oai-bench verify-run work/runs/<id>-reference uv run oai-bench view tasks/<id>

1. Fixture and checker. Build a plain reference RTL and timing model, then run them through the real evaluator:

sh uv run oai-bench models run tasks/<id> tests/fixtures/<id>_submission \ --image "$OAI_BENCH_IMAGE" --output work/model-runs/<id> uv run oai-bench check-rtl tasks/<id> tests/fixtures/<id>_submission \ --image "$OAI_BENCH_IMAGE" --output work/runs/<id>-rtl uv run oai-bench verify-evaluation work/runs/<id>-rtl

1. Mutants. Add deliberately wrong RTL and models (arithmetic, timing, counters, protocol) and show that each fails through the real checker. Map every requirement to a test in tests/coverage.md.

2. Calibration, for a physical score. Add a [power] replay hook and a [scoring] profile. Then run a local smoke case, a physical probe at the declared clock, and a full calibration over every graded case. Only a complete calibration with routed workload power, nonnegative slack and confirmed cleanup qualifies a profile.

The task-authoring prompts in skills/prompts/ walk through ideation, generation and an end-to-end episode, and docs/README.md is the documentation entry point.

08What we learned

Most failures are capacity, not correctness. Three GPU SIMT episode designs were killed at an 8 GiB routing memory cap. One CPU calibration ran for 4.5 hours, then failed exporting 1.3 GB of evidence past a 1 GiB bound. Seven concurrent power sessions under one memory cap got one of them killed. We treat these as unavailable infrastructure, never as a zero for the design, because confusing the two would quietly punish large designs.

Calibration is slow, so tune the flow first. A full calibration runs every graded case and the whole physical flow. Detailed routing dominates the routed implementation, and workload power dominates the rest. Every CPU repair call reached its best result within 500 attempts, so moving from 2,000 attempts on two threads to 500 on eight cut routing time from 2,740 to 923 seconds with a byte-identical netlist. A power-measurement cache later made activity replay 2.7 to 3.3 times faster with bit-identical results.

Flakiness lives in the harness. Routing was reproducible: the CPU netlist was byte-identical at two and eight threads. The scaffolding around it was less reliable. Simulator output truncated under host load, a test run ignored SIGTERM and left orphaned simulators, an operating-system auto-update restarted a live episode about 20 minutes in, and a team status message outgrew its size bound and ended a run. Each became a fix and a test.

Write down why each prompt rule exists. Every behavior rule in our prompts maps to an observed failure:

Table 5. From the prompt behavior-design table.¹⁰

09What's next

• More episodes, calibrated. The single-agent vs team question needs several calibrated episodes per arm on both CPU and GPU SIMT, with full evidence retained.
• Physical scores for the accelerators. Tensor, RT and Vector Core need calibrated objectives before they can be ranked.
• NoC. Finishing and merging the task, including its fixed-die floorplan and per-task flow capacity.
• A formal verification role. A planned eighth role focused on unbounded proofs. The trusted reference owns the obligations and assumptions; the agent supplies only lemmas, which must be proven, never assumed. Formal stays a gate, never a reward term. Local dry runs behave as intended: an honest lemma passes at depth 10 in 17 seconds, and a false one fails with a counterexample at step 4.
• Provisional limits. The 200 ps minimum clock period and the area and window limits are still provisional. At very fast clocks, a sequential design can exceed the cycle watchdog on held-out kernels.

So far, most of the work in this project has gone into a grader that measures real hardware and refuses to be fooled. If you know a block well, we would like to help you turn it into a task.

Notes