← Back to feed

Andrej Karpathy’s autoresearch pattern — propose, implement, measure, keep the wins — has been applied primarily to software: optimizing training loops, refining prompts, tuning gradient descent. Developer FeSens asked whether it generalises to a domain with a much harder feedback loop: CPU microarchitecture design.

The answer, published as auto-arch-tournament and picked up by IEEE Spectrum, is yes — with an important caveat about where the real work is.

Setup

The baseline is a textbook 5-stage in-order RV32IM pipeline in SystemVerilog: IF/ID/EX/MEM/WB, no branch predictor, no caches, no multi-issue. Locked at 2.23 CoreMark/MHz, 301 iter/s — the same methodology VexRiscv uses for its published benchmarks.

Each round, three agent slots run in parallel. An LLM proposes a microarchitectural hypothesis as structured YAML, a second LLM implements it in the rtl/ directory inside an isolated git worktree, and then a deterministic eval gate fires:

• riscv-formal: 53 symbolic BMC checks covering decode, traps, ordering, liveness, and M-extension correctness
• Verilator cosim: byte-identical RVFI trace against a Python ISS under ~22% random bus stalls
• nextpnr place-and-route: 3-seed median Fmax on a Gowin GW2A-LV18 FPGA (Tang Nano 20K)
• CoreMark CRC validation: the same 4 CRC values VexRiscv reports

Improvement merges. Regression or failure destroys the worktree. A diversity rotation forces each slot into a different category (micro_opt / structural / predictor / memory / extension) to prevent fixation.

Results

73 hypotheses. 10 accepted. 50 regressions. 9 formal or cosim failures. 4 placement failures. 9 hours 51 minutes wall-clock.

End state: 2.91 CoreMark/MHz, 577 iter/s, 199 MHz Fmax, 5,944 LUT4.

That is +91.9% CoreMark over the locked baseline. Compared against the human-optimised VexRiscv reference (2.57 CoreMark/MHz at 144 MHz), it lands +56% on CoreMark iter/sec (578 vs 370) and uses 40% fewer LUTs.

The step-function crosses the VexRiscv line at iteration 6 and stays above it. The gap between VexRiscv’s full no cache and linux balanced configurations — their next tier, with caches — is about 40 percentage points; the loop covered 13 of those in under 10 hours on a single FPGA target.

The 10 accepted optimisations, in order:

CoreMark/MHz	Fmax	Hypothesis
2.32	135 MHz	Backward-Branch Taken Predictor
2.35	138 MHz	IF Direct-Jump Predictor
2.35	160 MHz	Cold Multi-Cycle DIV/REM Unit
2.37	168 MHz	One-Deep Store Retirement Slot
2.37	179 MHz	Segmented RVFI Order Counter
2.89	164 MHz	Registered Lookahead I-Fetch Replay Predictor
2.89	175 MHz	Compressed Resetless I-Fetch Replay Tags
2.89	183 MHz	RTL-Only Hot/Cold ALU Opcode Split
2.91	199 MHz	Banked Registered I-Fetch Replay Predictor

The biggest single jump — moving DIV/REM off the single-cycle ALU path at iteration 3 — also halved the LUT count as a side effect. The agent did not know that would happen. It found out by doing it and watching the synthesiser.

The Actual Thesis

The project’s central argument is not about the loop. It is about the verifier.

Of 73 hypotheses, 63 were wrong. Several examples from the log illustrate why the eval gate is load-bearing:

The same DIV/REM idea arrived at round 1 as Move DIV/REM off the single-cycle ALU path and broke cosim before ever reaching the FPGA. Two hours later, the agent resubmitted it as Cold Multi-Cycle DIV/REM Unit — same idea, fixed implementation — and it became the breakthrough win. Without the cosim gate, the broken version would have shipped.

At round 24, after the peak of 577 iter/s was already locked, the agent proposed Registered Lookahead JALR Target Predictor. Fitness collapsed to 154 iter/s — a 73% regression. The comparison-against-baseline check caught it before it merged.

Two separate hypotheses attempted to write files outside the allowed rtl/** path. The path sandbox rejected both before any code was generated.

The conclusion: the agent loop is a commoditised component. Six months of moat at best. The formal verifier, the cosim harness, the schema validation, the budget check — those are the parts that make the loop safe to run unattended. Without them, the agent fixes one bug by introducing three.

What This Means for Hardware Design

The significance is domain transfer, not benchmark numbers. AI agents have become routine in software: SWE-bench Verified scores above 80% are now table stakes at the frontier. Hardware design involves a fundamentally different feedback cycle — synthesis, place-and-route, timing closure, formal property verification — and it has historically been insulated from this class of automation.

Auto-arch-tournament demonstrates that a general-purpose LLM, with no hardware-specific fine-tuning and no domain-specific training data beyond the SystemVerilog files in the repo, can navigate that feedback cycle autonomously and improve on a carefully hand-tuned baseline. The loop is the same. The verifier is the hard part.