70% Of Open-Source AI's Chip-Design Code Is Valid. Only Half Of It Works.

A new arXiv preprint reports open-source LLMs now produce syntactically valid Verilog, the hardware code used to design chips, on about 70% of a 50-task benchmark, but only about 52% of the generated designs actually pass simulation. The 18-point gap between "compiles" and "works" is the gap AST-repair tooling, not the model itself, is mostly closing.

The paper, "Benchmarking LLMs for Verilog Design Flows", is the work of Angshuman Chakravertty, Rahul Koshti, Buddhi Prakash Sharma, and Vinay Chamola across NMIMS Hyderabad, IIT Roorkee, and BITS Pilani. Posted as arXiv:2607.22759v1 in July 2026 and prepared for submission to IEEE Design & Test, it does not claim AI is replacing chip designers. It claims the open-source LLM stack has reached a checkpoint worth measuring.

Across 50 curated Verilog tasks spanning combinational, sequential, FSM, and mixed designs, raw LLM output had a 0% syntax validity rate. The pipeline, not a single model, lifted that to a 70.43% average. A companion figure: 51.8% of generated designs cleared simulation in Icarus Verilog, Verilator, and Yosys after the same pipeline ran.

That pair of numbers is the headline and the caveat in one breath. Syntax validity means the code parses. Simulation pass means the design behaves the way the testbench demands. The 18-point gap is the failure rate that a real chip-design flow still has to absorb, and the paper's own tooling does most of the bridging. The pipeline applies an AST repair stage, essentially a syntax-rewriting pass on the parsed tree, before simulation. Remove the repair, and the 70% number craters back toward 0%.

The benchmark itself is narrow by design. Three open-source models. 50 tasks. No commercial EDA baseline. No comparison to Cadence, Synopsys, or proprietary code-LLM tools. The paper is a reproducibility artifact, not a leaderboard, and the authors do not bill it as one. The institution mix matters for context: NMIMS Hyderabad, IIT Roorkee, and BITS Pilani are three Indian academic centers with active VLSI programs, and the work is a snapshot from research groups triaging what an open-source LLM can and cannot do on a chip-design task today.

The interesting failure mode is in what the 50 tasks are. Combinational designs, the kind that compute boolean functions on inputs without memory, generally clear the bar. Sequential designs, FSMs, and mixed-signal tasks take the simulation hit. The pattern matches the literature on code-LLM evaluation: the harder the state and timing logic, the faster the model loses the thread. An LLM that produces clean adders and muxes but stumbles on a 4-state FSM is not a 70% chip designer. It is a 70% syntax generator with a simulation ceiling around half its output.

The manuscript is not peer-reviewed. The headline numbers are author-reported on the authors' own platform. Independent replication on a different task suite would move the 70/52 pair up, down, or sideways, and there is no third-party run to anchor that yet. The honest read: a dated, citable checkpoint from three named research groups using a specific pipeline and a specific simulator stack, with the same 18-point gap between "valid" and "working" that anyone running AI-assisted RTL has run into.

For EDA teams triaging AI-assisted RTL pilots, the practical takeaway is simple. Treat "70% valid" as "70% past the syntax gate after repair." Treat "52% simulation pass" as "about half the candidates you would actually hand to a designer for review." The next paper worth watching is the one that closes that gap without leaning on the repair stage, or the one that reproduces the 70/52 pair on a different task suite. Until then, the open-source LLM stack for chip design is a measurable progress point, not a production workflow.