Two AIs Read the Same Missouri Statutes. They Disagreed on 43% of Key Numbers.

Somewhere in a statehouse IT shop, a piece of software is reading a statute before any human has. Across town, a second piece of software, built by a different team, reads the same statute and reaches a different answer about whether a number is buried in the text. Both pipelines are independently written. On numeric-threshold clauses in Missouri statutes, they disagreed 43% of the time.

That number, a 0.43 false-negative rate on whether a numeric threshold is present at all, is the load-bearing fact in a new arXiv preprint by Surya Saka titled "When Can a Machine Trust a Statute? A Survival Certificate for Machine-Extracted Legal Logic" (abstract). The paper treats the disagreement between two pipelines, not the output of either, as the only honest unit of trust in a machine-read statute.

The mechanism is built around a standard tool from formal concept analysis called a Duquenne-Guigues implication basis, a minimal set of if-then rules whose consequences together cover everything the statute's text logically implies. Saka's pipeline first extracts candidate if-then rules from raw statutory prose, then audits each one under controlled stress. For every rule, the paper replays the inter-extractor disagreement against that basis across 1,000 Monte Carlo trials, simulated passes in which the pipeline is deliberately perturbed to see when the rule still holds. A rule earns what the authors call a "survival certificate" only when a one-sided Wilson 95% lower bound, a conservative confidence floor on how often the rule still holds under noise, reaches 0.95. Each certified rule is shipped with the premise spans it depends on, the exact phrases in the statute that ground it, and a minimal counterexample, the smallest possible case that breaks it (full paper).

The audit was preregistered and tested against two real legal corpora: 29,365 sections of Missouri statute and 502 sections of India's central Acts. A held-out gate, sections the audit never saw during training, passed both. Ten statute families across 7 Missouri Titles matched exactly; sixteen families across 11 Titles matched within a 5% tolerance band. A preregistered gate on real US state law is rare; passing one twice is rarer.

The second test is the harder one. The authors asked what happens when the pipeline meets an error pattern it never saw during calibration, a phenomenon they call calibration-rate transfer: an error model learned in one statute family bleeding into another. Under one realistic, globally deployed error model, 93.2% of held-out chapters fell below the informativeness floor: the certificate could not, in good faith, vouch for them. A 2x2 factorial analysis pinned the collapse on the transfer itself, not on chapter selection. The paper's own verdict: the certificate is "usable but fragile." A buyer who deploys it chapter-by-chapter, or who tolerates a band of error, can use it. A buyer who assumes it generalizes without recalibration cannot.

The audit trail released alongside the paper includes a structural feature worth flagging: the authors explicitly retract one of their own claims. A paper that ships a known retraction in its own public record has decided trust is built by showing the seam, not by hiding it. That choice sits awkwardly next to a field where vendor pipelines are routinely marketed as "validated" without an inter-extractor gate at all. The code, data, and preregistration are to be released on GitHub, which makes the retraction auditable rather than rhetorical.

The practical read for a legal-tech buyer is narrow. Ask any vendor selling a statute-extraction product to publish, on a held-out set of statutes the buyer picks, the inter-extractor agreement rate on numeric-threshold clauses. A figure well below 43% is a signal of careful design. A refusal to publish the figure is a signal of something else. The certificate, in Saka's own framing, is a yardstick, not a shield, and the paper's own data is what makes that legible.