AI agents ship on demos. A new index wants proof before they go live.

An AI agent that handles a cardiology triage demo without a hiccup can still fumble the first time it touches a real patient's chart, a real bank transfer, or a real regulator's complaint inbox. The industry has no shared yardstick for telling those near-misses apart from the wins. A new preprint argues that an agent's "capability," the score it posts on a benchmark or a behavioral test, is not the same thing as production readiness, and proposes a four-dimension index, the ProofAgent Index (PAI), that tries to make the gap auditable before an agent goes live.

The index is published as arXiv:2607.27677, with a code artifact under the same name. The team's central claim is short enough to quote: "Capability is therefore not production readiness." The reason, the paper argues, is that the same agent that behaves well on a benchmark can behave badly in production because the surrounding environment has changed, the data has drifted, or a regulator has rules the test never enforced. A benchmark measures the model. Production readiness measures the system.

PAI breaks the readiness question into four dimensions. Each one is meant to catch a failure that a benchmark misses.

Evaluation asks whether the agent can do the work at all, but under conditions that resemble the real deployment: the same data distributions, the same tools, the same failure costs. The paper argues that a high benchmark score, without an evaluation harness that mirrors production, tells you very little about what will happen on Tuesday morning.

Context asks how brittle the agent is to the information it gets. A retail-style support agent that pulls from a product catalog might score fine on a clean eval, then misroute a refund the day the catalog schema changes. PAI treats context engineering as a first-class part of readiness, not a tuning afterthought. Context here covers what the agent sees: prompts, retrieved documents, and tool inputs.

Compliance asks whether the agent operates inside the rules that apply to its domain. In healthcare, that is HIPAA-style access control, audit trails, and a paper trail a regulator could read. In finance, it is KYC, transaction logging, and a way to show that no decision was taken without the data the regulator expects. PAI's compliance dimension is where a benchmark cannot help at all; it has to be wired into the deployment.

Governance asks whether the evidence stays visible. A system can pass all three of the other dimensions once and then quietly drift back into a state that would have failed. PAI treats governance as the practice of keeping the evaluation, context, and compliance evidence continuously inspectable, not as a one-time certification.

The whole index ships with an open-source ProofAgent Harness, documented at proofagent.ai/harness/docs and distributed as the proofagent-harness package on PyPI. That artifact matters: it turns the framework from a paper-only proposal into something a deployer, reviewer, or auditor can run today. The paper's own validation is narrow, however. The abstract restricts empirical work to two domains: healthcare and finance. Any claim that PAI is ready to score agents in, say, customer support, legal, or industrial control would be a paper-side extrapolation, not a tested result.

The honest read is that PAI is a readiness index, not a certification regime. It does not certify an agent as safe. It does not bind a vendor to a standard. It does not replace a regulator. A vendor with a weak agent and a generous auditor can still game the four dimensions, and procurement officers who reward demos will keep rewarding demos. The framework gives deployers, reviewers, and end users a vocabulary and a harness for pushing back on "it worked in the demo" reasoning, and a place to attach evidence that the next failure will land on.

The team's next milestone, per the paper, is broader-domain validation. Until that lands, PAI is best treated as a draft rubric, not a yardstick. For anyone who has to ship an agent into a real workflow this quarter, that is still more usable than the current alternative, which is faith.