The Robot Got Certified Safe. Six Months Later, Nobody Knows If It Still Is.
Nobody in the robotics industry can tell you who is responsible for checking whether a deployed robot is still safe six months after it was certified.
That is the gap a white paper from a panel at SAE World Congress, published May 11, put at the center of the embodied AI safety debate. The paper argued that embodied AI — robots that move through and act in the physical world — should be treated as a systems challenge, not a software challenge. Safe deployment depends not just on algorithms but on sensors, hardware robustness, operational design limits, human interaction, communications, cybersecurity, and organizational processes arXiv preprint. The sharper point: a robot that learns and adapts after installation may no longer be the system that was certified. The accountability question that follows is the part the industry has not answered.
Phil Koopman, a Carnegie Mellon researcher who has spent decades studying machine learning in life-critical systems, put the reliability gap in plain terms. In his reading of what the numbers actually demand in life-critical deployments, catastrophic failure rates need to reach roughly one in a billion per hour — a bar set not by any robot on the market today but by the mathematics of what acceptable risk looks like in systems where people work alongside machines Phil Koopman Substack. There is a wide gap between 99 percent accuracy in testing and that target. Those are not comparable numbers. A model that scores 99 percent accurate on a test set will still fail catastrophically often enough to be unacceptable in a factory where humans work alongside it.
The implication is not that robots are dangerous. It is that the certification framework used for static machinery does not map onto machines that learn after deployment.
The paper called for safety engineering to move away from single-metric optimization — minimizing net harm across all outcomes — toward multi-dimensional constraint satisfaction, meaning meeting multiple required harm thresholds simultaneously across different failure scenarios Phil Koopman Substack. The distinction has direct consequences for how a warehouse operator knows whether their fleet is still within safe operating bounds six months after install.
End-to-end AI architectures, which let a robot reason about tasks with greater computational efficiency by connecting perception directly to action, are becoming the standard approach in commercial humanoid development. Companies building general-purpose robots have broadly moved toward this architecture. The trade-off is a higher investment in post-development validation and testing because the system's decision-making is less interpretable — you cannot always trace a given action back to a specific rule arXiv preprint. When the system fails, you may not immediately know why.
Model performance metrics — accuracy percentages, benchmark scores, task-completion rates — are insufficient indicators of deployment readiness on their own. A model can perform well in testing while still failing to generalize to rare scenarios, degraded sensors, novel environments, or unexpected user behavior arXiv preprint. The numbers that cleared the certification are not the numbers that matter on a Tuesday afternoon in a live facility.
The question of who owns the gap between certification and deployment is where the story gets commercially sensitive. The SAE panel white paper identified this as an open problem: the field has not established what a post-deployment reassessment mechanism would look like, or who would trigger one when a robot's behavior has materially changed arXiv preprint. Several companies point to their initial certifications and monitoring dashboards. None have published what would amount to a lifecycle accountability standard — and the panel paper frames that absence as an open problem the industry is still working through, not a documented failure across every robotics shop.
The white paper itself acknowledged it was a summary of panel discussion, not a research result. Koopman's billion-to-one failure rate is his own quantitative framing, drawn from his experience in life-critical ML systems, not a figure the panel derived from its discussion Phil Koopman Substack. And the industry has been naming lifecycle safety as an open problem for years without a standard emerging. That is worth noting plainly: this is a recognized gap with no agreed solution in sight, not a crisis in progress.
Koopman's bottom line, and the paper's, is that trust will be earned through performance that is observable, explainable, and repeatable under real operating conditions arXiv preprint. Not through a certificate obtained at commissioning.
The robot on your warehouse floor is not the same machine it was on delivery day. The question is whether the people who signed off on it are watching closely enough to know when it stops being safe.