The Vision Thing: How Roboflow Is Making the Physical World Programmable

Computer vision has a perception problem. While language models have captured the world's attention with ChatGPT moments and reasoning breakthroughs, a quieter revolution has been unfolding in factories, farms, and sports stadiums — one where machines are learning to see, but still cannot see everything, everywhere, all at once.

Joseph Nelson, CEO of Roboflow, has a clear-eyed frame for where computer vision stands today: roughly where language capabilities were three years ago with GPT-4. The Vision Transformer arrived around 2020, about three years after the original Transformer paper. GPT-4 landed in 2023. The implication is not that progress has stalled — it is that the hard part is still ahead.

"We have a ways to go before foundation models will be able to do it all," Nelson said in a recent episode of The Cognitive Revolution podcast. The reason, he explained, comes down to a fundamental asymmetry between language and vision. "The world's much bigger than just language. The amount of distinct scenes in a day is more diverse than the number of unique words you probably read in a day." Language is a human construct, optimized for being understood. The visual world is not. A manufacturing defect on a rivet. A crack in a solar panel. A tennis ball at Wimbledon. The heterogeneity of visual scenes is what makes general-purpose vision so hard, and what makes Roboflow's position in the ecosystem so interesting.

The failures are revealing. Roboflow maintains visioncheckup.com, a site that tracks what multimodal frontier models still cannot do consistently: spatial reasoning, precision measurement, and grounding — the ability to precisely locate and segment objects in an image. Nelson described the site's purpose and the pattern it reveals: capabilities are improving on benchmarks, but frontier models are not yet reliable across all the tasks people attempt, including Roboflow's own tests. The site functions as a permanent benchmark for AI vision, updated as new failure cases surface.

Grounding is a particular pain point. Nelson described asking a frontier model to identify the starting position in an ARC AGI puzzle and getting strange results. When he simplified the problem to text — describing the puzzle grid as a string of characters — the model performed better. If it just thinks about it like text, if it does not also have to think about where in the puzzle that issue is, it almost does better. The labs, he suspects, are not particularly interested in solving the segmentation problem for its own sake. If segmentation is just a tool in a chain of reasoning, it becomes an engineering problem rather than a research priority.

There is also a reproducibility problem that AI practitioners rarely discuss publicly. Nelson recounted trying to use Gemini 3 to auto-label a dataset and finding that results were inconsistent, not because of his prompts, but because the model itself does not always produce the same output for the same input. He described a pattern where failure cases he reported to the team would not reproduce on a second run, a phenomenon he attributed not to his use of the models but to the model's own non-determinism.

A Million Engineers, Half the Fortune 100

Roboflow positions itself as the infrastructure layer between foundation models and production deployment. Its platform now supports more than a million engineers downloading open-source models every 30 days, and half the Fortune 100 as customers. That scale gives Nelson a bird's-eye view of where computer vision is actually working in the wild — and where it is still failing.

The cloud frontier models can do remarkable things — but according to Nelson, they can take 40 seconds to do them, and they require a data center. Vision is most valuable precisely where there is no human already watching: a manufacturing line, a microscope slide, a sports broadcast booth. You need fast reaction times in addition to large-scale reasoning, Nelson said.

The implication is an 18-month gap between what a frontier cloud model can do and what can actually run at the edge — on a Jetson Orin, an iPhone, or a microcontroller with an NPU. That is not a failure of progress; it is the normal lag between capability and productization. And it is exactly the problem Roboflow has built its business around solving.

The core offering is a pipeline: download an open-source model, fine-tune it on your proprietary data, and deploy a task-specific specialist to production. The specialist does not need to see everything — it needs to see the things that matter in your factory, your warehouse, your stadium. And it needs to see them now.

RF-DETR: A Paper That Ships

The most technically interesting thing Roboflow has built is RF-DETR, a real-time detection model released in March 2025. Unlike most research outputs, this one came with working code and an Apache 2.0 license. In November 2025, it was accepted to ICLR 2026.

The paper, authored by Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, and Neehar Peri, describes a neural architecture search approach that trains thousands of network configurations simultaneously using weight sharing — within a single training run. The result is not one model but a Pareto frontier: a family of models trading off accuracy against latency. Pick your budget; Roboflow gives you the optimal architecture for that budget on your dataset.

RF-DETR is built on Meta's DINOv2 backbone, a strong visual feature extractor. Roboflow took those features, ran NAS to find the right detection head architecture for specific target domains, and shipped a family of models that outperform prior real-time detectors on COCO. RF-DETR nano hits 48.0 AP, beating D-FINE nano by 5.3 AP at similar latency. RF-DETR 2x-large is the first real-time detector to exceed 60 AP on COCO, and it runs 20 times faster than GroundingDINO tiny while outperforming it on Roboflow100-VL.

The productization matters. Roboflow has made the NAS process available to any user on its platform — not just researchers with GPU clusters. Anyone can now run it on their own dataset and come out the other end with an N-of-1 model optimized specifically for their problem, Nelson said.

YOLO26, released by Ultralytics in January 2026, represents another branch of the edge-optimization tree. It removes Non-Maximum Suppression — a post-processing step that adds latency — and uses a MuSGD optimizer borrowing from Kimi K2's LLM training techniques. YOLO26-N is 43 percent faster on CPU than YOLO11-N. It does not beat RF-DETR on accuracy, but it is a different trade-off point on the Pareto frontier, and Roboflow supports both.

What the Data Says About ROI

Roboflow's Vision AI Trends: 2026 Report, released in February, analyzed 200,000 projects on its platform. The headline finding is that vision AI has moved from observation to high-stakes decision-making.

In manufacturing, 68 percent of projects are focused on closed-loop defect reduction — not just flagging problems, but feeding the signal back into the production process. In healthcare, 66 percent of projects are imaging and diagnostics for AI-assisted clinical decision support. In energy, 32 percent are inspecting critical infrastructure for predictive maintenance.

The report contains a counterintuitive insight: perfect accuracy is not required for ROI. In industrial settings, a model with only 50 percent accuracy can still save millions by catching defects that previously went unnoticed. This reframes the deployment calculus. The question is not "does this model perform perfectly?" but "does it catch enough of what we were missing?"

The open-source ecosystem, Nelson noted, is largely dependent on Meta — for model weights, for training infrastructure, for the open-vocabulary detection capabilities that Roboflow and others build on. If Meta's new AI leadership changes priorities, he said, NVIDIA will fill the gap. It is an honest assessment of the fragility of the current stack, and a reminder that infrastructure dependencies cut both ways.

The Next Three Years

Nelson's vision for where computer vision is headed includes world models, Vision-Language-Action models in robotics, inference-time scaling for vision, and wearables now selling millions of units per year. The common thread is that AI needs to get out of screens and into the physical world to reach its full potential.

Visual AI, visual understanding, and at least even that part of multi-agent reasoning is going to be bigger and more important than just language, he said. Whether that is the ChatGPT moment for vision depends on what you mean by moment. The capability is coming. The infrastructure to deploy it is here. The fat tail of reality is what remains.