The new Auto Endpoints product exposes the serving code, GPU choice, and per running copy latency data that managed AI services for open weight models — freely downloadable, tunable AI model files — usually hide.
Modal is making a contrarian bet about how engineering teams want to consume open-weight AI models, the freely downloadable, tunable alternatives to closed systems like OpenAI's GPT-4. Its new Auto Endpoints product exposes the "inference stack," the software that actually runs a trained model on a GPU and serves predictions back to the application, in a way most managed AI services deliberately do not.
Where a typical OpenAI-compatible API gives you a URL and a bill, Modal's deploys a service where you can see (and tweak) which GPU the model is running on, which region it is deployed in, and which inference engine is generating tokens. A team can deploy a frontier open-weight model, in the example GLM-5.2 in FP8 (8-bit floating point, a memory-efficient number format that lets large models fit on cheaper hardware) from zai-org, with a single command, no sales call, and no waiting for procurement.
The pitch is built on three things Modal's launch post says it does not hide: the serving code itself (GPU selection, regionalization, inference engine flags); the metrics, including speculative-decoding acceptance length and per-replica engine-side token-latency quantiles, which are measurements of how fast individual model copies are producing tokens; and the deployment process. Speculative decoding is a speed-up trick where a small, cheap "draft" model proposes several tokens at once and the large model only verifies the ones it agrees with, instead of generating one token at a time.
The company frames this as "inference you own," and that framing is the first thing to read carefully. Inference, the act of running a trained AI model to produce an answer, is the part of the AI stack that happens after training. For open-weight models, the "ownership" question is genuinely contested: the upstream weight licensor (in this case, zai-org for GLM-5.2) still controls the model files, the GPU allocator still controls the hardware, and the procurement contract still controls what "ownership" legally means. Modal is exporting the inference control surface, the knobs and dials a customer can turn, but it is not dissolving those upstream vetoes.
That distinction matters because Modal is positioning against two existing options. The first is the black-box managed API offered by providers like Together, Fireworks, and OpenAI's own open-model endpoints, which hide the serving code behind a clean interface. The second is self-hosting on infrastructure like vLLM, SGLang, or TensorRT-LLM, which gives full control but requires the team to own autoscaling, engine tuning, observability, and on-call. Modal's bet is that a meaningful slice of the market wants the third option: visibility into the serving stack without the operational burden of running it.
The product is built on top of Modal's existing serverless AI infrastructure, which the company has been running for general AI workloads, and is documented in a separate endpoints guide. The technical claims are backed by a companion engineering post on speculative decoding, which explains how Modal tunes draft-model acceptance length and per-replica latency quantiles to make open-model serving cost-competitive with closed APIs. Modal cites Cognition, Decagon, Fathom, and DoorDash as customers already running on its infrastructure, though the company does not claim all of them are using Auto Endpoints specifically.
The launch has landed lightly so far: a Hacker News thread at the time of writing had six points and minimal discussion, which is too thin a sample to gauge engineer sentiment. That alone is a story. Open-weight inference is a category that is growing fast, but the audience that cares about which vendor serves it is still small.
The open question is whether "inference ownership" is a real buyer concern, or whether Modal is describing a problem most teams do not actually feel. Companies that build consumer products on top of open models rarely want to think about which inference engine is generating tokens, as long as latency and cost are good. Companies that build internal tools, or that operate in regulated industries where model provenance and serving auditability matter, may. Modal's commercial outcome will depend on which group is bigger, and on whether the visible control surface turns out to be a feature engineers reach for, or a level of detail they pay to be spared from.