AI Agents Are Getting Cheaper to Run — and More Expensive to Deploy

AI inference, the cost of actually running a model to answer a question or take an action, keeps getting cheaper. NVIDIA says its B200 chip can serve GPT-OSS-120B at around $0.02 per million tokens, roughly 15 times lower than the Hopper generation it replaces, as cited in a recent industry roundup (infotechlead). AMD pushed day-zero support for Qwen3.8 27B onto its Radeon AI PRO R9700 and Ryzen AI Max+ 395, with AMD-stated local inference benchmarks of up to 51.8 tokens per second on a single R9700 and 24.5 tokens per second on the Ryzen AI Max+ 395 (AMD blog, AMD R9700 product page). AWS is positioning its Trainium chip around the same cost-per-token race.

That price drop is real. It is also a trap.

A token is the small chunk of text a model reads or writes at each step. The cost-per-token number is what every vendor publishes, what every benchmark celebrates, and what most procurement teams track. But it measures the wrong thing for the workloads enterprises are actually deploying now: AI agents. An agent does not answer a question and stop. It plans the work, calls external APIs, invokes a model again, reads the result, revises, and self-evaluates before producing a final answer. One user request can fire a dozen model calls and burn through tens of thousands of tokens, not the few hundred a chatbot would use.

This is the agent cost paradox: the unit price is falling, but agents consume so many more units per business task that the line item an IT leader actually sees can keep climbing. The vendors are not lying about the price drop. They are just measuring a number that no longer tracks enterprise reality.

NVIDIA's B200 figure of about $0.02 per million tokens for GPT-OSS-120B is a vendor claim cited through the industry roundup, not yet a published NVIDIA primary, and the "15x lower than Hopper" comparison is the same vendor framing (infotechlead). AMD's day-zero support for Qwen3.8 across Ryzen AI Max, Radeon, and the Instinct MI300X, MI325X, and MI355X server line signals that AMD wants to compete on the same axis (AMD Instinct day-0 article). The R9700 workstation card sits around $1,299 (runaihome guide), cheap enough to put a local agent loop on a single workstation. Velocity Micro reproduces AMD's claim of up to 496% faster tokens-per-second throughput on large prompts for the R9700 versus a reference card, citing AMD RPW-495 benchmarks from May 2025 (Velocity Micro).

The 2026 Gartner cost-curve illustration the roundup cites has the right shape: falling per-token cost paired with greater model capability. But the curve shows the same paradox: more capability per token is exactly what invites agents to spend more tokens per task.

The shift is from cost per token to cost per completed task. A customer-support agent that resolves one ticket in eight model calls costs eight times what a single-shot chatbot would, even at the new lower per-token rate. A coding agent that reads a repo, plans a fix, writes a patch, and tests it can easily run hundreds of thousands of tokens before producing useful output. Tracking only the per-token line hides the multiplier.

AMD's SGLang and vLLM support on Instinct, plus ROCm day-zero coverage for Qwen3.8, are what decide whether cheaper silicon can hit the benchmark rates in production (AMD Instinct day-0 article). Software stack, kernel optimization, and utilization matter as much as the silicon price. Caching, kernel optimization, and model architecture compress tokens per task in ways a hardware price cut cannot. Utilization, the share of purchased inference capacity that is actually doing useful work, decides whether the new per-token economics survive contact with real agent workloads.

The next AI budget cycle should not be negotiated on cost-per-token. It should be negotiated on cost-per-completed-task, with a separate line for total inference spend, and a watched variable for tokens consumed per business workflow. Cheaper tokens are a feature, not a refund.