Count the sessions, not the queries

Load balancers were built to count arrivals, because the web was built from short, stateless requests. Real-time AI is built from long, stateful conversations, and the arrival count is the wrong number.

The standard workload unit for two decades has been a query per second. A server that finishes a hundred short web requests in 50 milliseconds each looks twenty times busier than a server handling five requests that each turn into a 20-minute voice session. By the work each machine is actually carrying, the second server is heavier. The dashboard says the first server is hot; the queue says the second one is on fire. QPS (queries per second) and CPU (central processing unit) utilization are arrival and processing signals. Neither measures how many conversations a server has already committed to keeping alive.

This is the seam breaking under voice assistants and live chat agents as they move from demo to production. A box can look idle on CPU while it holds twenty silent sessions, then spike the moment its users start speaking at once. Operators watching only QPS will keep sending traffic to a server already promised to other people. Google's engineering blog frames the fix as a second signal: the count of active sessions a server has already taken on, not just the rate at which new ones are arriving.

The mechanism is portable. The next time a "real-time AI" outage reads like a capacity surprise, the question is whether the dashboard was counting arrivals or commitments.