Voice AI can catch a sigh and then ignore it

A new benchmark, Hear2Act, isolates the gap: giving voice AI the audio barely helps (15.3% vs 14.6%), but writing the heard concern as text first lifts success to 39.6%.

Voice AI today can hear the worry in a person's tone but almost never uses that information. The cleanest proof is in a new benchmark's numbers: feeding a voice-capable AI the raw audio alongside the transcript bumps the rate at which it picks the right action from 14.6% to 15.3%. Forcing the same model to first write down what the speaker's tone is signaling, then reason over that note, pushes the rate to 39.6%. The signal is in the audio stream. The decision layer just isn't listening to it.

Hear2Act, published on arXiv by an eight-author research team, tests 480 persona-grounded scenarios where a user sounds one way and means another: a calm "thanks" that is really a complaint, a cheerful "no problem" that hides frustration. In each case, what the user says out loud is fine on its own; what their tone adds is the part that should change what an assistant does next. The benchmark is built so the right answer is unambiguous once the hidden concern is known, and rollouts, 54,240 of them, count the cases where the assistant actually picks the right next action.

The clearest contrast is the 25-point lift between "give the model audio" and "make the model write the heard concern down as text before deciding." A 0.7-point lift from acoustic access versus a roughly 25-point lift from an explicit intermediate representation is a falsifier for the common story that voice AI just needs better audio pipelines. The audio is already arriving. The models are throwing it away at the decision step.

The authors frame it that way directly: prosody, the rhythm and tone of speech, matters most when the words alone are insufficient, and current audio-capable large language models can recover information from speech but do not reliably carry it into action without an explicit intermediate step. In benchmark terms, "intermediate representation" is just a short written note the model makes itself, something like "the speaker sounds frustrated even though they said they're fine." The paper's claim is that this step, not the audio pipeline, is where the gap lives.

To stress-test that read, the benchmark also runs an "explicit lexical feedback" arm, where the user's concern is stated out loud in the words. There, the contrast between audio and text inputs largely disappears: when the concern is in the words, tone is redundant, and the model does about as well with a transcript as with full audio. That is the right signature for a tone-only signal. It should only matter when the words leave a gap, and only fail to act when the system is asked to bridge that gap from tone alone.

Two audio-capable systems were tested on the audio path, Qwen2.5-Omni-7B and Qwen2-Audio-7B-Instruct, alongside five text-only models (Claude Opus 4.6, Kimi K2.5, DeepSeek-V3.2, GLM-5, and Qwen3-32B) for comparison. The text-only models, fed a transcript plus the same written concern note, hit 40.7%, essentially the ceiling. The voice-capable models, when given the audio and asked to produce that note themselves, land within a point of the ceiling. The pattern holds across both audio systems and across the persona set.

When the assistant writes down what it heard and reasons over the note, it does almost as well as one that was told the concern in words. When it goes from audio to action without that step, it is barely better than one that never heard anything at all. The architectural nudge is the same in either case: turn acoustic evidence into a written state the rest of the system already knows how to use. The product fix is not a new speech model; it is a small structural change that any team building on top of a current voice LLM can adopt without waiting for the next model generation.

The caveat is the same one the authors flag. Hear2Act is an arXiv preprint, not a peer-reviewed result, and only two audio-capable systems were exercised on the audio path. The contrast is large enough, and the lexical-feedback arm validates the read tightly enough, that the mechanism is worth taking seriously. Whether the same gap shows up in deployed voice assistants, with their mix of turn-taking, tool calls, and latency budgets, is a separate question the benchmark does not yet answer. The next milestone worth watching is whether any shipping voice product adopts the explicit intermediate-representation pattern, and whether the 25-point gap closes when it does.