XPerf Replays Real Agent Traffic to Benchmark LLM Serving

Agentic AI applications call a large language model, read the response, and decide what to ask next. That self-steering loop makes the traffic hitting an LLM serving system unpredictable from run to run, which is why conventional serving benchmarks misrepresent real load.

XPerf, an open-source benchmarking framework whose release is documented in a new arXiv paper, attacks that problem with fine-grained trace replay. The tool records traffic from real agentic applications and replays it reproducibly on different serving systems so engineers can compare them honestly. It ships with eight default workloads across coding, deep research, and Q&A, and produces detailed profiles of where the serving system and the underlying hardware actually bottleneck under that traffic.

The authors report that XPerf replays agentic workloads accurately, scales to larger serving systems, and helps debug serving issues that static-prompt benchmarks miss. They say the code will be released on GitHub, though the repository and license were not confirmed in the abstract.

Two limits are worth flagging. Trace replay is only as faithful as the traces captured, and eight default agentic workloads, while a useful sample, do not cover every pattern a serving team will see. The paper is also an unrefereed arXiv preprint with no third-party validation of replay fidelity yet, so any specific latency or throughput number from the study should be treated as author-reported until independently reproduced.