The cross-chip math problem that kept AI video codecs out of your phone

Your phone's AI chip and your laptop's AI chip compute the same neural network differently. Small differences in how each chip rounds numbers, accumulates partial sums, and scales intermediate values mean a video frame encoded by one chip can produce a stream the other chip literally cannot decode. The decoder reads entropy-coded bits, hits a value it does not expect, and the picture collapses.

That specific cross-chip numerical incompatibility is the reason learned video codecs have lived in research papers for the past decade while H.264, the codec finalized in 2003 that still compresses most of the streaming video watched today, has barely been challenged. A new open-source codec from Microsoft Research, called MLVC (Multi-platform Learned Video Codec), is the first published approach to sidestep that requirement. The paper on arXiv, the GitHub repository, and a Microsoft Tech Community announcement describe a neural-network-based video compression system that runs the same on Apple's M-series Neural Engine, Intel NPUs, and Qualcomm AI accelerators.

The reason this matters starts with the 14-year lag. AlexNet landed in 2012. Since then, neural networks have rewritten image generation, speech recognition, and machine translation. Video compression has not followed, because the codecs that ship in every phone, every video-call app, and every streaming service are still H.264, H.265/HEVC, and AV1: hand-engineered formats with dedicated hardware acceleration built into every modern chip. They win on power efficiency and on ecosystem inertia. Every operating system, every browser, and every social-media pipeline already knows how to read them.

The bottleneck for learned codecs is not compression quality. It is numerical agreement between encoder and decoder. A learned video codec uses one neural network to predict and compress each frame, then a second network on the receiving end to reconstruct it. For the reconstruction to work, the two networks have to agree on the bit-level output of every arithmetic operation. The author's post on r/MachineLearning, which discloses the author as one of the paper's researchers, walks through the trap. Apple M3 Neural Engine, for instance, simulates 8-bit integer math using 16-bit floating-point math because true INT8 silicon would be slower. Even on chips with real INT8 hardware, the vendor controls the rounding mode, the accumulator type, and the scale multiplication, so two chips from different vendors, or even two driver versions on the same chip, produce numerically different results. A learned codec that works on one phone will produce a bitstream the other phone's decoder cannot decode.

MLVC sidesteps the requirement for bit-identical math by transmitting the entropy model's scale parameters through the bitstream's hyperprior, a small set of extra numbers the codec sends alongside the video so the decoder can rebuild the picture. The encoder can run on any NPU with whatever numerical behavior that NPU happens to have. The decoder reads the scales it needs from the bitstream itself and uses them to align its reconstruction. The network no longer has to produce the same numbers down to the last bit. It just has to produce a picture close enough.

The performance claim, per the author's Reddit post, is that both encoder and decoder run at roughly 100 frames per second for 360p and 540p video on consumer NPUs. That figure is not independently verified and should be cross-checked against the paper's reported numbers before any production claim runs.

The strongest counterargument is the one the author concedes in plain language: traditional codecs still ship because they have hardware acceleration, are power-efficient, and have 14 years of battle-tested infrastructure. A learned codec running on an NPU draws more power, on average, than a hardware-accelerated H.264 encoder that has been hand-tuned for a decade. Microsoft has open-sourced the code. The ecosystem, the silicon support, and the power budget that determines whether MLVC ever shows up in a real streaming app have not followed.

The watch item, then, is whether any of the consumer-NPU vendors whose silicon is now the de facto deployment target, Apple, Intel, or Qualcomm, ships a learned codec in a production app rather than a benchmark demo. The paper and the code are public. The hardware-accelerated video pipeline, the OS-level decoder integration, and the battery-cost math that decides whether MLVC or any successor lands in a real streaming app are the work that follows, and none of those vendors has published a timeline for any of them.