A new preprint argues document image search efficiency should preserve the visual evidence a query can still select rather than minimize token count, and reports a 16× ColPali storage cut.
A new preprint called SaMer reports compressing ColPali-style document-image search storage by 16× while improving retrieval accuracy, by merging image tokens around the objects they describe rather than pruning or pooling them.
ColPali and its siblings (ColQwen, ColPali-2) power a growing slice of visual retrieval: invoice search, scanned-document lookup, visual RAG over PDFs. They store one token embedding per image patch and score queries by late-interaction MaxSim, matching each query term to its nearest image patch. That design is precise, but it is expensive: per-image memory scales with patch count, and production deployments strain GPU and storage as the corpus grows. Standard fixes, including token pruning, average pooling, and attention-based reduction, cut costs by destroying exactly the fine-grained patch-level signals that MaxSim depends on.
SaMer's first move is to keep that fine-grained interface while collapsing the image-side token matrix down to K representative centroids, defaulting to K=64. Per the author project page, the merge step uses a soft weighting over a combined feature-similarity and 2D spatial distance: patches near each other in both feature space and on the page end up in the same centroid, so the surviving centroids are roughly object-shaped rather than grid-shaped.
The second move is training-time object-awareness. Per the released code, training injects Flickr30k-Entities bounding boxes as an instance-inconsistency penalty: tokens belonging to the same annotated object should merge into the same centroid, and tokens from different objects should not. At inference, no bounding boxes are needed. The penalty shaped the geometry, and the code runs bbox-free.
The third constraint is scope. The vision encoder and language backbone stay frozen; only the shared projection layer is trained, against a contrastive retrieval loss. That keeps SaMer close to a drop-in adapter, a practical signal for teams that already run a ColPali checkpoint in production and want to test the storage cut without rewriting the stack.
The numbers are author-reported and unreplicated. With K=64, the arXiv preprint says SaMer removes more than 93% of image-side tokens, cuts ColPali storage by 16.09×, and improves R@1 on Flickr30K and MSCOCO against compression baselines of the paper's choosing. It also reports stronger phrase-level grounding, a result consistent with merging around object identity rather than collapsing it.
Three caveats. The object prior during training is Flickr30k-Entities alone; nothing in the paper shows the geometry transfers cleanly to documents, charts, or screen UIs where object boundaries are softer or absent. No independent benchmark is included. And the scope is multi-vector retrieval, not generative VLMs, so readers expecting a general vision-language compression result will be looking in the wrong place.
ColPali efficiency has usually been treated as a count problem: fewer tokens, smaller index, lower bill. SaMer treats it as an evidence-selectability problem: how many of the original patches a query can still reach, and which objects they represent. If the reframe holds up under replication and on retrieval workloads outside Flickr-style images, retrieval-infra teams gain a cleaner objective than raw token count to optimize against.
Watch for replication on document and chart corpora, independent benchmarks against pruning and pooling baselines outside the paper's comparison, and whether bbox-free inference preserves object identity when the input is not a natural photograph. Code is released.