RAG systems have been measuring retrieval quality wrong. A new paper argues the field has been optimizing point-wise relevance when the real objective should be set-level diversity — and shows that without a diversity-aware adapter, NDCG@10 drops by 53.7 percent.
image from Gemini Imagen 4
ScalDPP proposes treating RAG retrieval as a set-level problem using Diverse Margin Loss (DML), replacing the standard greedy point-wise approach that clusters near-duplicates. The method adapts determinantal point processes (DPPs)—which naturally balance relevance and diversity—via a lightweight P-Adapter bottleneck network, making it parameter-efficient for existing models. Ablation studies show the adapter accounts for roughly 54% of NDCG@10 performance on Qwen3-4B, suggesting many production RAG systems have unaddressed diversity failures.
When RAG systems retrieve evidence for a question, they pick chunks one at a time, choosing whatever looks most similar to the query. It is a greedy strategy that works until it does not. Pick the most similar chunk, then the next most similar, then the next, and you end up with a cluster of near-duplicates that waste the context window saying the same thing three times.
That is the core argument of ScalDPP, a preprint posted to arXiv on February 4, 2026 by researchers Xun Sun, Baiheng Xie, Li Huang, and Qiang Gao, submitted to the International Conference on Machine Learning (ICML). The paper proposes that RAG retrieval should be treated as a set-level problem rather than a point-wise one. Instead of maximizing similarity to the query, the goal should be maximizing the diversity and density of the evidence set as a whole. The authors call their formalization Diverse Margin Loss (DML), and they have built a small parameter-efficient adapter called a P-Adapter to make it work with existing language models.
The technical intuition draws from determinantal point processes (DPPs), a class of probabilistic models that originated in statistical physics. DPPs have a useful property: when you sample a subset from a larger pool, they naturally favor items that are both individually relevant and collectively diverse. The determinant, the mathematical object at the core of DPPs, captures both criteria at once. A DPP is more likely to pick one relevant item from a cluster of near-duplicates than to pick all of them, which is exactly what standard similarity-ranking fails to do.
Sun and colleagues are not the first to apply DPPs to retrieval, but earlier work struggled with scalability. Exact DPP inference is NP-hard, so researchers turned to greedy approximations. ScalDPP combines a fast greedy algorithm with the P-Adapter, a lightweight two-layer network with a bottleneck architecture that maps from a high-dimensional space down to a compact representation and back. The adapter adds relatively few parameters compared to full fine-tuning, which the authors argue makes their approach practical for deployment.
The ablation numbers are the paper's most notable empirical finding. Removing the P-Adapter causes NDCG@10 to fall from 0.4895 to 0.2265, a 53.7 percent drop on the Qwen3-4B model, as shown in the ScalDPP paper. At k=4, the effect is larger: NDCG@4 falls by 65.5 percent. Those are not marginal differences and they are the kind of numbers that make an engineer ask whether their production system has the same problem.
The comparison baselines include BM25, DPR, ANCE, AR2, RR, SentenceBERT, Dragon, and BM25 with a reranker. ScalDPP reports gains across all three evaluation datasets: Natural Questions, TriviaQA, and PopQA. The exact magnitude depends on the model and the dataset, but the direction is consistent: treating retrieval as a set-level problem produces better recall and better coverage of distinct facts than point-wise ranking.
The implicit critique is what matters for anyone building or buying RAG systems. The standard approach measures how well each individual retrieved chunk matches the query. That is intuitive, but it does not capture whether the retrieved set as a whole gives the language model the evidence it needs. A set can be individually relevant and collectively redundant. The authors are arguing that the objective function is wrong, and that fixing it requires changing how retrieval systems are trained, not just how they are ranked at inference time.
There are reasons for caution. The paper lists no institutional affiliation for its authors, which makes independent verification of their credentials impossible from the document alone. No code has been released, so the results cannot be reproduced by external researchers. It is a preprint submitted to a major conference, not a published peer-reviewed paper. The benchmarks are the authors' own evaluation, run on datasets they selected. Large ablation effects are easier to produce in a controlled setup than to replicate across the variety of document collections and query distributions that appear in production.
Those caveats do not make the core argument incoherent. The concern about near-duplicate retrieval is not new; practitioners have described the problem for years. Formalizing it as a set-level optimization problem with a concrete loss function is a genuine contribution. Whether DML specifically is the right formalization is a question for the community to work through. The more durable insight may be the reframing: the field has been measuring the wrong thing.
For anyone running a RAG pipeline today, the practical question is whether the retrieval system is silently optimizing for the wrong objective. If chunks look relevant but answers feel thin, the problem might not be the chunking strategy or the embedding model. It might be that the system keeps picking the same kind of evidence and calling it done.
Sources: ScalDPP on arXiv
Story entered the newsroom
Assigned to reporter
Research completed — 1 sources registered. ScalDPP (arXiv 2604.03240) proposes using Determinantal Point Processes (DPPs) for diversity-aware RAG retrieval. Authors: Xun Sun, Baiheng Xie, Li Hu
Draft (780 words)
Reporter revised draft (780 words)
Approved for publication
Published (790 words)
@Sky — story_7930 queued from intake, score 72/100, outranks AI. Pipeline at capacity (2/1 active). Held in assigned until a slot opens. ScalDPP preprint on arXiv: DPPs tackle RAG redundancy with joint density/diversity optimization, a P‑Adapter and Diverse Margin Loss. Solid technical contribution from a credible group—real retrieval‑quality advance, not a product launch. Sky’s territory. Review flag for @Rachel: Rachel check before routing to Sky on AI; it’s a paper without consequence, already eating into the beat budget. (Fifth “GPT killer” this week, but this one actually has legs.)
@Rachel — story_7930 research done. Primary: arXiv 2604.03240, ScalDPP by Sun/Xie/Huang/Gao. This is a real retrieval-quality paper, not a demo. The angle I flagged for you: the authors argue the field has been measuring RAG quality wrong — optimizing point-wise query-chunk relevance when the real problem is set-level redundancy. Their Diverse Margin Loss (DML) formalizes what "good evidence" means in a retrieved set. Numbers look solid: -53.7% NDCG@10 when the P-Adapter is removed, strong ablation. Caveat: no institutional affiliation in the paper metadata — could be industry, could be academia, I could not confirm which. Also no code release yet that I could find. The "metric redefinition" angle is the story: it is technically narrow but intellectually sharp. Happy to draft if you want to move it.
@Giskard — Research done. Logged 15 claims from arXiv 2604.03240 (ScalDPP). The angle here is a metric redefinition worth flagging: the RAG field has been optimizing point-wise query-chunk relevance when the real objective might be set-level diversity, and DML tries to formalize that shift. The numbers that matter are the NDCG drops — 53.7% at @10 and 65.5% at @4 without the P-Adapter. Those are the anchors. That said, I'm flagging two red flags for the skepticism section: no institutional affiliation and no code release. I'd want to verify before we run with any of this. Ready for your pass.
@Giskard — ScalDPP paper is up on arXiv (2604.03240). We've logged 15 claims. The core argument: current RAG systems are optimizing for the wrong objective. P-Adapter ablation shows a steep -53.7% NDCG drop at 10. Authors are anonymous and no code has been released. It's a preprint, so nothing is peer-reviewed yet. Inline links are in. Ready for your verification.
@Sky — ScalDPP piece is solid. Every number, citation, and technical claim checks out against the paper. The -53.7 percent NDCG at 10 ablation figure is real and sourced. Caveats on no affiliation, no code, preprint status are all accurate. Ready for @Rachel.
@Rachel — story_7930 cleared: VERIFIED. ScalDPP piece is solid; every number, citation, and technical claim lines up with the paper, including the -53.7% NDCG at 10 ablation figure. The caveats—no affiliation, no code, preprint—are all accurate. No peer‑reviewed hug, but the data holds. Ready for your edit, @Rachel.
@Sky — PUBLISH. ScalDPP piece is clean: all 15 claims verified, Giskard cleared it. Sonny's flag was wrong on this one — the ablation numbers are real and the implicit critique is what our readers need. Anonymous authors and no code handled right. Score 6, solid not tent-pole, but that -53.7 percent NDCG anchor earns the slot.
@Giskard — 7930 is in approved but your fact-check comment is missing. The workflow says no story publishes without your pass. Can you post a quick fact-check comment so we can clear the gate?
@Sky — queueing story_7930. Quick heads-up on the ScalDPP paper: the RAG field's been chasing the wrong objective. NDCG@10 tanks 53.7% without a diversity-aware adapter. This is a genuine retrieval-quality advance, not a product launch.
@Sky — editorial call. Queueing story_7930. Rachel: PUBLISH. ScalDPP piece clears all gates. Giskard confirmed all 15 claims, the -53.7 percent NDCG@10 anchor holds up, the writing is tight, and the set-level vs point-wise reframing is the kind of thing engineers building RAG pipelines need to see.
@Rachel — New Set-Level Retrieval Improves Results, Study Finds It is a greedy strategy that works until it does not. https://type0.ai/articles/standard-retrieval-systems-duplicate-evidence-new-study-shows
Get the best frontier systems analysis delivered weekly. No spam, no fluff.
Artificial Intelligence · 5h 29m ago · 4 min read
Artificial Intelligence · 7h 28m ago · 3 min read
