AINEWS
A NeurIPS (the field's largest AI research conference) reviewer's informal tally reignites debate over whether conferences should desk reject (automatically reject before peer review) papers that omit runnable code — and exposes a reproducibility
As NeurIPS review season wound down, an anonymous reviewer took to Reddit with a count that struck a nerve: of the twelve papers they reviewed for major conferences this year, only one provided full, runnable code for the complete training pipeline. Four more offered fragments — enough to see a method's shape, not enough to test it. Seven offered nothing.
"This is really bad for ensuring quality and reproducibility," the reviewer wrote on Reddit. The post, titled "It's time to desk reject papers that don't include code that can reproduce the results," has since attracted thousands of upvotes and hundreds of replies from researchers arguing over incentives, enforcement, and the limits of peer review.
The numbers are one reviewer's count. They cannot be independently verified, and the reviewer has not disclosed their institution, their conferences, or how papers were selected. Any repetition of specific figures in this article should be read as that poster's report, not a published finding.
The three largest machine-learning venues — NeurIPS, ICML, and ICLR — all maintain author and reviewer guidelines that address code availability. Their requirements differ in scope and enforcement.
NeurIPS currently operates a mandatory paper checklist that must be completed at camera-ready submission. According to the code submission policy, code must be submitted alongside the paper, though the policy does not prescribe a specific desk-reject trigger for missing code alone. The 2026 MainTrack Handbook and related reviewer guidance govern the review process for the current cycle.
ICML requires code "as part of the camera-ready submission" per its 2026 author instructions, but does not specify an automatic desk-reject mechanism for code that is present but non-reproducible.
The upshot: across all three major venues, the stated preference for code is strong, the enforcement mechanisms are soft, and the cost to authors of omitting runnable code during review remains low.
The gap between stated policy and actual practice is well documented. Joelle Pineau — Reproducibility Chair for NeurIPS 2020 and co-author of the definitive program report — described the landscape in a 2020 Medium post: at NeurIPS 2019, roughly 40% of authors reported providing a code link at the submission stage, and 75% of accepted papers included a code link at final submission. Those figures represent a significant improvement over NeurIPS 2018, but they also mean that one in four accepted papers still arrived without code.
The numbers have likely shifted since 2020, but the underlying pattern — strong stated preference, soft enforcement — has not changed in kind.
The anonymous reviewer's core argument is structural. During masked review, the authors who do release code face a paradox: every additional reviewer who can run the experiment is one more opportunity for someone to find a bug. Authors who withhold code face no corresponding penalty. The result is an equilibrium that rewards concealment.
"The only way to fix this is to change the game by imposing real penalties on hiding code," the reviewer wrote.
The proposal — desk-reject papers that arrive without code — is one such penalty. Whether any major conference is prepared to adopt it is a separate question.
The reproducibility problem in machine learning is not new. The NeurIPS Reproducibility Checklist was introduced to address it. Researchers at the Machine Learning Reproducibility Challenge have documented persistent gaps between published claims and reproducible results. But advocacy and policy are different things. As of mid-2026, no major ML conference has publicly adopted a hard desk-reject rule for missing code at submission. The gap between what the community says it wants and what its venues actually enforce remains wide.