Ai2: LLMs are unreliable judges of research novelty

What happened?

The Allen Institute for AI (Ai2) released Old Ideas, Novel Problems, a study of how reliably LLMs judge the novelty of research ideas (Hugging Face / Ai2). The paper comes with a benchmark, a dataset, code and an automated pipeline for building new evaluation sets.

The context: automated science systems increasingly generate research ideas, and whether those ideas are actually new is usually decided by another LLM. Ai2 calls this a novelty judge. These judges have mostly been validated on human-written papers, not on the AI-generated ideas they are meant to evaluate.

How the benchmark is built

• It is built on ICLR 2026 peer reviews. Instead of treating acceptance as a novelty signal, Ai2 mined passages where reviewers explicitly discussed originality.
• The high-novelty pool has 154 ideas from accepted papers in the top 10% of their area, where reviewers praised novelty and nobody disputed it.
• The lower-novelty pool has 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns.
• The Human+Generated setting pairs the 154 human ideas with 154 ideas from a simple LLM generator with no search or feedback.
• As checks, an annotator agreed with the extracted reviewer signal in 47 of 50 cases, and an expert preferred the human idea in all 30 pairs reviewed.

What they found

Six judges were tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The team varied one factor at a time, such as the prompt, reasoning budget, access to related papers or idea formatting.

• The prompt mattered most. On Human+Generated pairs, GPT-5.4 fell from 93.5% to 40.9% accuracy, below chance, when the prompt no longer said each pair contained exactly one novel idea. It changed its verdict on more than half of the pairs.
• On human-only ideas the same change moved accuracy by about 7 points on average, but still by 21 points for Claude Opus 4.6.
• There is no universally better prompt: dropping one instruction helped some judges and hurt others.
• When comparing two AI generators, prompt changes flipped roughly a quarter to a third of verdicts on average.

Source: Ai2. 154 pairs, each with a human idea praised as novel by ICLR reviewers and one generated by Claude Sonnet 4.5, under two different prompts.

Does more search or reasoning help?

Not necessarily. GPT-5.6 Sol with an extra-high reasoning budget and live arXiv search scored 0.78 macro-F1, 11 points behind Claude Opus 4.6 at 0.89, while costing roughly 10x more per decision.

Purpose-built systems did not do better either. A simple Claude Opus 4.6 prompt with low reasoning and majority voting over three calls reached 0.854 macro-F1 for $2.59 in total. Idea Novelty Checker scored 0.642 for $30.66, and the novelty evaluator in Sakana's AI Scientist scored 0.772 for $82.05.

What it means for developers

If you use LLM-as-a-judge to score generated content, this is an uncomfortable but useful warning: a judge that looks stable on human text can behave very differently on AI-generated input. Ai2 recommends testing judges on the kind of input they will see in production and measuring their sensitivity across prompts and models.

The practical takeaway is simple: a pricier pipeline and more compute do not guarantee better verdicts. The open dataset and pipeline make it possible to run your own judges through the benchmark before trusting them with decisions.