Can we trust LLMs to judge whether a research idea is novel?
Ai2 releases a benchmark showing LLM-based novelty judges are highly unreliable for evaluating AI-generated research ideas, with prompt wording causing 50+ point accuracy swings and simple baselines outperforming purpose-built systems.