First-hand information for everyone
Researchers created a new benchmark that spices up real scientific papers with four kinds of clear‑cut mistakes—general layout problems, wrong math, wrong numbers, and missing citations—at three severity levels. The benchmark then asks various reviewer systems to flag them and records how many errors each finds and how often the system raises a flag on a clean paper. The test is run on two 15‑paper collections—one from astrophysics and one from machine‑learning—chosen from recent arXiv submissions.
| System | Sev | XFamJud | NonML | FP |
|---|
| Mech |
|---|
| ErrType |
|---|
| Scaff |
|---|
| HumEval |
|---|
| Release |
|---|
| ReviewerGPT [12] | – | – | – | – | – | ✓ | – | ∘ | ✓ |
| Liang et al. [11] | – | – | ✓ | – | – | ✓ | – | ✓ | ∘ |
| MARG [5] | – | – | – | – | – | ✓ | ✓ | ✓ | ✓ |
| AgentReview [9] | – | – | – | – | – | ✓ | ✓ | – | ✓ |
| TreeReview [4] | – | – | ∘ | – | – | – | ✓ | ✓ | ✓ |
| CycleResearcher [23] | – | – | – | – | – | – | ∘ | ✓ | ✓ |
| DeepReviewer [30] | – | – | – | – | ∘ | – | ✓ | – | ✓ |
| PaperAudit-Bench [20] | – | – | – | ∘ | ∘ | ✓ | ✓ | – | ✓ |
| AAAI-26 Pilot [2] | – | – | – | – | ✓ | ✓ | ✓ | ✓ | – |
| FLAWS [24] | – | – | – | – | – | – | – | ✓ | ✓ |
| Dycke & Gurevych [6] | – | – | – | ✓ | – | – | ✓ | ✓ | ✓ |
| This work | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ∘ | ✓ |
In the astrophysics set the multi‑agent system called Skepthical spots about 70 % of all injected errors, the best performance among all reviewers. On the smaller machine‑learning set Skepthical ties the strongest single‑LLM system, each catching roughly 81 % of the problems. Structured prompts that guide the model’s format improve detection by as much as 48 percentage points for top LLMs on the astrophysics corpus, while free‑form prompts give lower rates.
False‑positive rates stay below 10 % for most systems on clean copies, with Skepthical at 2.8 % on astrophysics and 4.0 % on machine‑learning. Most of these false alarms come from the system marking a math or numeric claim as wrong.
| Reviewer | Citations | Numerics | Mathematics | General | Total | |
|---|---|---|---|---|---|---|
| Astrophysics corpus | ||||||
| Single LLM free-form | Claude Opus 4.7 | 9.3±10 | 55.8±15 | 32.5±17 | 48.8±15 | 36.7±8.5 |
| GPT-5.5 | 22.2±13 | 80.0±10 | 73.8±13 | 66.7±12 | 60.5±6.0 | |
| Gemini 3.1 Pro | 4.4±5.6 | 13.3±13 | 2.4±3.6 | 26.7±12 | 11.9±5.8 | |
| Single LLM structured | Claude Opus 4.7 | 20.9±12 | 79.1±12 | 55.0±16 | 58.1±18 | 53.3±9.1 |
| GPT-5.5 | 18.2±8.7 | 79.5±12 | 75.6±𝟏𝟐 | 72.7±11 | 61.3±6.6 | |
| Gemini 3.1 Pro | 11.1±7.8 | 57.8±18 | 45.2±19 | 35.6±13 | 37.3±5.7 | |
| Agentic | Skepthical | 46.7±𝟏𝟑 | 80.0±11 | 73.8±9.5 | 82.2±8.9 | 70.6±6.2 |
| Claude Code, structured | 26.7±11 | 91.1±7.8 | 71.4±14 | 55.6±16 | 61.0±5.0 | |
| Claude Code, free-form | 8.9±7.8 | 42.2±18 | 19.0±11 | 44.4±12 | 28.8±6.3 | |
| External baselines | Stanford Agentic Reviewer | 8.9±10 | 53.3±12 | 50.0±19 | 51.1±11 | 40.7±6.4 |
| OpenReviewer-8B | 0.0 | 0.0 | 0.0 | 4.4±5.6 | 1.1±1.4 | |
| PaperAudit-Bench Deep | 0.0 | 6.7±8.9 | 2.4±3.6 | 0.0 | 2.3±2.5 | |
| Machine-learning corpus | ||||||
| Single LLM free-form | Claude Opus 4.7 | 13.3±8.9 | 55.6±12 | 40.0±13 | 66.7±14 | 43.9±6.1 |
| GPT-5.5 | 28.9±14 | 80.0±10 | 77.8±14 | 68.9±13 | 63.9±7.2 | |
| Gemini 3.1 Pro | 0.0 | 4.4±5.6 | 11.1±10 | 28.9±12 | 11.1±5.8 | |
| Single LLM structured | Claude Opus 4.7 | 57.8±13 | 84.4±11 | 77.8±13 | 68.9±11 | 72.2±6.1 |
| GPT-5.5 | 57.8±14 | 91.1±7.8 | 91.1±𝟏𝟎 | 84.4±8.9 | 81.1±5.3 | |
| Gemini 3.1 Pro | 26.7±11 | 57.8±13 | 62.2±13 | 57.8±6.7 | 51.1±5.6 | |
| Agentic | Skepthical | 53.3±12 | 91.1±6.7 | 77.8±11 | 84.4±8.9 | 76.7±5.6 |
| Claude Code, structured | 60.0±𝟏𝟐 | 86.7±14 | 71.1±16 | 75.6±13 | 73.3±8.6 | |
| Claude Code, free-form | 15.6±10 | 42.2±19 | 24.4±16 | 55.6±11 | 34.4±8.6 | |
| External baselines | OpenReviewer-8B | 0.0 | 2.2±3.3 | 0.0 | 4.4±5.6 | 1.7±1.7 |
| PaperAudit-Bench Deep | 0.0 | 4.4±5.6 | 2.2±3.3 | 4.4±5.6 | 2.8±1.9 | |
The benchmark probes only objective, locatable defects and does not assess novelty, significance or overall review quality. The two corpora also come from single days and contain short papers; longer or more diverse manuscripts may show different results.