First-hand information for everyone
NPCBench is a newly released benchmark for nasopharyngeal carcinoma (NPC), a cancer common in Southern China and Southeast Asia. It tests AI systems on a full care pathway, from initial diagnosis to ongoing treatment decisions, using validated multimodal data—histology, MRI images, and patient's clinical history.
The benchmark is built around 100 patient case histories that contain 417 scored clinical tasks. Those tasks include 100 pathology assessments, 100 MRI T‑stage assessments and 217 care‑decision questions. A key feature is a joint, episode‑level check: a care decision counts as correct only if the diagnosis, MRI stage and treatment choice are all correct for that patient.
Across 30 evaluated models, performance varies widely. In the highest‑scoring multimodal category, a model called Claude‑4.6 achieves a best strict CarePath success of 23 % (i.e., 23 % of patient episodes have all components correct). The same model reaches 89.9 % Care accuracy when diagnosis and treatment are evaluated independently. In contrast, Gemini‑3.1‑Pro‑Preview scores 11.0 % strict success but 89.9 % Care accuracy. A table of these results is shown in Figure 1.
| Model | Input interface | Inference |
|---|---|---|
| Medical-Specific LLMs | ||
| HuatuoGPT-o1-72B | Text | Local |
| HuatuoGPT-o1-7B | Text | Local |
| Baichuan-M2-32B | Text | Local |
| Fleming-R1-32B | Text | Local |
| Fleming-R1-7B | Text | Local |
| Medical-Specific MLLMs | ||
| HealthGPT-Pro-4B | Text and image | Local |
| HealthGPT-Pro-8B† | Text and image | Local |
| Hulu-Med-4B | Text and image | Local |
| Hulu-Med-7B | Text and image | Local |
| Hulu-Med-32B | Text and image | Local |
| Lingshu-7B | Text and image | Local |
| Lingshu-32B | Text and image | Local |
| Open-Source LLMs | ||
| Qwen3-30B-A3B | Text | Local |
| GPT-OSS-20B | Text | Local |
| GPT-OSS-120B | Text | Local |
| DeepSeek-V3.1 | Text | API |
| Open-Source MLLMs | ||
| Qwen3-VL-8B-Instruct† | Text and image | Local |
| Qwen3-VL-30B-A3B-Instruct† | Text and image | Local |
| Qwen3.5-9B | Text and image | Local |
| Qwen3.5-27B† | Text and image | Local |
| Qwen3.5-35B-A3B | Text and image | Local |
| Qwen3.6-27B | Text and image | Local |
| Qwen3.6-35B-A3B | Text and image | Local |
| InternVL3.5-38B | Text and image | Local |
| Proprietary (M)LLMs | ||
| GPT-5.1 | Text and image | API |
| GPT-4.1 | Text and image | API |
| Gemini-3.1-Pro-Preview | Text and image | API |
| Gemini-2.5-Pro | Text and image | API |
| Gemini-2.5-Flash | Text and image | API |
| Claude-Sonnet-4.6 | Text and image | API |
When models are asked to revise decisions over time (the L5 longitudinal revision stage), Gemini‑3.1‑Pro‑Preview is the top performer. It correctly answers 87.9 % of individual revision turns and achieves patient‑level all‑turn success of 71.0 %. Claude‑4.6 comes next with 85.2 % turn accuracy and 63.0 % all‑turn success. These findings are illustrated in Figure 2.
| Model | Turn | Patient macro | All-turn success |
|---|---|---|---|
| API models | |||
| DeepSeek-V3.1 | 74.4 | 74.2 | 42.0 |
| GPT-5.1 | 75.7 | 73.7 | 45.0 |
| GPT-4.1 | 78.7 | 77.5 | 45.0 |
| Gemini-3.1-Pro-Preview | 87.9 | 88.6 | 71.0 |
| Gemini-2.5-Pro | 82.0 | 81.6 | 56.0 |
| Gemini-2.5-Flash | 69.5 | 71.7 | 42.0 |
| Claude-Sonnet-4.6 | 85.2 | 85.6 | 63.0 |
| Local models | |||
| HuatuoGPT-o1-72B | 63.9 | 62.1 | 26.0 |
| HuatuoGPT-o1-7B | 55.1 | 53.4 | 20.0 |
| Baichuan-M2-32B | 66.2 | 65.5 | 31.0 |
| Fleming-R1-32B | 75.7 | 74.1 | 40.0 |
| Fleming-R1-7B | 59.3 | 59.3 | 23.0 |
| HealthGPT-Pro-4B | 26.2 | 24.9 | 4.0 |
| HealthGPT-Pro-8B | 38.7 | 37.3 | 9.0 |
| Hulu-Med-4B | 59.3 | 59.1 | 26.0 |
| Hulu-Med-7B | 47.2 | 47.1 | 16.0 |
| Hulu-Med-32B | 70.2 | 70.4 | 37.0 |
| Lingshu-7B | 49.2 | 49.4 | 14.0 |
| Lingshu-32B | 70.5 | 70.1 | 37.0 |
| Qwen3-30B-A3B | 63.3 | 63.5 | 27.0 |
| GPT-OSS-20B | 48.9 | 48.0 | 13.0 |
| GPT-OSS-120B | 60.3 | 58.0 | 23.0 |
| Qwen3-VL-8B-Instruct | 69.8 | 70.0 | 38.0 |
| Qwen3-VL-30B-A3B-Instruct | 68.5 | 67.6 | 36.0 |
| Qwen3.5-9B | 60.7 | 60.7 | 24.0 |
| Qwen3.5-27B | 76.4 | 74.5 | 46.0 |
| Qwen3.5-35B-A3B | 72.8 | 72.9 | 41.0 |
| Qwen3.6-27B | 79.7 | 79.6 | 53.0 |
| Qwen3.6-35B-A3B | 76.1 | 74.6 | 49.0 |
| InternVL3.5-38B | 58.4 | 58.4 | 21.0 |
The benchmark highlights a composition gap: models that perform well on isolated tasks often fail to combine those skills within a patient episode. For example, Gemini‑3.1‑Pro‑Preview can achieve high Care accuracy, but because its strict CarePath success is only 11 %, it still makes errors when diagnosis, stage and treatment are judged together.
NPCBench was assembled by experts who verified that each task’s evidence is available at the moment the question would be asked in real practice. The data are from a single tertiary center and include assigned pathology slides in 75% of the cases, which the authors note may affect generalizability.
Future work will need multi‑center, prospectively collected data to show whether improvements on NPCBench translate into better clinical outcomes.