arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

New NPCBench benchmark shows large‑language models still miss critical steps in nasopharyngeal carcinoma care | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Artificial IntelligenceEnglishPublished

New NPCBench benchmark shows large‑language models still miss critical steps in nasopharyngeal carcinoma care

November 7, 2026arXiv: 2611.08369v1

NPCBench is a newly released benchmark for nasopharyngeal carcinoma (NPC), a cancer common in Southern China and Southeast Asia. It tests AI systems on a full care pathway, from initial diagnosis to ongoing treatment decisions, using validated multimodal data—histology, MRI images, and patient's clinical history.

The benchmark is built around 100 patient case histories that contain 417 scored clinical tasks. Those tasks include 100 pathology assessments, 100 MRI T‑stage assessments and 217 care‑decision questions. A key feature is a joint, episode‑level check: a care decision counts as correct only if the diagnosis, MRI stage and treatment choice are all correct for that patient.

Across 30 evaluated models, performance varies widely. In the highest‑scoring multimodal category, a model called Claude‑4.6 achieves a best strict CarePath success of 23 % (i.e., 23 % of patient episodes have all components correct). The same model reaches 89.9 % Care accuracy when diagnosis and treatment are evaluated independently. In contrast, Gemini‑3.1‑Pro‑Preview scores 11.0 % strict success but 89.9 % Care accuracy. A table of these results is shown in Figure 1.

Table A7 lists 30 evaluated models and reports their strict CarePath success and Care accuracy; the row for Claude‑4.6 shows a strict success of 23 % and an accuracy of 89.9 %.
ModelInput interfaceInference
Medical-Specific LLMs
HuatuoGPT-o1-72BTextLocal
HuatuoGPT-o1-7BTextLocal
Baichuan-M2-32BTextLocal
Fleming-R1-32BTextLocal
Fleming-R1-7BTextLocal
Medical-Specific MLLMs
HealthGPT-Pro-4BText and imageLocal
HealthGPT-Pro-8B†Text and imageLocal
Hulu-Med-4BText and imageLocal
Hulu-Med-7BText and imageLocal
Hulu-Med-32BText and imageLocal
Lingshu-7BText and imageLocal
Lingshu-32BText and imageLocal
Open-Source LLMs
Qwen3-30B-A3BTextLocal
GPT-OSS-20BTextLocal
GPT-OSS-120BTextLocal
DeepSeek-V3.1TextAPI
Open-Source MLLMs
Qwen3-VL-8B-Instruct†Text and imageLocal
Qwen3-VL-30B-A3B-Instruct†Text and imageLocal
Qwen3.5-9BText and imageLocal
Qwen3.5-27B†Text and imageLocal
Qwen3.5-35B-A3BText and imageLocal
Qwen3.6-27BText and imageLocal
Qwen3.6-35B-A3BText and imageLocal
InternVL3.5-38BText and imageLocal
Proprietary (M)LLMs
GPT-5.1Text and imageAPI
GPT-4.1Text and imageAPI
Gemini-3.1-Pro-PreviewText and imageAPI
Gemini-2.5-ProText and imageAPI
Gemini-2.5-FlashText and imageAPI
Claude-Sonnet-4.6Text and imageAPI
Table A7. Table showing strict CarePath success and Care accuracy for multimodal models, with Claude‑4.6 achieving the highest strict success of 23 %.Wang, Pengkai, Zhang, Wei-Wei, Li, Yan, Tang, Min, Hou, Zhitian, Liu, Zeyu et al. · arXiv:2611.08369v1 · Table A7: · CC BY 4.0

When models are asked to revise decisions over time (the L5 longitudinal revision stage), Gemini‑3.1‑Pro‑Preview is the top performer. It correctly answers 87.9 % of individual revision turns and achieves patient‑level all‑turn success of 71.0 %. Claude‑4.6 comes next with 85.2 % turn accuracy and 63.0 % all‑turn success. These findings are illustrated in Figure 2.

Table A10 presents longitudinal decision performance for 100 patients; Gemini‑3.1‑Pro‑Preview has 87.9 % turn accuracy and 71.0 % all‑turn success, the highest among the models listed.
ModelTurnPatient macroAll-turn success
API models
DeepSeek-V3.174.474.242.0
GPT-5.175.773.745.0
GPT-4.178.777.545.0
Gemini-3.1-Pro-Preview87.988.671.0
Gemini-2.5-Pro82.081.656.0
Gemini-2.5-Flash69.571.742.0
Claude-Sonnet-4.685.285.663.0
Local models
HuatuoGPT-o1-72B63.962.126.0
HuatuoGPT-o1-7B55.153.420.0
Baichuan-M2-32B66.265.531.0
Fleming-R1-32B75.774.140.0
Fleming-R1-7B59.359.323.0
HealthGPT-Pro-4B26.224.94.0
HealthGPT-Pro-8B38.737.39.0
Hulu-Med-4B59.359.126.0
Hulu-Med-7B47.247.116.0
Hulu-Med-32B70.270.437.0
Lingshu-7B49.249.414.0
Lingshu-32B70.570.137.0
Qwen3-30B-A3B63.363.527.0
GPT-OSS-20B48.948.013.0
GPT-OSS-120B60.358.023.0
Qwen3-VL-8B-Instruct69.870.038.0
Qwen3-VL-30B-A3B-Instruct68.567.636.0
Qwen3.5-9B60.760.724.0
Qwen3.5-27B76.474.546.0
Qwen3.5-35B-A3B72.872.941.0
Qwen3.6-27B79.779.653.0
Qwen3.6-35B-A3B76.174.649.0
InternVL3.5-38B58.458.421.0
Table A10. Table showing turn accuracy and all‑turn patient success for L5 longitudinal revision, with Gemini‑3.1‑Pro‑Preview achieving 87.9 % turn accuracy and 71.0 % all‑turn success.Wang, Pengkai, Zhang, Wei-Wei, Li, Yan, Tang, Min, Hou, Zhitian, Liu, Zeyu et al. · arXiv:2611.08369v1 · Table A10: · CC BY 4.0

The benchmark highlights a composition gap: models that perform well on isolated tasks often fail to combine those skills within a patient episode. For example, Gemini‑3.1‑Pro‑Preview can achieve high Care accuracy, but because its strict CarePath success is only 11 %, it still makes errors when diagnosis, stage and treatment are judged together.

NPCBench was assembled by experts who verified that each task’s evidence is available at the moment the question would be asked in real practice. The data are from a single tertiary center and include assigned pathology slides in 75% of the cases, which the authors note may affect generalizability.

Future work will need multi‑center, prospectively collected data to show whether improvements on NPCBench translate into better clinical outcomes.

Source paper

NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma

Artificial Intelligence
View on arXiv