First-hand information for everyone
This paper shows that for clinical large language models (LLMs) safety and accuracy do not always improve together. The authors introduce SaFE-Scale, a framework to measure how safety changes when models are scaled or deployed differently. They also build RadSaFE-200, a radiology benchmark of 200 multiple-choice questions where each question has clinician-written “clean” evidence, conflicting evidence, and option-level labels for high-risk errors, unsafe answers, and evidence contradiction.
The team tested 34 locally run LLMs from several families (for example, Qwen, Llama, Gemma, MedGemma, DeepSeek, Mistral, and OpenAI-OSS) under six deployment conditions: closed-book prompting (no evidence), clean evidence, conflict evidence, standard retrieval-augmented generation (RAG), agentic RAG (an LLM-driven multi-step retrieval and reasoning pipeline), and max-context prompting (dumping all retrieved passages into a long context). For each run they recorded the chosen answer, the model’s confidence, and latency, and mapped outputs to the predefined safety labels. Main outcomes were rates of high-risk errors, unsafe answers, evidence contradiction, and dangerous overconfidence; accuracy was reported as a secondary endpoint.
The clearest result was that clinician-written clean evidence produced the largest and most consistent safety gains. Averaged across the 34 models, accuracy rose from 73.5% (closed-book) to 94.1% with clean evidence. High-risk error fell from 12.0% to 2.6%, contradiction from 12.7% to 2.3%, and dangerous overconfidence from 8.0% to 1.6%. By contrast, standard RAG and agentic RAG did not match that safety profile: agentic RAG increased accuracy slightly over standard RAG (about 76.0% to 78.1%) and reduced contradiction (11.7% to 9.0%), but high-risk errors and dangerous overconfidence stayed elevated. Longer contexts increased response latency but did not close the safety gap, and extra inference-time compute (for example, self-consistency sampling or small ensembles) produced only limited additional benefit and sometimes preserved synchronized failures across models. The authors also found that clinically important errors tended to concentrate in a small subset of questions.
Why this matters: in medicine a single confident, high-risk mistake can cause harm even when average accuracy looks good. The study shows that safer behavior is not an automatic side effect of bigger models, longer contexts, or more compute. Instead, safety depends on how evidence is gathered and presented and how the system is deployed. RadSaFE-200 was built so that wrong answers are not all treated the same; they are labeled for clinical risk and contradiction to evidence, which lets the researchers see types of failures that accuracy alone would hide.
Important caveats: the work focuses on radiology multiple-choice questions and a specific controlled benchmark, so results may not transfer directly to other clinical tasks or to different retrieval sources. The safe-behavior gains depended strongly on clinician-written clean evidence; retrieval systems using a clinical corpus (here, Radiopaedia) did not reproduce the same safety improvements. The authors also note that safety-relevant errors are sparse and concentrated, which means average metrics can hide a small number of dangerous failures. Overall, the paper argues that measuring safety under the exact evidence, retrieval, context, and compute conditions of deployment is necessary before claiming a clinical LLM is safe.