Vision‑language models (VLMs) that analyze medical images often give incoherent results: the chosen answer, the confidence level and the written explanation can disagree. Researchers tested four open‑source VLMs on breast‑density and BI‑RADs categorisation from two mammography datasets. The tests used six controlled perturbations – the original image, an unrelated image, a blank image, no image, a changed question, and shuffled answer options – to see how each output changes.

The models behaved inconsistently across all conditions. On the RSNA‑BI‑RADs task, MedGemma‑4B achieved 42.4 % accuracy, Lingshu‑7B only 12.0 %, Qwen2‑VL‑7B barely 2.9 % and InternVL3‑8B 7.7 %. Qwen2‑VL‑7B also chose “None of the above” 62.8 % of the time. For breast‑density, MedGemma‑4B scored 33.9 %, Lingshu‑7B 49.3 %, Qwen2‑VL‑7B 24.8 % and InternVL3‑8B 42.0 %. The same model often kept a high confidence score even when the image was removed, and sometimes increased confidence on junk images – a sign that confidence does not follow evidence.

Shuffling the multiple‑choice options flipped the answer on 38 % to 99 % of cases, showing that the models rely on option order rather than on visual evidence. When the image was scrambled or omitted, some models predicted the correct “None of the above” answer purely by abstaining, but others kept a wrong category.

The study limits itself to a templated, multiple‑choice format, so the sensitivity to option wording or order may reflect the specific task rather than general VLM behaviour. The authors also note that their perturbations do not include common clinical changes such as swapped laterality or view differences, and that hallucinated or mirage reasoning could explain the confidence–evidence mismatch.

These findings warn that accuracy alone is insufficient when evaluating medical VLMs; coherence between answer, confidence and explanation must also be verified before such models can be trusted in clinical settings.