A study found that Grad‑CAM, a common explanation tool for deep learning models, pinpointed disease findings in chest X‑rays less often than human radiologists. The hit rate for Grad‑CAM was at most 0.575, while radiologists achieved 0.888. The researchers trained two models, a DenseNet‑121 convolutional network and a Vision Transformer (ViT‑B/16), on the NIH ChestX‑ray14 dataset and evaluated them on 303,768 decisions at their training hospital and at two external sites in the United States and Vietnam. Each explanation was scored by how concentrated it was within the thoracic region, and the ability of the models to detect errors beyond their confidence was assessed.

After transferring to the external hospitals, performance varied. At the CheXpert site, the macro AUROC fell from 0.857 to 0.824 for DenseNet‑121 and from 0.844 to 0.803 for ViT‑B/16. At VinDr‑CXR, the AUROC rose from 0.829 to 0.893 for DenseNet‑121 and from 0.817 to 0.886 for ViT‑B/16, suggesting that cleaner consensus labels might improve transfer.

Attempts to improve failure detection by adding anatomical enrichment to a confidence‑based detector had little effect. The area‑under‑curve increased by no more than 0.011 at any site, and for the convolutional network the relationship between enrichment and correctness reversed between hospitals, actually degrading the transferred detector. These results suggest that model confidence, rather than saliency maps, should guide deployment monitoring.

The findings highlight that current saliency techniques do not match radiologists’ ability to localize findings and may limit their usefulness for clinical monitoring when models are deployed in new settings.