First-hand information for everyone
The authors created Trick‑or‑Treat, a benchmark that pairs real chest X‑rays with synthetic photographs of the same patient. The synthetic images differ only in disease‑relevant findings while keeping identity and background the same. This design lets researchers test whether a model can reconcile evidence from two modalities.
In matched pairs, the X‑ray and photograph both hint at the same diagnosis. In mismatched pairs, the photograph shows a different finding that would not support the X‑ray diagnosis. Three leading multimodal models were evaluated on 40 matched and 40 mismatched pairs for each of several conditions.
When questioned about photographs alone, the models made clinically significant hallucinations in 25–37.5% of mismatched images and missed real abnormalities in 10–27.5% of matched images. Adding the corresponding radiograph to a mismatched pair increased hallucination rates to 45–57.5%, while it lowered false‑positive and false‑negative rates in matched pairs.
| Matched |
|---|
| Mismatched |
|---|
| Model | Setting | FP | FN | Any | FP | FN | Any |
|---|
| Claude Fable 5.1 | Photo only | 25.0 (12.7–41.2) | 27.5 (14.6–43.9) | 30.0 (16.6–46.5) | 25.0 (12.7–41.2) | 2.5 (0.1–13.2) | 27.5 (14.6–43.9) |
|---|
| Photo + radiograph | 17.5 (7.3–32.8) | 12.5 (4.2–26.8) | 20.0 (9.1–35.6) | 45.0 (29.3–61.5) | 2.5 (0.1–13.2) | 47.5 (31.5–63.9) |
|---|
| GPT-6 Astra | Photo only | 12.5 (4.2–26.8) | 10.0 (2.8–23.7) | 15.0 (5.7–29.8) | 37.5 (22.7–54.2) | 2.5 (0.1–13.2) | 40.0 (24.9–56.7) |
|---|
| Photo + radiograph | 2.5 (0.1–13.2) | 2.5 (0.1–13.2) | 2.5 (0.1–13.2) | 57.5 (40.9–73.0) | 0.0 (0.0–8.8) | 57.5 (40.9–73.0) |
|---|
| Gemini 4 Argon | Photo only | 10.0 (2.8–23.7) | 7.5 (1.6–20.4) | 12.5 (4.2–26.8) | 35.0 (20.6–51.7) | 2.5 (0.1–13.2) | 37.5 (22.7–54.2) |
|---|
| Photo + radiograph | 7.5 (1.6–20.4) | 2.5 (0.1–13.2) | 10.0 (2.8–23.7) | 45.0 (29.3–61.5) | 5.0 (0.6–16.9) | 47.5 (31.5–63.9) |
|---|
The results reveal a pattern: multimodal context can amplify reasoning errors. Models tend to favor findings that match the X‑ray, ignoring mismatches, and may invent or over‑emphasize abnormalities that align with the radiograph’s suggestion.
These findings also demonstrate that synthetic photographs can successfully generate controlled adversarial tests for medical vision‑language systems, providing a practical path for future benchmark development.
The study acknowledges that synthetic images were produced with clinician review, but highlights that subtle artifacts could still influence model predictions.
Missing data arise from limited test samples; the benchmark contains 40 images per condition per setting, which limits statistical power.
Overall, Trick‑or‑Treat exposes a serious weakness in current multimodal models and suggests that synthetic data can be valuable in designing rigorous, controlled tests of clinical reasoning.