First-hand information for everyone
Researchers tested six large‑language‑model chatbots that many people use for health advice. The bots answered the same four questions about quitting smoking for 12 synthetic user profiles that vary by age, occupation, and how much they claim to know about health. The responses differed in length, amount of facts, and in whether the models named prescription drugs that can help people quit. Teen‑age profiles were far less likely to get drug names – only 22–57 % of answers mention a medication – while older adults (mid‑career, retired) got drug names in 75–98 % of cases. The pattern held for every bot tested. When a brief system instruction was added that forces the model to always provide key medical facts, the bias toward older users changed differently for each bot, showing that a single prompt does not work the same way across models.
| Model | Teen | Coll. | Mid-career | Retired | Gap |
|---|---|---|---|---|---|
| Claude Haiku | 0.57 | 0.70 | 0.78 | 0.88 | 0.32 |
| Claude Sonnet | 0.50 | 0.98 | 0.97 | 0.97 | 0.48 |
| Gemini 2.5 Flash | 0.47 | 0.78 | 0.90 | 0.85 | 0.43 |
| Gemini 2.5 Pro | 0.15 | 0.37 | 0.65 | 0.75 | 0.60 |
| GPT-5.5 | 0.22 | 0.83 | 0.88 | 0.88 | 0.67 |
| Grok 4.3 | 0.55 | 1.00 | 0.98 | 1.00 | 0.45 |
The study was carried out with 2,880 answers in total (12 profiles × 4 questions × 6 models). The profiles mixed age, smoking history, and job titles, so the researchers note that the differences seen can be due to any of those factors, not only age or expertise. The measurements rely on automatic word matching, which counts whether a drug name appears but cannot judge whether it is used correctly or omitted properly.
The experiments were run under the same conditions: each model was prompted with the written question, and for the prompt‑condition every model received a short instruction that lists the core medical facts that should always be mentioned.
The results show that commercial chatbots are not consistent when answering health questions and that the context of the user influences which information is provided. However, the designs used in the study cannot separate the effect of age from smoking history or occupation, and the automatic scoring method cannot confirm the clinical appropriateness of the medication mentions.