A large audit of 41 language‑model chatbots found that most of the time the models say a mental‑health professional should be consulted—about 85 % of 35 055 responses contained that recommendation, and 11 % also warned that the model is not a doctor【fact2】. The audit used a two‑arm design. In the open‑ended arm each model was given the same 19 case vignettes phrased in three ways—by a clinician, a caregiver, or the person affected—and asked what to do; each prompt was run three times, producing 855 replies per model【fact1】【fact6】. In the structured arm the investigators gave the models a list of possible interventions and asked them to rate each option on a seven‑point scale; 29 of the 41 models completed this additional task【fact1】【fact6】.

Footing mattered. When a vignette was presented from a caregiver or patient perspective the models mentioned a professional referral in 95 % of responses, compared with 66 % for clinician‑style prompts【fact3】. Even the presence of a disclaimer changed over two thirds of replies, showing that the model’s response style shifts with the implied user role.

In the rating arm the researchers identified 83 options that were recommended in at least 5 % of the open‑ended outputs but earned an average rating below zero—negative scores indicate the model judged that option a poor first choice. These tended to be common procedures, medications, or digital tools such as a specific prescription digital therapeutic (a total of 27 of 202 medicines, 4 of 213 digital tools, and 2 of 132 therapies)【fact5】.

The audit did not try to prove cause. It relied on human coders to validate that an intervention was actually suggested, but no single expert standard was applied; agreement was measured with Krippendorff’s alpha on a binary ‘advised‑against’ outcome【fact4】. Because only footing, intervention type, and prompt wording were systematically varied, other hidden factors could still influence what the chatbots recommend【fact7】.