A study of a Claude‑based assistant found that when the model cannot see official National Weather Service (NWS) alerts it sometimes tells users that conditions are calm even though an official warning is active. This false reassurance was seen in 37.7 % of cases when the assistant ran a neutral prompt without alert data and in 44.9 % when the prompt also asked the model to disclose its blind spot. When the same alerts were fed to the production pipeline, the rate of false reassurance dropped to 0 % (0/66, 95 % CI 0.0–5.5).

The experiments used 117 frozen cases created from real U.S. NWS alerts—69 alert cases, 32 no‑alert controls, and 16 non‑U.S. cases—and ran them through five pipeline settings: production (A), alert‑only with neutral prompt (A2), blind neutral (B), blind with disclosure (C), and plain question (D). A single rater hand‑labeled all 342 alert‑case responses with a seven‑category scheme, after an initial binary pass.

Comparison tests showed that real alert data in the production pipeline eliminated false reassurance compared to five cases that had the same alert data but a neutral prompt (χ² = 24.04, p < 0.0001). Adding a disclosure instruction did not lower the false‑reassurance rate compared to the blind neutral condition (χ² = 0.70, p = 0.40). Removing a single sentence from the production prompt had no effect on the false‑reassurance rate.

The study also found that alert data alone did not guarantee safe behavior: the blind neutral prompt flagged alarms on 87.5 % of no‑alert controls, while the production pipeline flagged none.

These findings come with limitations. The sample is a pilot of only 117 cases, and many high‑stakes hazards such as tornado and tropical‑storm alerts were missing because the data were collected in late October. Labeling was performed by the author alone with no inter‑rater reliability, and only headline‑level alert information was used. The results apply only to this single Claude Sonnet‑4.6 pipeline and may not generalize to other models or designs.