First-hand information for everyone
Main finding: When given a stated update and asked for a single probability, every tested model configuration moved on average in the annotated direction. The mean directional movement s across configurations lies between about 17.6 and 28.3 percentage points. Not every item changed, but the average shift was robust across models. Directional movement s is defined as the posterior change signed so that movement toward a strengthener counts positive and toward a weakener counts negative; s is averaged over paired strengthener/weakener items so the prior cancels out.
How the study was done: The authors sampled items from the δ‑SNLI subset and used 240 premise–hypothesis pairs (distinct images). For each pair they selected one human‑written strengthener and one weakener. Two models (Qwen3‑8B and OLMo‑2‑7B) were run locally on 240 pairs each. Five additional configurations were run through agent command‑line harnesses (including codex/gpt‑6.1‑sol) on 120 pairs each. The protocol varied presentation wording (stated: “Here is something else you learned: U” vs reported: “Someone said that U”) and elicitation style (direct single probability vs five weighted estimates aggregated by weighted mean). Hypotheses, smallest effect sizes of interest (SESOIs: 5 points for probability movement; 10 percentage points for endpoint‑rate differences), missing‑data rules, and analysis methods were frozen before the formal runs. Inferential statistics used a cluster bootstrap with B = 5,000 and Holm correction within each configuration.
Concrete results: Under stated presentation with direct elicitation, the three originally reported configurations had s ≈ 18.6 points for Qwen3, ≈ 20.0 points for OLMo‑2, and ≈ 25.3 points for codex.
Reporting wording: Recasting the same update as a reported statement (“Someone said that …”) reduced the mean directional movement in every configuration. Point estimates fell by roughly 0.9–5.4 points across the seven configurations. For the three original configurations the stated minus reported differences were about 2.0 points for Qwen3, 4.8 points for OLMo‑2, and 5.4 points for codex. The proportion of reverse movements (items moving opposite the annotated direction) increased for each configuration under the reported wording. The effect is consistent across configurations but small in magnitude.
Multi‑estimate elicitation: Asking models for five weighted estimates reduced extreme endpoint responses (reported values ≤5% or ≥95%) in all seven configurations. For the three original configurations the pre‑specified weighted endpoint rate (prior 1/2, stated strengthener 1/4, stated weakener 1/4) fell from about 29.6%, 42.0% and 25.2% under direct elicitation to about 4.3%, 4.9% and 14.4% under multi‑estimate elicitation. In the two locally run models multi‑estimate elicitation also substantially reduced directional movement (reductions of about 43% and 68%). In the harnessed configurations the point changes in s were much smaller (roughly between −3.7 and +0.5 points). In other words, de‑extremizing endpoint reports via multi‑estimate elicitation need not preserve sensitivity to evidence in every model.
Significance and caveats: The results show that modest changes in how an update is presented and how probabilities are elicited can alter model behavior on a standard defeasible‑NLI setup. Important caveats the authors report: δ‑NLI annotations indicate only the direction of an update, so the study does not claim what the correct magnitude of an update should. For configurations accessed through agent harnesses the researchers could not fix system prompts, sampling parameters, or seeds; conclusions about those runs are therefore qualified as applying to “the model inside the harness.” Materials screening excluded pairs mentioning protected attributes through a model‑based screening workflow rather than human screening. Parsing and preprocessing rules were revised after development diagnostics. Parsing failures triggered a single resample; if parsing still failed, units were recorded as missing and handled according to the pre‑specified missing‑data rules. In one configuration three units were recorded as missing under those rules. Finally, although all configurations moved on average, individual update sentences sometimes produced no change. These limitations and the pre‑specified SESOIs and statistical procedures are reported by the authors.