First-hand information for everyone
A new paper on arXiv suggests caution for anyone treating a language model’s citation of a statute or court precedent as evidence that the model’s legal conclusion actually follows from that authority. The authors perform a direct, counterfactual test and find that models often “name” the right law while their underlying verdicts do not reliably depend on it.
What the researchers did
The team tested seven openly available language models (sizes from about 8 billion to 70 billion parameters) on four legal reasoning benchmarks that cover judicial and contract questions. Instead of simply reading a model’s final text, they performed a counterfactual audit: holding the facts of a case fixed, they swapped the named legal authority (for example, a statute or precedent) for an unrelated one, then decoded the model’s evolving verdict from its hidden states to see whether the decision actually changed.
Key findings
- Models were often willing to identify the “correct” governing authority when explicitly prompted to justify a verdict: across experiments, the named authority matched the expected one in about 66.7%–100% of generations. - But changing the named authority did not reliably change the verdict. The fraction of cases where a verdict actually flipped when authorities were swapped was low in many settings: 0.0%–21.7% on the CaseHOLD benchmark, 30.0%–76.7% on ECHR and SCOTUS benchmarks, and 43.3%–50.0% on ContractNLI. - Increasing model size and training a dedicated legal-reasoning variant (a best-effort LoRA reproduction) did not close this gap.
The paper also ran a red‑teaming test: hiding an adversarial instruction inside the case facts to see whether models would follow it. Compliance with that instruction (73.3%–96.4%) was far higher than the sensitivity of verdicts to substituted authorities. In other words, models were more likely to obey a secret instruction in the facts than to actually change their decision when the supposed legal basis changed.
Why this matters
Many people — developers, auditors, regulators, and users — have treated a model’s citation of a statute or precedent as a kind of explanation or proof that the model reasoned from that authority. This paper shows that such “name‑dropping” is an unreliable proxy for genuine dependence of a decision on the cited law. A model can assert the correct authority but still be making the decision for other reasons, and separately it can be quite vulnerable to adversarial instructions embedded in the input.
Caveats and robustness
The authors checked for confounds such as prompt wording and sampling artefacts and found their results to be robust. They also limited their study to open‑weight models and a specific hidden‑state decoding technique; real-world proprietary models or different interpretability methods could behave differently. The reproduced legal model was a best‑effort LoRA adaptation rather than a fully bespoke large legal model.
Recommendations
The paper’s findings argue for caution when using generated legal explanations as audit or compliance artifacts. Practical steps to reduce risk include: - Treating citations as claims to be independently verified, not proofs. - Combining model outputs with retrieval of primary law and chainable, verifiable reasoning steps. - Keeping humans in the loop for high‑stakes legal decisions. - Using counterfactual and adversarial audits as part of standard evaluation for legal applications.
Bottom line
Citing a statute or case is not the same as reasoning from it. This counterfactual audit highlights a gap between the surface plausibility of legal explanations generated by language models and the models’ actual reliance on the cited authorities — a gap that has practical consequences for safety, trust, and regulation in legal AI.