arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

Large language models keep most of their MNLI performance above chance after noun removal, but drop sharply on ANLI | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Natural Language ProcessingEnglishPublished

Large language models keep most of their MNLI performance above chance after noun removal, but drop sharply on ANLI

November 14, 2026arXiv: 2611.16843v1

Recent research tested eight large language models (LLMs) and fine‑tuned encoders by removing nouns from input data. On the standard natural language inference set MNLI, LLMs retained 70–72 % of their above‑chance accuracy even after noun removal. The same perturbation left only 14–29 % of their ANLI performance intact. For the more recent ANLI benchmark, noun removal caused a sharper decline, with only 14–29 % of performance preserved. Fine‑tuned encoders showed a similar pattern: they kept 72 % of MNLI performance but only 24 % on ANLI. These figures come from a controlled experiment that varied the perturbation—drop‑NOUN, drop‑VERB, keep‑NOUN+VERB, and shuffled word order—and measured how much accuracy remained relative to a clean baseline. The results suggest that LLMs rely less on simple noun cues in MNLI but are more sensitive to perturbations on ANLI, pointing to dataset‐specific shortcuts rather than true language understanding.

Table displaying accuracy and retained performance for each model on MNLI and ANLI when nouns are removed
MNLIRTEANLI (R1–R3 average)
System
Clean
R noun
R verb
Clean
R noun
R verb
Clean
R noun
R verb
RoBERTa (fine-tuned)‡87.8728170.8198556.12466
Qwen3.8-27B88.9637491.3407375.22960
OLMo 3 32B89.3707990.6307661.61949
OLMo 3 7B88.8687687.0406651.41450
Muse Glimmer 30B67.2476680.9297655.62556
GPT-4o81.5567180.1487571.42664
DeepSeek-V4-Pro74.2496488.5377357.72558
Mistral Large 3†80.7365085.6176363.62347
Grok 4.7 (reasoning)†§71.8436088.8348777.93868
Table 1. Table 1 shows the retained share of above‑chance performance for open‑weight LLMs and fine‑tuned encoders under the drop‑NOUN perturbation: 70–72 % for MNLI and 14–29 % for ANLI, with fine‑tuned encoders matching 72 % for MNLI and 24 % for ANLI.Talman, Aarne, Chatzikyriakidis, Stergios · arXiv:2611.16843v1 · Table 1: · CC BY 4.0

Source paper

Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models

Natural Language Processing
View on arXiv