Large language models keep most of their MNLI performance above chance after noun removal, but drop sharply on ANLI
arXiv: 2611.16843v1
Recent research tested eight large language models (LLMs) and fine‑tuned encoders by removing nouns from input data. On the standard natural language inference set MNLI, LLMs retained 70–72 % of their above‑chance accuracy even after noun removal. The same perturbation left only 14–29 % of their ANLI performance intact. For the more recent ANLI benchmark, noun removal caused a sharper decline, with only 14–29 % of performance preserved. Fine‑tuned encoders showed a similar pattern: they kept 72 % of MNLI performance but only 24 % on ANLI. These figures come from a controlled experiment that varied the perturbation—drop‑NOUN, drop‑VERB, keep‑NOUN+VERB, and shuffled word order—and measured how much accuracy remained relative to a clean baseline. The results suggest that LLMs rely less on simple noun cues in MNLI but are more sensitive to perturbations on ANLI, pointing to dataset‐specific shortcuts rather than true language understanding.
Table displaying accuracy and retained performance for each model on MNLI and ANLI when nouns are removed
MNLI
RTE
ANLI (R1–R3 average)
System
Clean
R noun
R verb
Clean
R noun
R verb
Clean
R noun
R verb
RoBERTa (fine-tuned)‡
87.8
72
81
70.8
19
85
56.1
24
66
Qwen3.8-27B
88.9
63
74
91.3
40
73
75.2
29
60
OLMo 3 32B
89.3
70
79
90.6
30
76
61.6
19
49
OLMo 3 7B
88.8
68
76
87.0
40
66
51.4
14
50
Muse Glimmer 30B
67.2
47
66
80.9
29
76
55.6
25
56
GPT-4o
81.5
56
71
80.1
48
75
71.4
26
64
DeepSeek-V4-Pro
74.2
49
64
88.5
37
73
57.7
25
58
Mistral Large 3†
80.7
36
50
85.6
17
63
63.6
23
47
Grok 4.7 (reasoning)†§
71.8
43
60
88.8
34
87
77.9
38
68
Table 1. Table 1 shows the retained share of above‑chance performance for open‑weight LLMs and fine‑tuned encoders under the drop‑NOUN perturbation: 70–72 % for MNLI and 14–29 % for ANLI, with fine‑tuned encoders matching 72 % for MNLI and 24 % for ANLI.Talman, Aarne, Chatzikyriakidis, Stergios · arXiv:2611.16843v1 · Table 1: · CC BY 4.0