Large‐scale bilingual benchmark shows AI models hit 84 % accuracy on detecting text hallucinations but still miss fine‑grained errors
arXiv: 2611.08231v1
HaluCheck, a bilingual test set with more than 42,000 cases, lets researchers measure how well large language models (LLMs) spot invented facts in both written text and in multi‑turn agent conversations. The benchmark mixes carefully crafted mistruths into Wikipedia and QA passages and into real agent behavior, then confirms each example with several independent models and human checks.
Figure 2. Construction of HaluCheck. The figure shows how controlled, fine‑grained perturbations are injected into Wikipedia/QA passages and real agent trajectories, and how hallucinated samples are validated through a multi‑model synthesis pipeline.Qiu, Hanyu, Liu, Ze, Zhao, Zhe, Xiong, Kuang, Chen, Blaze, Yan, Zhenxiang et al. · arXiv:2611.08231v1 · Figure 2: · CC BY 4.0
The creators ran 14 different LLMs on the test. The best result came from Google’s Gemini3.1‑Pro, which correctly flagged wrong statements in 84.35 % of the text instances it was tested on. Its performance on agent hallucinations was lower, with the same model getting 70.07 % right for that task.
Table summarizing performance of fourteen models on HaluCheck’s HaluReal task, showing Gemini3.1‑Pro’s 84.35 % accuracy as the highest.
Model
CH
EN
HaluReal
HaluFake
HaluAgent
HaluReal
HaluFake
HaluAgent
Acc
UR
SpanF1
CR
UR
Recall
Acc
UR
RTAcc
Acc
UR
SpanF1
CR
UR
Recall
Acc
UR
RTAcc
Gemini3.1-Pro
84.35
9.40
70.07
82.27
44.60
75.70
64.55
2.20
54.93
84.87
16.63
72.95
86.00
93.20
53.40
72.28
1.35
76.16
Gemini3-Flash
77.62
0.25
59.80
72.60
1.20
96.20
50.60
0.00
54.02
81.53
0.85
60.12
72.93
12.20
93.90
72.12
0.05
72.76
GPT5.4
63.08
16.85
39.32
56.42
60.00
67.80
50.45
0.00
57.09
71.33
15.75
42.88
63.83
92.00
54.00
72.36
0.65
73.31
GPT5-Mini
63.35
43.40
43.55
57.43
7.40
94.10
50.90
0.30
58.40
65.28
55.55
42.61
57.16
6.60
96.70
70.73
0.85
70.24
DeepSeek-V4-Pro
68.57
5.45
49.33
67.96
6.80
94.00
50.00
0.00
50.10
72.10
8.50
46.99
64.57
42.20
78.90
65.80
0.00
71.83
DeepSeek-V4-Flash
70.90
7.50
43.13
58.18
55.00
66.50
52.15
0.00
56.92
70.97
4.85
38.01
44.20
78.40
60.60
68.45
0.00
72.91
HY3-Preview
62.65
19.80
39.78
58.13
63.60
65.00
50.00
0.00
51.00
62.20
23.10
36.88
50.25
85.00
57.50
65.76
0.30
73.16
Llama-3.3-70B-Instruct
59.50
9.50
19.21
45.54
28.40
64.20
45.55
0.00
57.86
63.50
15.60
25.29
40.45
58.00
70.40
70.60
0.00
70.43
Llama-3.1-8B-Instruct
52.88
7.55
16.71
18.13
29.00
63.90
49.85
0.00
51.96
54.80
1.60
23.65
21.12
10.40
90.40
52.80
0.00
54.32
Qwen3-235B-A22B
63.12
15.85
37.99
44.53
24.60
84.10
50.50
0.00
58.01
61.75
27.10
28.33
39.81
85.40
57.30
70.25
1.70
70.99
Qwen3-30B-A3B
58.03
14.65
24.94
26.61
48.80
52.80
46.05
0.00
68.07
57.55
43.80
20.41
15.88
91.40
52.90
66.93
0.45
69.64
Qwen3-32B
60.22
6.45
31.03
40.40
14.20
83.10
47.50
0.00
59.66
63.68
10.65
26.63
33.23
20.20
83.10
68.80
0.00
68.84
Qwen3-14B
57.47
6.15
26.30
33.43
9.20
89.00
50.55
0.00
59.01
58.07
17.95
22.36
20.79
54.80
70.80
67.77
0.05
66.06
Qwen3-8B
52.80
0.00
23.47
22.21
0.00
91.00
54.90
0.00
61.08
53.75
0.30
25.38
14.99
2.20
95.90
66.55
0.00
65.29
Table 3. Table 5: Main results on HaluCheck. The table lists accuracy and related metrics for fourteen LLMs on the HaluReal task, highlighting the best text‑hallucination detection accuracy of 84.35 % achieved by Gemini3.1‑Pro.Qiu, Hanyu, Liu, Ze, Zhao, Zhe, Xiong, Kuang, Chen, Blaze, Yan, Zhenxiang et al. · arXiv:2611.08231v1 · Table 3: · CC BY 4.0
Even the strongest models still struggle with more detailed tasks. Fine‑grained tracing of where a hallucination appears, correcting the wrong words, and figuring out whether an agent’s behavior matches reality all score much lower. Adding explicit reasoning steps, letting the model browse the web, or fine‑tuning it on known patterns only gives modest gains, and none of these tricks solves the problem entirely.
Large‐scale bilingual benchmark shows AI models hit 84 % accuracy on detecting text hallucinations but still miss fine‑grained errors | arXiv News
