arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Machine LearningEnglishPublished

Attention Heads in Language Models Specialize to Syntax Only When Enough Training Data Is Available

November 22, 2026arXiv: 2611.26228v1

A new study shows that the way attention heads in transformer models learn to focus on syntactic relationships in text depends strongly on how many training examples the model sees. In experiments with BERT‑Small masked‑language models trained on BookCorpus and English Wikipedia, the fraction of heads that develop a clear syntactic bias rises sharply once a critical amount of data has been processed, and this change coincides with sudden drops in training loss and test error.

To understand this phenomenon, the authors constructed a solvable high‑dimensional model of multi‑head attention. In this “flipping‑spike” model, attention weights are learned by regularized empirical risk minimization (ERM) with a quadratic penalty. Theoretical analysis using the replica method shows that as the sample complexity α=N/D increases, the model undergoes a second‑order phase transition: head–feature overlap matrices split into distinct blocks, and test error shows a non‑analytic change.

Simulations of the synthetic model confirm the theory. In the ``permutation‑symmetry‑broken'' (PSB) regime occurring at low regularization, a secondary phase appears that delays specialization, producing a re‑entrant boundary in the sample‑complexity plane. This PSB phase is visible in the head‑self‑overlap plots of Figure 3.

Figure 3. Phase diagram of the synthetic model. Left panel: phase boundaries versus sample complexity and regularization λ. The diagram labels the permutation‑symmetric (PS), permutation‑symmetry‑broken (PSB), and specialized (SPEC) phases. The PSB region appears at low λ and pushes the boundary for full specialization to higher α. Center panel: head‑self‑overlaps q11 and q22 for the two heads. Right panel: head‑feature overlaps m12 and m22. The panels collectively illustrate that, over the PSB region, mutual overlaps remain small while self‑overlaps split, and that the boundary between PSB and SPEC is marked by a non‑analytic change in test error.
Figure 3. Phase diagram of the flipping‑spike model showing the emergence of a permutation‑symmetry‑broken phase at low regularization and the re‑entrant specialization boundary at low sample complexity.Catteau, Léo, Duranthon, O., Zdeborová, Lenka · arXiv:2611.26228v1 · Figure 3: · CC BY 4.0

The authors also studied a non‑isotropic Gaussian data model in which multiple latent features have different variances. Here, successive specialization transitions recover weaker features one at a time, creating a cascade of critical sample complexities α_{c}^{(r)}. Theoretical predictions agree with simulations of test error and head‑feature overlaps; the cascade is illustrated in Figure 4.

Figure 4. Phase diagram of the non‑isotropic Gaussian model. Left panel: phase boundaries versus sample complexity α and anisotropy δ. Red line indicates a cut at fixed δ=1/9. Center panel: test error ℰ and training error ℰ_train along the cut, showing non‑analyticities at each α^(r)_c. Right panel: head‑feature overlaps m_h^f for each head and latent feature, indicating that features are recovered one by one as α increases. The three sub‑panels together display the successive specialization transitions and the associated cascade of critical points.
Figure 4. Cascade of specialization transitions in the non‑isotropic Gaussian model: critical sample complexities α^(r)_c for each feature strength r are plotted and marked, with corresponding test errors and head‑feature overlaps. Catteau, Léo, Duranthon, O., Zdeborová, Lenka · arXiv:2611.26228v1 · Figure 4: · CC BY 4.0

The authors note that the replica‑symmetric analysis is heuristic for this non‑convex ERM problem and may fail in some regimes, and that finite‑size experiments cannot conclusively demonstrate true non‑analyticities. Nevertheless, the sharp changes observed in both synthetic and real BERT models argue for a statistical phase‑transition interpretation of head specialization.

Source paper

Attention Head Specialization: Phase Transitions in Sample Complexity

Machine Learningcond-mat.dis-nnStatistical ML
View on arXiv
Attention Heads in Language Models Specialize to Syntax Only When Enough Training Data Is Available | arXiv News