First-hand information for everyone
A new study shows that the way attention heads in transformer models learn to focus on syntactic relationships in text depends strongly on how many training examples the model sees. In experiments with BERT‑Small masked‑language models trained on BookCorpus and English Wikipedia, the fraction of heads that develop a clear syntactic bias rises sharply once a critical amount of data has been processed, and this change coincides with sudden drops in training loss and test error.
To understand this phenomenon, the authors constructed a solvable high‑dimensional model of multi‑head attention. In this “flipping‑spike” model, attention weights are learned by regularized empirical risk minimization (ERM) with a quadratic penalty. Theoretical analysis using the replica method shows that as the sample complexity α=N/D increases, the model undergoes a second‑order phase transition: head–feature overlap matrices split into distinct blocks, and test error shows a non‑analytic change.
Simulations of the synthetic model confirm the theory. In the ``permutation‑symmetry‑broken'' (PSB) regime occurring at low regularization, a secondary phase appears that delays specialization, producing a re‑entrant boundary in the sample‑complexity plane. This PSB phase is visible in the head‑self‑overlap plots of Figure 3.
The authors also studied a non‑isotropic Gaussian data model in which multiple latent features have different variances. Here, successive specialization transitions recover weaker features one at a time, creating a cascade of critical sample complexities α_{c}^{(r)}. Theoretical predictions agree with simulations of test error and head‑feature overlaps; the cascade is illustrated in Figure 4.

The authors note that the replica‑symmetric analysis is heuristic for this non‑convex ERM problem and may fail in some regimes, and that finite‑size experiments cannot conclusively demonstrate true non‑analyticities. Nevertheless, the sharp changes observed in both synthetic and real BERT models argue for a statistical phase‑transition interpretation of head specialization.