A recent study shows that in deep Mamba neural networks, the very first token behaves differently from the rest. The first token tends to settle into one of a few fixed points, and its final position attracts the other tokens that follow it. This clustering, called an attention‑sink, grows stronger the deeper the network.

The researchers developed a new mathematical framework. It uses a high‑dimensional invariant box and a sharp exit‑time argument. With this approach they proved that token trajectories converge exponentially fast to their limiting points and confirmed the first token’s special role.

Simulations in the paper match the theory. In Figure 1 the trajectories of a first token converge to coordinate unit vectors ±e_d. Subsequent tokens then stick to the same attractor, creating clusters. Figure 2 shows that, across many simplified Mamba‑2 blocks, the first token ends up with only a single non‑zero element about 90 % of the time. Both figures illustrate how the first token dominates the dynamics.