Large Language Models Use Gated Associative‑Lookup Circuits for Few‑Shot Classification
arXiv: 2611.05344v1
Researchers found that a family of 3‑4 billion‑parameter transformer models solves a few‑shot sentence‑classification task with arbitrary labels by building a key‑value memory inside the prompt. The memory is implemented as a small, gated associative‑lookup circuit. Four functional blocks compose the circuit:
1. **Writers** deposit the class of each demonstration sentence into the residual stream at the corresponding label token.
2. **Readers** attend from the query token to those label tokens whose class matches the query’s class, then copy the stored label value.
3. **Gate** activates only when the prompt contains repeated labels, preventing the readers from returning a null action when all classes appear once.
4. **Brakes** write against the most recently seen label, suppressing a bias toward the last demonstration.
Table listing the terminology for the four functional blocks of the gated associative‑lookup circuit: slot, key, value, writer heads, reader heads, query code, gate, null action, brake heads, copier heads, margin, twin, knockout, handoff, steering, hold.
Term
Meaning
slot
the residual stream at a label token, which holds a key and a value
key
the class of the sentence the label followed; written into the slot by writer heads, read through the readers’ WK
value
the identity of the label; read through the readers’ WV
writer heads
about twenty middle-layer heads that copy a sentence’s class into its label token
reader heads
about ninety upper-layer heads that attend from the query arrow to slots whose key matches the query’s class, and copy their values
query code
the representation of the query’s class that the readers match against the keys
gate g
a direction at the query arrow that signals that some label has repeated, and opens the read
null action
the readers putting their attention on the sink at position zero and copying nothing
brake heads
about two dozen heads that write against the most recently seen label
copier heads
heads in the last six layers that turn what the readers collect into logits
margin
log-probability of the correct label minus that of the other label, in nats
twin
a prompt identical to another except inside one span
knockout
mean-ablation of a head’s output
handoff
patching an activation from a twin into the intact run
steering
adding a direction to the residual stream at the dose the model itself uses
hold
fixing a downstream component at its intact value while a handoff runs
These blocks were uncovered through a series of systematic interventions. Pulling out any single reader head changed the model’s margin by only a few hundredths of a nat, while removing 90 reader heads together lowered the margin by 0.74 nats. By contrast, disabling the 24 brake heads increased the margin by 1.39 nats (±0.22), showing that brakes suppress recency bias. Similar patterns appeared in two other 3–4 b models—SmolLM3‑3B and Llama‑3.2‑3B—demonstrating that the circuit transfers across architectures and sizes.
Figure 13. No single head carries the answer. Each cell is the effect on the 12‑shot class margin of knocking out one head at the query position (mean over 144 prompts …).Gao, Andrew, Gao, William · arXiv:2611.05344v1 · Figure 13: · CC BY 4.0
The writers and readers cooperate over a two‑slot memory: each label token stores a key (the class of the preceding sentence) and a value (the label identity). In the label tokens, middle‑layer heads write the key, while upper‑layer readers use the key to find matched slots and copy the corresponding value. An early‑layer signal (the gate) checks whether any label repeats and only then allows the readers to read.
A limitation appears when prompts grow long. In 18‑demonstration configurations the softmax read law—predicting the probability that a reader will attend a slot—fails, indicating that the circuit relies on prompt length. Likewise, small models lacking the learned gate signal suffer higher rates of null action and lower accuracy.
The described phenomena were measured on three transformer families—Qwen3‑4B‑Base, SmolLM3‑3B, and Llama‑3.2‑3B—using a collection of prompt batteries, including concept, slotgate, and many‑class batteries. Margins were evaluated as the log‑probability difference between the correct and alternate label.
**Illustration 1** described above shows a table that defines the four functional blocks and the terminology used for each part of the circuit (Writers, Readers, Gate, Brakes). **Illustration 2 (Figure 13)** displays the effect of knocking out individual heads and the collective impact of the brake heads, confirming that no single reader head carries the answer while the brakes substantially improve performance.
These findings deep‑enlighten how pre‑trained language models perform in‑context learning without parameter updates, revealing a modular attention‑based memory that emulates associative lookup logic.
