First-hand information for everyone
Researchers have discovered that training deep residual networks (ResNets) with intermediate exit points leads to a "turnpike" phenomenon: hidden‑state representations inside the network quickly converge to a common region of the loss landscape, regardless of how many layers the network has. This means that deeper models do not need more layers to reach a given loss level, and intermediate representations across models of different depth become similar.
The discovery stems from recasting the standard joint training objective— a weighted sum of cross‑entropy losses for the final output and each intermediate exit head—as a discrete‑time optimal control problem (OCP). In this view, each residual block is a dynamic step, and the control inputs are the trainable parameters. The OCP formulation shows that for a δ‑suboptimal solution, the overall training objective is bounded above by a constant that does not depend on network depth or exit placement. Thus, the loss turnpike bound is independent of how deep the network is.
| Optimal Control Problem min𝐰JN(𝐰)=∑k=0N−1ℓk(𝐱k,𝐰k)+Jf(𝐱N)(3a)s.t.k=0,…,N−1𝐱k+1=fk(𝐱k,𝐰k),(3b)𝐱0=𝐱0.(3c) |
| Optimal Control Problem |
| min𝐰JN(𝐰)=∑k=0N−1ℓk(𝐱k,𝐰k)+Jf(𝐱N)(3a)s.t.k=0,…,N−1𝐱k+1=fk(𝐱k,𝐰k),(3b)𝐱0=𝐱0.(3c) |
| Optimal Control Problem |
| min𝐰JN(𝐰)=∑k=0N−1ℓk(𝐱k,𝐰k)+Jf(𝐱N)(3a)s.t.k=0,…,N−1𝐱k+1=fk(𝐱k,𝐰k),(3b)𝐱0=𝐱0.(3c) |
Empirical studies on CIFAR‑10 confirm that this convergence happens on the training data, with a generalization gap of about 7 percentage points on the test set. The authors also note that while the theoretical guarantees hold strictly for training data, test accuracies still exhibit similar convergence patterns, and using fewer exits per several layers can slightly improve test performance.
A figure in the paper (see Figure 3) illustrates the mathematical equivalence between the joint‑exit training objective and the OCP formulation, making clear how the residual‑block dynamics and exit losses fit into the control framework.
The work provides a new theoretical lens on why early‑exit criteria based on softmax entropy or maximum probability tend to work, linking them to the overthinking problem in deep networks.