First-hand information for everyone
Researchers identified a short‑lived improvement in test accuracy when linearly mixing the weights of two aligned neural networks. The effect, called BUMP (Brief Unexpected Mid‑Interpolation Peak), appears whenever the mixing coefficient λ is close to 0 or 0.9 and works across many datasets, architectures and training setups.
The technique is simple: given two permutation‑aligned models A and B, a new model is formed by mixing their weights as (1−λ) A + λ B. For λ around 0.1 or 0.9 a measurable accuracy gain appears, as shown in the experiments on CIFAR‑10 with ResNet‑20.
On the 16×‑wide ResNet‑20, BUMP boosts test accuracy by 0.181 % over the better endpoint model, while classical logit ensembling gives 0.228 %. For smaller widths (4× and 8×) the gain is smaller or even negative, although logit ensembling still achieves positive improvements.
| Accuracy gain % — Aligned pairs | ||
|---|---|---|
| Width | Fixed-λ | Test |
| 4x-width | 0.053(±0.096) | 0.087(±0.081) |
|---|
| 8x-width | 0.118(±0.111) | 0.164(±0.117) |
|---|
| 16x-width | 0.181(±0.137) | 0.227(±0.140) |
|---|
To combine the strengths of weight mixing and logit mixing, the authors proposed Weight‑Prediction Mixed Ensembling (WPME). Its logit variant reaches 0.226 % accuracy on the same data, surpassing both BUMP and standard logit ensembles, and yields better calibration (expected calibration error of 0.010).
BUMP is robust: under extreme 95 % pruning the accuracy gain grows to 1.6 % for 4× width, outpacing the dense‑model gain of 0.18 %. Conversely, moving toward random directions produces almost no gain, confirming that BUMP depends on the structure of the target model’s weights.
The method preserves most of the closer endpoint’s top‑1 predictions, with only a handful of cases switching to the runner‑up, indicating that the improved accuracy comes from subtle but consistent decision‑level adjustments rather than wholesale changes.
These findings reveal an unexpected behavior of weight interpolation that can be exploited to squeeze extra accuracy from already‑trained neural networks, but the benefit depends on network width, alignment and degree of pruning.