Low‑rank adaptation (LoRA) often fails to use the capacity it is configured for. Experiments show that the effective rank of the adapter’s update is far below the set nominal rank and gets worse when the rank is increased.

To fix this, authors add a differentiable, SVD‑free Rényi‑2 effective‑rank penalty to the language‑modeling loss. The penalty encourages the adapter to spread its updates over a higher dimensional subspace.

In mixed‑domain continual‑learning fine‑tuning, using a penalty weight of λ = 0.01 consistently reduces mean Wiki perplexity at every tested rank of at least 16. Training perplexity is similar or slightly better compared to unpenalized LoRA, and the penalty raises the reported effective rank.

The benefit depends on the metric and setting. The regularizer still requires tuning λ for each task and might not help when deliberate under‑fitting or very narrow‑domain retention is desired.

This work demonstrates that explicit capacity maximization can narrow the learning‑forgetting trade‑off for LoRA adaptations.