Scientists have built a system called Rimuru that lets modern large language model (LLM) servers reuse memory between different kinds of tensors. Traditional servers keep model weights, attention states, and expert masks in separate memory regions. This rigid separation wastes capacity. Rimuru establishes a shared page pool to enables elastic memory sharing across heterogeneous tensors. It splits and merges pages on demand, so memory that a block of experts no longer needs can be reclaimed for a sequence cache. A special fused MoE kernel works directly with non‑contiguous expert weights, avoiding copies.

The team tested Rimuru on top of an existing LLM serving framework. In a benchmark that reflected real‑world workloads, Rimuru delivered up to 19.03‑times the throughput of baselines and cut the time to first token (TTFT) by an average of 98.1 %. Those figures were measured while keeping the same hardware and model configurations.

Rimuru’s design shows that dynamic, page‑level memory sharing improves resource use and latency. The noted speed gains come from reduced copying and the ability to let high‑memory‑usage tensors offload when demand is low. The authors do not claim any clinical effectiveness or prepare the system for commercial deployment; the study focuses specifically on throughput and TTFT under the tested conditions.