arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
cs.DCEnglishPublished

Aggron trims growing memory in AI agents, cutting latency and lifting throughput

November 21, 2026arXiv: 2611.26435v1

A new serving system called Aggron grabs long‑running AI agents and keeps their memory use in check by summarizing recent history at every turn. In typical AI agents, every dialogue turn adds to a context known as a KV cache; this growth limits how many agents can run simultaneously. Aggron sidesteps the size problem by launching summarization jobs from a side queue while the main agent request is stalled at the front of the normal queue. The side queue uses unused compute and memory that would otherwise sit idle during those stalls.

When the system predicts that a summarization will finish before the stalled agent resumes, Aggron compresses only the parts of the cache that would actually free up space, allowing the agent to continue. Across a range of high‑load workloads, this approach cut the 99th‑percentile program latency by 27 % to 31 % and raised overall throughput by about 34 % to 40 %. The gains come from compressing at every turn rather than waiting for a large backlog, which lets the system keep the GPU hardware busy.

However, when the queue is short, the gating logic that decides when to compress can’t hide the small delay introduced by the extra summarization step. In those cases, throughput fell by only 0.2 % to 3.8 % and 99th‑percentile latency rose by 1.3 % to 47 %. Table 3 shows that, at a load of 0.2 requests per second, 83 % of the run time the head of the queue was blocked; compressing the tail of the head’s context could free about 2.4 K tokens and allow a new request to start in roughly 46 % of that blocked time.

Table displaying three columns—head‑blocked percentage of run, proportion of blocked time that could be admitted by compressing the tail, and proportion of blocked time with at least one admitted—increasing with request rate.
λ (req/s)Head blockedCompressed tail admits≥1 admitted
(% of run)(% of blocked)(% of blocked)
0.058.7%4.8%12.8%
0.174.8%4.5%44.0%
0.283.0%4.9%46.0%
0.588.7%5.0%46.5%
186.2%5.0%46.2%
Table 3. Table 3 shows queue‑blocking statistics under a 4 K‑token prose cap: at a load of 0.2 req/s, the head request was blocked for 83 % of the run, and compressing its tail would allow a new request to start in roughly 46 % of that blocked time.Ravi, Rishabh, Hong, Tony, Wang, Chen, Zhu, Yue, Ye, Mengmei, Franke, Hubertus et al. · arXiv:2611.26435v1 · Table 3. · CC BY 4.0

Thus, Aggron demonstrates a practical way to keep AI agents’ memory footprints manageable while improving service performance, especially under heavy traffic, though it incurs modest penalties when the system is lightly loaded.

Source paper

Aggron: Aggressive Compression of Agentic KV Caches during Head-of-Line Stalls in the Request Queue

cs.DCcs.AR
View on arXiv
Aggron trims growing memory in AI agents, cutting latency and lifting throughput | arXiv News