First-hand information for everyone
A new serving system called Aggron grabs long‑running AI agents and keeps their memory use in check by summarizing recent history at every turn. In typical AI agents, every dialogue turn adds to a context known as a KV cache; this growth limits how many agents can run simultaneously. Aggron sidesteps the size problem by launching summarization jobs from a side queue while the main agent request is stalled at the front of the normal queue. The side queue uses unused compute and memory that would otherwise sit idle during those stalls.
When the system predicts that a summarization will finish before the stalled agent resumes, Aggron compresses only the parts of the cache that would actually free up space, allowing the agent to continue. Across a range of high‑load workloads, this approach cut the 99th‑percentile program latency by 27 % to 31 % and raised overall throughput by about 34 % to 40 %. The gains come from compressing at every turn rather than waiting for a large backlog, which lets the system keep the GPU hardware busy.
However, when the queue is short, the gating logic that decides when to compress can’t hide the small delay introduced by the extra summarization step. In those cases, throughput fell by only 0.2 % to 3.8 % and 99th‑percentile latency rose by 1.3 % to 47 %. Table 3 shows that, at a load of 0.2 requests per second, 83 % of the run time the head of the queue was blocked; compressing the tail of the head’s context could free about 2.4 K tokens and allow a new request to start in roughly 46 % of that blocked time.
| λ (req/s) | Head blocked | Compressed tail admits | ≥1 admitted |
|---|---|---|---|
| (% of run) | (% of blocked) | (% of blocked) | |
| 0.05 | 8.7% | 4.8% | 12.8% |
| 0.1 | 74.8% | 4.5% | 44.0% |
| 0.2 | 83.0% | 4.9% | 46.0% |
| 0.5 | 88.7% | 5.0% | 46.5% |
| 1 | 86.2% | 5.0% | 46.2% |
Thus, Aggron demonstrates a practical way to keep AI agents’ memory footprints manageable while improving service performance, especially under heavy traffic, though it incurs modest penalties when the system is lightly loaded.