On real Claude Code traces, plain LRU beats fancy KV-cache eviction
A reproduction on 68k requests from 393 sessions finds learned eviction policies all lose to LRU — for agent inference, spend effort on capacity, not policy.
Copy markdown
LRU wins at every cache size tested
On 68,266 requests from 393 real Claude Code sessions, plain LRU held an 83.5% cache-hit rate at 8,000 blocks — and every fancier policy lost: a Bayesian 'hazard' predictor managed 82.9%, cost-aware 71.8%, session-coherent 68.6%. The clever eviction schemes from recent papers didn't beat the boring baseline on real agent traffic.
Why liveness prediction doesn't help agents
The median gap between an agent's requests is 2.1 seconds, so nearly every session looks 'about to return' — there's almost no signal for a predictor to exploit. A 5-minute TTL evictor produced byte-identical results to LRU, because under real load LRU always freed space first anyway.
Optimize capacity, not policy
The real waste is tight tool loops exhausting the cache: idle gaps under 10 seconds account for 33% of recomputed tokens, not long-idle eviction. If you run agent inference (vLLM, SGLang, your own proxy), the leverage is compression, tiering, and admission control — and first knowing whether you're TTL-bound or capacity-bound, because that regime decides everything.
You can rerun it on your own workload
The whole study reproduces from `make setup data repro` against public AgentX and Mooncake traces, and it re-derived the published Mooncake curves to within a few points before trusting its own numbers. Before you build around a caching claim, that's a rare chance to check it against traffic that looks like yours.