Logical KV savings are not physical memory savings. Interactive figures.
Paged KV cache runtimes allocate and free memory in whole blocks, so the unit of reuse is a fully free block, not a single dead token. Hover any point or bar for exact values, click a legend entry to hide a series, and use the controls to switch distributions or axis scales. These are memory-accounting simulations, not measurements from a real GPU or model.
Logical saving versus physical reclaim
Under random scattered eviction a block is freed only when all B of its tokens are evicted, so the freed fraction is qB while the logical saving is only q. Reclaim efficiency, the physical reclaim earned per unit of logical saving, is qB−1 and collapses as the block grows.
Logical saving (dashed) versus physical block reclaim qB. At B=16 a 50% token cut frees about 0.0015% of blocks. Toggle a log axis to see the smaller block sizes.Reclaim efficiency qB−1. Even a large token cut earns almost no reusable blocks at production block sizes until the eviction ratio is very high.
Eviction strategy decides physical reclaim
Four strategies remove the same number of tokens, so logical saving is held fixed and only the physical consequences differ. Lowest-importance eviction minimizes quality loss yet frees no blocks. Block-wise eviction frees blocks at higher quality loss. Reclaimability-aware eviction sits between them.
Physical reclaim by strategy at q=0.5. The dashed line is the logical saving. Random and lowest-importance token eviction sit near zero.Quality versus reclaim tradeoff. Lowest-importance is low-loss but low-reclaim, block-wise is high-reclaim but higher-loss, reclaimability-aware occupies the favorable interior.Internal fragmentation of surviving blocks. Scattered strategies strand about half of each live block. Block-wise eviction leaves near zero.Physical reclaim as the eviction ratio grows (lognormal importance). Block-wise tracks the ideal, reclaimability-aware trails it, scattered strategies stay flat until very high ratios.
Compaction cost and benefit
Compaction moves survivor entries into fewer blocks and frees the emptied ones. Its copy cost depends on how scattered the survivors are. Block-aligned holes reach the minimal footprint with little or no copying.
Where free blocks come from. Scattered patterns get free blocks only through copy-heavy compaction. Clustered and block-coherent holes are already free at eviction.Compaction ROI, reclaimed capacity per token copied, on a log axis. Higher means cheaper physical reclaim. Block-aligned patterns are orders of magnitude cheaper.Survivor tokens copied to reach the minimal footprint. This is the bandwidth bill of compaction, and it is large exactly for scattered eviction.
New request admission
A full pool of 512 blocks is evicted by 50% and new requests each need 8 free blocks. The logical saving is identical across strategies, yet admission is not, because admission is gated by physically free blocks.
New requests admitted after an identical 50% logical eviction. The dashed line is what logical saving alone would predict. Random and lowest-importance admit zero.Admission as the eviction ratio grows. Block-wise and random-plus-compaction overlap at the ceiling. Use the legend to separate overlapping lines.
Reproduce with python kv_reclaimability_blog_experiments.py, which writes the static figures, results_summary.csv, and the interactive_data.json that drives this page. Fixed seed, numpy, pandas, matplotlib only. Importance scores are synthetic proxies for token value, not attention weights from a real model.