Logical KV Savings Are Not Physical Memory Savings
Token-level KV cache eviction can report large logical savings while returning little usable GPU memory to a paged allocator. How much it actually returns is...
I received my PhD from the ECSE Department at Rensselaer Polytechnic Institute, where I was advised by Tong Zhang.
Before that, I earned my BE from Southern University of Science and Technology.
My research focuses on memory architecture and computer systems, with a particular interest in DRAM and SSDs. I work on improving performance and efficiency for data processing and AI infrastructure.
Recent Posts
Why a Busy GPU Is Not a Useful GPU in LLM Serving
A reproducible mechanism study of schedulability, KV-cache lifecycle, and useful tokens per dollar in LLM serving, with a controlled scheduler ablation and h...
The Case for System-Closed Reliability in the HBM Supply Chain
Thoughts on a possible path for scaling HBM bandwidth through system-level reliability