Why a busy GPU is not a useful GPU. Interactive figures.

Hover any series or bar for exact values. Click a legend entry to hide or show it. Bars and error bars show the mean and one standard deviation over multiple seeded replications. These are simulations of mechanisms, not measurements from real GPUs.

Batching and schedulability

Continuous (iteration level) batching turns an irregular arrival stream into regular per step work. It improves throughput and mean latency, yet the tail still grows as arrivals approach capacity.

Completed throughput versus arrival rate. Continuous batching scales while static saturates.
Latency versus arrival rate. Switch metric and axis scale above. Continuous P99 climbs near capacity.
Queueing time distribution at the reference rate. Static holds arrivals until a whole batch finishes.

KV cache lifecycle

A fixed key value capacity is the only binding constraint here and weights are excluded on purpose, so concurrency is gated by key value lifetime rather than model size. Compression converts rejected work into useful work, while eviction by value discards already spent compute.

Request outcomes by policy, stacked and averaged over seeds. Reject turns away most arrivals under pressure.
Useful output by policy. Compression leads on tokens, and eviction by value falls below reject per unit compute.
Resident key value occupancy over time, all pinned at the capacity ceiling.

Utilization versus useful tokens

Two schedulers run at almost the same utilization. Scheduler A maximizes slot utilization, Scheduler B maximizes useful value under a latency target. A stays busy far longer yet delivers a fraction of the useful tokens.

Utilization over time. Scheduler A keeps the device busy long after B has cleared the useful backlog.
Processed positions versus useful tokens per step. Nearly equal busywork, very different useful output.
Latency target violations and delivered value per unit compute for the two schedulers.
Where the wasted work goes. Scheduler A burns compute on rejected drafts and doomed requests.

Reproduce with python3 experiments.py for the static figures and python3 make_interactive.py for this page. Seeds and parameters are recorded in run_manifest.json. Aggregate values are in results_summary.csv.