Why a busy GPU is not a useful GPU. Interactive figures.

Hover any series or bar for exact values. Click a legend entry to hide or show it. Bars and error bars show the mean and one standard deviation over twelve seeded replications. These are simulations of mechanisms, not measurements from real GPUs.

Batching and schedulability

Continuous batching admits new work every decode step. We compare it against a right-sized dynamic batcher that forms a full cohort and returns each request as soon as it finishes, and against a naive run-to-slowest static batcher kept only as a labeled reference. Continuous leads, yet its honest margin is over the right-sized batcher, not over the strawman.

Completed throughput versus arrival rate. Continuous keeps scaling, the right-sized batcher saturates at its cohort barrier, and naive static stays a flat reference.
Latency versus arrival rate. Switch metric and axis scale above. Continuous P99 still climbs near capacity, so continuous batching does not repeal queueing theory.
Queueing time distribution at the reference rate. Naive static holds arrivals until a whole batch finishes, the right-sized batcher waits only for a cohort, and continuous admits almost at once.

KV cache lifecycle and the cost of compression

A fixed key value capacity is the only binding constraint here and weights are excluded on purpose, so concurrency is gated by key value lifetime rather than model size. Compression admits more work, but it is a tradeoff and not a free win, because compressed context loses quality and the merge itself costs compute.

Request outcomes by policy, stacked and averaged over seeds. Reject turns away many arrivals under pressure.
Useful output by policy at the base latency target and the base quality penalty. Compression still leads on tokens, yet its useful tokens per unit compute now trails reject once the penalty is charged.
Useful tokens per unit compute across a sweep of SLA latency targets. All three policies separate, so the metric discriminates everywhere, not only for eviction.
Useful tokens delivered as the compression quality penalty grows, with reject as a flat reference. Compression wins only below a penalty near four tenths, then falls behind.
Resident key value occupancy over time for one seed, all pinned at the capacity ceiling.

Utilization versus useful tokens

Two schedulers run at almost the same utilization and the same speculation depth. Scheduler A maximizes slot utilization, Scheduler B drops doomed requests and admits by value under a latency target. A stays busy yet delivers a fraction of the useful tokens. The decomposition attributes that gap to one lever at a time.

Utilization over time for one seed. Scheduler A keeps the device busy long after B has cleared the useful backlog.
Processed positions versus useful tokens per step. Nearly equal busywork, very different useful output.
Latency target violations and delivered value per unit compute for the two schedulers.
Where the wasted work goes. Scheduler A burns compute on rejected drafts and doomed requests.
Goodput decomposition, one lever at a time from the busy baseline at equal speculation unless noted. Drop-doomed dominates, value-order alone trades token count for delivered value, and speculation depth is a small effect.

Reproduce with python3 experiments.py for the static figures and python3 make_interactive.py for this page. Seeds and parameters are recorded in run_manifest.json. Aggregate values are in results_summary.csv.