How LLM inference works

LLM inference has two distinct phases with very different computational characteristics — understanding them is key to optimizing serving cost and latency.

P(w_t | w_1, \ldots, w_{t-1}) = \text{softmax}(W_U \cdot h_t)[w_t]

This two-phase split is a direct consequence of the Transformer replacing RNNs. An RNN had to process the prompt sequentially too — one token at a time, same as decode — so there was no separate 'prefill' phase to speed up; the entire prompt cost scaled linearly with GPU-unfriendly sequential steps. The Transformer's self-attention lets every prompt token be processed in parallel during prefill, which is why a 2,000-token prompt on a modern LLM can be processed in a fraction of a second, while generating 200 output tokens afterward — one sequential step per token — takes far longer despite being 10× less total computation. This asymmetry is the single most important fact to internalize about LLM serving cost.

Why decode is memory-bandwidth-bound: During decode, generating each token requires reading all model weights (~140GB for a 70B model in FP16) from GPU HBM to SRAM — but only performs a tiny amount of compute (one token's worth). An H100 has 3.35TB/s memory bandwidth but 2,000 TFLOPS. For a single-token forward pass, the model is memory-bound by ~1000×. This is why batching decode requests dramatically improves GPU utilization — amortizing the weight read cost across many concurrent tokens.

KV cache and memory management

The KV cache is the single most important memory structure in LLM inference — understanding it explains why long contexts are expensive and why batching is complex.

\text{KV cache size} = 2 \times n_{\text{layers}} \times n_{\text{heads}} \times d_{\text{head}} \times T \times B \times \text{bytes}

Inference optimization techniques

Speculative decoding in depth: Speculative decoding works because: (1) generating draft tokens with a small model (e.g., 3B) is much faster than the main model (70B), and (2) verifying k tokens in parallel with the main model is no slower than generating 1 token — the forward pass is the same shape. If the draft model has ~80% token acceptance rate, you get ~3× speedup for free with identical output. Production LLM providers use speculative decoding routinely; Medusa (self-speculative with multiple heads) avoids needing a separate draft model.

LLM inference infrastructure

Cost benchmark (2024–2025, illustrative): As a rough historical reference: GPT-4o cost roughly $10–15 per million output tokens via API in 2024; Claude 3.5 Sonnet was similar; Llama 3.1 70B via a third-party inference provider was closer to $0.88/M, and self-hosting Llama 3.1 70B on vLLM (4× A100 80GB) could get under $0.20/M at full utilization. The 50–75× gap between frontier closed models and self-hosted open-source models at that time is exactly why high-token-volume companies increasingly self-host or fine-tune open models — that dynamic still holds today even as absolute prices and specific models have moved on.

Latency vs throughput tradeoffs

Latency and throughput are fundamentally in tension for LLM serving — optimizing one hurts the other. Choosing the right operating point depends on your use case.

Streaming and perceived latency: Streaming (Server-Sent Events, SSE) returns tokens as they are generated — the user sees text appearing word-by-word rather than waiting for the full response. This dramatically improves perceived responsiveness even if total generation time is identical. A response that takes 5s to complete feels fast if you see the first tokens in 200ms. All major LLM APIs support streaming; always use it for interactive applications.

Test-time compute: when 'more inference' becomes the point

Through 2023, more inference always meant the same thing operationally: serve the same trained model faster and cheaper. Starting with OpenAI's o1 (released September 2024) and the reasoning-model wave that followed, a new class of models inverted this: instead of minimizing tokens generated, they deliberately produce a long hidden chain of thought before answering, spending far more inference compute per query in exchange for better accuracy on hard reasoning, math, and coding tasks. This is 'test-time compute' — trading inference cost for capability, the same way pretraining trades training compute for capability.

Two axes of model cost: A model's effective cost is now determined by two largely independent axes: how big it is (pretraining compute, parameter count) and how long it "thinks" before answering (test-time / inference compute). A small model that reasons for 30 seconds can beat a much larger model that answers instantly on tasks like competition math or multi-step code debugging — which is why comparing "model quality" without also stating its reasoning budget is increasingly misleading.

This has concrete consequences for serving infrastructure: reasoning traces can run to tens of thousands of tokens before the visible answer even starts, which multiplies KV cache pressure and makes PagedAttention and continuous batching more important than they were for a standard chat model that just emits a direct answer. It also changes pricing: providers increasingly bill hidden reasoning tokens separately from the visible output, since a query that reasons through 8,000 tokens before a 200-token final answer costs roughly 40× more to serve than the visible response alone would suggest.

Practice questions

1. What is the difference between throughput and latency in LLM inference, and why is there a fundamental trade-off? (Answer: Latency: time from request to first token (TTFT) + time to generate full response. User-facing — measures how fast responses feel. Throughput: total tokens generated per second across all concurrent users. Server-side — measures capacity. Trade-off: batching requests improves throughput (processes many tokens simultaneously) but increases latency for individual users (must wait for batch). At batch_size=1: minimum latency. At batch_size=256: maximum throughput. Production serving optimizes the Pareto frontier between these, using continuous batching to approach both simultaneously.)
2. What is continuous batching (iteration-level scheduling) and why did it transform LLM serving? (Answer: Traditional static batching: group N requests together, wait until ALL finish generating, then start next batch. If request 1 finishes in 10 tokens and request 2 in 1000 tokens, request 1's GPU slot sits idle for 990 token-steps. Continuous batching (Orca, vLLM): as soon as a request finishes, its slot is immediately replaced with a new request. The batch changes composition at every token generation step. Result: GPU utilization goes from ~20% (static) to ~80%+ (continuous). vLLM pioneered this; it is now the standard in all production LLM serving systems.)
3. What is TTFT (Time to First Token) and why is it more important than total generation time for user experience? (Answer: TTFT: elapsed time from request submission until the first output token is generated. Covers: network latency + prompt processing (prefill) + scheduling queue wait. User experience: TTFT determines how quickly the UI can show 'something is happening.' A response that streams from token 1 in 500ms feels faster than a response that starts in 2000ms — even if both complete in 5 seconds. This is why streaming is universal in production LLM APIs: show the first token immediately rather than waiting for completion.)
4. What hardware is used for LLM inference and what determines model serving cost? (Answer: Primary hardware: NVIDIA H100 (80GB, $30K), H200 (141GB, $40K), A100 (80GB, $10K). AMD MI300X: competitive, gaining traction. Google TPUv5: used for internal Google serving. Cost drivers: (1) GPU VRAM (must hold model weights + KV cache). (2) GPU compute (tokens/sec per GPU). (3) Memory bandwidth (memory-bandwidth-bound decoding phase). Pricing: H100 SXM: $2–3/GPU-hour on cloud. Serving a 70B model: ~4 H100s needed, ~$8-12/hour, ~100 tokens/second → $0.023–0.033/1K output tokens (similar to commercial API pricing).)
5. What is PagedAttention (used in vLLM) and how does it reduce memory waste in LLM serving? (Answer: Standard KV cache: pre-allocated contiguously for max_sequence_length. For 2048-token max: 2048 positions reserved even if request only generates 100 tokens → 95% waste. PagedAttention (Kwon et al. 2023): divides KV cache into fixed-size pages (blocks), allocating pages on demand like virtual memory. Non-contiguous pages are accessed via a block table. Result: near-zero internal fragmentation, memory utilization from 20–30% to 90%+, supports 2–4× more concurrent requests on same hardware. PagedAttention is the core innovation of vLLM.)
6. Why does a reasoning model with test-time compute sometimes cost more per query than a much larger non-reasoning model? (Answer: Total inference cost scales with total tokens processed, not just parameter count. A reasoning model may generate a hidden chain-of-thought of 5,000–20,000+ tokens before its final answer — all of that is billed compute even though the user never sees it. A larger non-reasoning model answering directly in 300 tokens can therefore cost less per query despite being a bigger, more expensive-per-token model, because it generates far fewer total tokens. This is why "cost per query" and "cost per token" can rank models in opposite orders.)