How LLM inference works

LLM inference has two distinct phases with very different computational characteristics — understanding them is key to optimizing serving cost and latency.

P(w_t | w_1, \ldots, w_{t-1}) = \text{softmax}(W_U \cdot h_t)[w_t]

Why decode is memory-bandwidth-bound: During decode, generating each token requires reading all model weights (~140GB for a 70B model in FP16) from GPU HBM to SRAM — but only performs a tiny amount of compute (one token's worth). An H100 has 3.35TB/s memory bandwidth but 2,000 TFLOPS. For a single-token forward pass, the model is memory-bound by ~1000×. This is why batching decode requests dramatically improves GPU utilization — amortizing the weight read cost across many concurrent tokens.

KV cache and memory management

The KV cache is the single most important memory structure in LLM inference — understanding it explains why long contexts are expensive and why batching is complex.

\text{KV cache size} = 2 \times n_{\text{layers}} \times n_{\text{heads}} \times d_{\text{head}} \times T \times B \times \text{bytes}

Inference optimization techniques

Speculative decoding in depth: Speculative decoding works because: (1) generating draft tokens with a small model (e.g., 3B) is much faster than the main model (70B), and (2) verifying k tokens in parallel with the main model is no slower than generating 1 token — the forward pass is the same shape. If the draft model has ~80% token acceptance rate, you get ~3× speedup for free with identical output. Claude uses speculative decoding in production; Medusa (self-speculative with multiple heads) avoids needing a separate draft model.

LLM inference infrastructure in 2025

Cost benchmark (early 2025): GPT-4o: ~$10–15/M output tokens. Claude 3.5 Sonnet: ~$15/M. Llama 3.1 70B via Together AI: ~$0.88/M. Self-hosted Llama 3.1 70B on vLLM (4× A100 80GB): ~$0.20/M at full utilization. The 50–75× cost gap between frontier closed models and self-hosted open-source explains why companies with high token volumes increasingly fine-tune open-source models for production.

Latency vs throughput tradeoffs

Latency and throughput are fundamentally in tension for LLM serving — optimizing one hurts the other. Choosing the right operating point depends on your use case.

Streaming and perceived latency: Streaming (Server-Sent Events, SSE) returns tokens as they are generated — the user sees text appearing word-by-word rather than waiting for the full response. This dramatically improves perceived responsiveness even if total generation time is identical. A response that takes 5s to complete feels fast if you see the first tokens in 200ms. All major LLM APIs (OpenAI, Anthropic, Groq) support streaming; always use it for interactive applications.

Practice questions

1. What is the difference between throughput and latency in LLM inference, and why is there a fundamental trade-off? (Answer: Latency: time from request to first token (TTFT) + time to generate full response. User-facing — measures how fast responses feel. Throughput: total tokens generated per second across all concurrent users. Server-side — measures capacity. Trade-off: batching requests improves throughput (processes many tokens simultaneously) but increases latency for individual users (must wait for batch). At batch_size=1: minimum latency. At batch_size=256: maximum throughput. Production serving optimizes the Pareto frontier between these, using continuous batching to approach both simultaneously.)
2. What is continuous batching (iteration-level scheduling) and why did it transform LLM serving? (Answer: Traditional static batching: group N requests together, wait until ALL finish generating, then start next batch. If request 1 finishes in 10 tokens and request 2 in 1000 tokens, request 1's GPU slot sits idle for 990 token-steps. Continuous batching (Orca, vLLM): as soon as a request finishes, its slot is immediately replaced with a new request. The batch changes composition at every token generation step. Result: GPU utilization goes from ~20% (static) to ~80%+ (continuous). vLLM pioneered this; it is now the standard in all production LLM serving systems.)
3. What is TTFT (Time to First Token) and why is it more important than total generation time for user experience? (Answer: TTFT: elapsed time from request submission until the first output token is generated. Covers: network latency + prompt processing (prefill) + scheduling queue wait. User experience: TTFT determines how quickly the UI can show 'something is happening.' A response that streams from token 1 in 500ms feels faster than a response that starts in 2000ms — even if both complete in 5 seconds. This is why streaming is universal in production LLM APIs: show the first token immediately rather than waiting for completion.)
4. What hardware is used for LLM inference and what determines model serving cost? (Answer: Primary hardware: NVIDIA H100 (80GB, $30K), H200 (141GB, $40K), A100 (80GB, $10K). AMD MI300X: competitive, gaining traction. Google TPUv5: used for internal Google serving. Cost drivers: (1) GPU VRAM (must hold model weights + KV cache). (2) GPU compute (tokens/sec per GPU). (3) Memory bandwidth (memory-bandwidth-bound decoding phase). Pricing: H100 SXM: $2–3/GPU-hour on cloud. Serving a 70B model: ~4 H100s needed, ~$8-12/hour, ~100 tokens/second → $0.023–0.033/1K output tokens (similar to commercial API pricing).)
5. What is PagedAttention (used in vLLM) and how does it reduce memory waste in LLM serving? (Answer: Standard KV cache: pre-allocated contiguously for max_sequence_length. For 2048-token max: 2048 positions reserved even if request only generates 100 tokens → 95% waste. PagedAttention (Kwon et al. 2023): divides KV cache into fixed-size pages (blocks), allocating pages on demand like virtual memory. Non-contiguous pages are accessed via a block table. Result: near-zero internal fragmentation, memory utilization from 20–30% to 90%+, supports 2–4× more concurrent requests on same hardware. PagedAttention is the core innovation of vLLM.)