LLM Serving at Scale: PagedAttention, RadixAttention, and Production Inference

For production LLM systems, serving efficiency is a primary determinant of cost and latency. The dominant constraint is memory: the attention key–value (KV) cache grows with sequence length and batch size, and its management governs how many requests a given accelerator can serve concurrently. This article surveys three serving systems—vLLM, SGLang, and Text Generation Inference—and the algorithms that distinguish them.

The inference problem

Production serving is not simply model invocation; it entails request batching, KV-cache memory management, scheduling across concurrent requests, and a throughput–latency trade-off. Inefficiency in any of these translates directly into higher cost per token or degraded tail latency.

vLLM and PagedAttention

vLLM [1] introduced PagedAttention [2], which applies virtual-memory paging to the KV cache. Conventional serving allocates a contiguous KV region per request, sized for the maximum sequence length, which wastes memory on shorter sequences. PagedAttention instead allocates the cache in fixed-size blocks (pages) mapped through an indirection table, reducing internal fragmentation to the last block of each sequence.

The vLLM authors report that, with PagedAttention, memory waste is confined to under 4% of the cache, and that vLLM achieves up to 24× higher throughput than HuggingFace Transformers and up to 3.5× higher than the prior Text Generation Inference baseline, on LLaMA-7B (A10G) and LLaMA-13B (A100) workloads sampled from ShareGPT [1]. The PagedAttention algorithm and its evaluation are described in the accompanying SOSP 2023 paper [2].

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3-70B-Instruct")
outputs = llm.generate(
    prompts=["Explain quantum computing"],
    sampling_params=SamplingParams(max_tokens=500),
)

SGLang and RadixAttention

SGLang [3] targets structured generation and introduces RadixAttention, which caches shared prefixes across requests in a radix tree, improving reuse for workloads with common prefixes (few-shot prompts, shared system messages, and tree-structured generation). SGLang additionally provides primitives for constrained decoding (regular expressions, JSON, grammars) and control flow:

from sglang import function, gen, select

@function
def chain_of_thought(question: str):
    s = f"Question: {question}\nLet me think step by step.\n"
    s += gen("reasoning", max_tokens=200)
    s += "\nTherefore, the answer is: "
    s += select("answer", ["A", "B", "C", "D"])
    return s

The design and evaluation of SGLang and RadixAttention are reported in the accompanying paper [3].

Text Generation Inference

Text Generation Inference (TGI) [4], from Hugging Face, emphasises operational maturity: container-first deployment, an OpenAI-compatible API, and integration with the Hugging Face Hub. It is frequently selected where deployment simplicity and ecosystem integration are weighted above peak throughput.

services:
  tgi:
    image: ghcr.io/huggingface/text-generation-inference
    command: --model-id meta-llama/Llama-3-70B-Instruct

Comparative characteristics

Rather than reproduce cross-system throughput figures—which are highly sensitive to hardware, model, sequence-length distribution, and configuration, and are rarely comparable across independent reports—the systems are better summarised by their design emphases:

Teams for whom serving performance is material should benchmark candidate systems on their own model and traffic distribution, since published numbers seldom transfer.

Self-hosting versus managed APIs

The decision to self-host inference trades operational burden for control over cost, latency, and model choice:

The break-even point is workload-specific; at sufficient sustained volume, the per-token economics favour self-hosting, but only where the operational capacity to run it exists.

Integration with orchestration

Inference is one layer of a larger system. A robust architecture places a gateway and load balancer between the orchestration layer and one or more serving instances, and—critically—handles inference failures (timeouts, rate limits) at the workflow level through retries and checkpointing rather than by restarting the enclosing workflow.

graph TB
    WF[Workflow engine] --> GW[Gateway / load balancer]
    GW --> V1[vLLM instance 1]
    GW --> V2[vLLM instance 2]

Discussion

Progress in LLM serving over this period has been driven primarily by memory management—PagedAttention and RadixAttention being the salient examples—rather than by changes to the models themselves. Emerging directions include speculative decoding, disaggregated prefill/decode serving, and improved continuous-batching schedulers. The consistent lesson is that inference should be treated as a first-class infrastructure concern.

References

1. Kwon, W. et al., "vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention" (project blog, 2023). https://blog.vllm.ai/2023/06/20/vllm.html
2. Kwon, W. et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP 2023; arXiv:2309.06180. https://arxiv.org/abs/2309.06180
3. Zheng, L. et al., "SGLang: Efficient Execution of Structured Language Model Programs," arXiv:2312.07104. https://arxiv.org/abs/2312.07104
4. Hugging Face, Text Generation Inference (documentation). https://huggingface.co/docs/text-generation-inference