Production LLM Inference at Scale: vLLM, SGLang, and TensorRT-LLM on GPU Cloud

The Economics and Physics of Large Language Model Serving

Transitioning a Large Language Model (LLM) from an offline prototype or Jupyter Notebook into a high-concurrency production API is one of the most demanding engineering challenges in software infrastructure today. Unlike traditional web applications where CPU utilization scales linearly with HTTP requests, LLM inference is governed by strict laws of silicon physics: memory bandwidth constraints, quadratic attention scaling, and Key-Value (KV) cache memory bloat.

Naive deployment using Hugging Face Transformers or basic PyTorch pipelines quickly collapses under concurrent user loads. On an 80GB GPU, up to 70% of high-bandwidth memory can be squandered by internal memory fragmentation and static buffer pre-allocations, causing out-of-memory (OOM) crashes with as few as ten simultaneous users.

To overcome these bottlenecks, modern LLM inference engines—most notably vLLM, SGLang, and NVIDIA TensorRT-LLM—have re-engineered the serving paradigm. This comprehensive guide breaks down the architectural inner workings of high-performance serving, details the exact hardware requirements across Runpod, Lambda, and CoreWeave, and provides battle-tested production configurations for high-throughput LLM deployments.

The Three Pillars of Modern LLM Serving Engines

1. PagedAttention & Dynamic KV Cache Allocation

Introduced by the UC Berkeley team behind vLLM, PagedAttention is inspired by virtual memory paging in operating systems. In standard autoregressive generation, the model must store the Key and Value vectors for every previous token in the conversation to compute the next token attention. Because sequence lengths are unpredictable, legacy systems statically pre-allocated memory for the maximum possible context length (e.g., 8,192 tokens), wasting up to 80% of VRAM.

PagedAttention breaks the KV cache into small, fixed-size memory blocks (typically 16 or 32 tokens). Blocks do not need to reside in contiguous physical memory. The engine maintains a virtual block table, allocating physical memory blocks on-demand as tokens are generated. This slashes memory waste from 70%+ down to under 4%, allowing you to fit 3x to 5x more concurrent requests into the exact same GPU hardware.

2. Continuous (Iteration-Level) Batching

In traditional static batching, an inference engine groups eight requests together and processes them until every request finishes generating. If seven requests complete in 50 tokens while one request takes 1,000 tokens, the GPU sits mostly idle for the remaining 950 steps, wasting compute cycles.

Continuous Batching operates at the single-token iteration level. As soon as a request completes generation, its memory is freed and a newly arrived request is injected into the next execution step immediately. This maintains near-100% GPU tensor core saturation, boosting aggregate token throughput by 300% to 400%.

3. Chunked Prefill & FP8 Quantization

LLM requests consist of two phases: prefill (processing the prompt, compute-bound) and decode (generating new tokens one by one, memory-bandwidth-bound). A massive 10,000-token prompt can freeze token generation for active users, causing severe latency spikes (Inter-Token Latency or ITL).

Chunked Prefill chops large prompt sequences into manageable chunks (e.g., 512 tokens), interleaving prompt processing with token decoding steps. Combined with modern FP8 (W8A8) quantization on Hopper architectures (NVIDIA H100 and H200), memory consumption is halved with virtually zero degradation in model reasoning capability.

Engine Face-Off: vLLM vs. SGLang vs. TensorRT-LLM

Hardware Sizing: VRAM & KV Cache Calculator

To prevent out-of-memory crashes and calculate optimal GPU cluster sizing, use the following empirical formula for memory allocation:

Total VRAM Required = Model Weight Memory + KV Cache Memory + Activation Overhead

1. Model Weight Memory:
   - FP16 (16-bit): ~2.0 GB per billion parameters
   - FP8 (8-bit):   ~1.0 GB per billion parameters
   - INT4 (4-bit):  ~0.6 GB per billion parameters

2. KV Cache Memory per Token (Bytes) = 
   2 * n_layers * n_kv_heads * head_dim * precision_bytes

Example: Llama-3-70B (FP8 precision, 80 layers, 8 KV heads, 128 head_dim, 1 byte/val):
KV Cache per token = 2 * 80 * 8 * 128 * 1 = 163,840 bytes = 160 KB per token.
For 100 concurrent requests with 4,096 context tokens:
100 * 4,096 * 160 KB = 65,536,000 KB = ~65.5 GB dedicated strictly to KV cache!

From this formula, it is immediately clear why a 70B model running at high concurrency cannot fit on a single 80GB GPU. Even in FP8 (which consumes 70GB for weights alone), there is barely 10GB left for KV cache, limiting you to only a few concurrent streams before hitting OOM. You require multi-GPU Tensor Parallelism.

Multi-GPU Tensor Parallelism (TP) Configurations

When distributing an LLM across multiple GPUs using Tensor Parallelism (TP), the weight matrices of each transformer layer are partitioned across cards. During each layer’s forward pass, an all-reduce communication step must occur across all participating GPUs.

• TP=1 (Single GPU): Best for models up to 14B parameters. Zero inter-GPU communication latency. Supported on single RTX 4090 (24GB) or A100 (80GB).
• TP=2 (Two GPUs): Optimal for 27B–34B models, or 70B models on 2x H200 141GB GPUs. Requires high-speed NVLink (600–900 GB/s). Avoid running TP over standard PCIe slots without NVLink bridges.
• TP=4 (Four GPUs): The gold-standard sweet spot for serving Llama-3-70B in FP8 across four NVIDIA H100 SXM5 cards ($10.76/hr combined).
• TP=8 (Eight GPUs): Required for massive 405B models or high-concurrency unquantized FP16 70B models with 32K+ context windows. Must run on a single physical node with full all-to-all NVLink mesh (e.g. AWS P5 or Lambda 8x H100 node).

Battle-Tested Production Deployment: vLLM Docker Compose

Deploying vLLM in production requires tuning specific GPU flags to extract maximum performance. Here is a production-ready docker-compose.yml file engineered for a 4x NVIDIA H100 SXM5 instance on Runpod or Lambda:

version: '3.8'
services:
  vllm-server:
    image: vllm/vllm-openai:latest
    container_name: production-vllm-llama3
    runtime: nvidia
    restart: always
    environment:
      - HUGGING_FACE_HUB_TOKEN=hf_YOUR_SECRET_TOKEN
      - NCCL_DEBUG=INFO
      - VLLM_ATTENTION_BACKEND=FLASH_ATTN
    ports:
      - "8000:8000"
    volumes:
      - /workspace/hf_cache:/root/.cache/huggingface
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 4
              capabilities: [gpu]
    command: >
      --model meta-llama/Meta-Llama-3.1-70B-Instruct
      --tensor-parallel-size 4
      --dtype auto
      --kv-cache-dtype fp8
      --max-model-len 8192
      --gpu-memory-utilization 0.94
      --swap-space 16
      --enable-chunked-prefill
      --max-num-batched-tokens 4096
      --max-num-seqs 256
      --disable-log-requests
      --port 8000

Crucial Command Flags Explained:

• --kv-cache-dtype fp8: Stores Key-Value cache in 8-bit precision instead of 16-bit, cutting KV memory consumption by 50% and doubling maximum concurrent users.
• --enable-chunked-prefill: Prevents massive user prompts from causing lag spikes for ongoing token generation streams.
• --gpu-memory-utilization 0.94: Instructs vLLM to reserve 94% of physical VRAM for model weights and KV cache, leaving 6% for CUDA context and system activation spikes.
• ipc: host: Crucial for high-speed multi-GPU inter-process shared memory communication. Without this, Tensor Parallelism will suffer severe latency degradations.

Serverless vs. Dedicated 24/7 Pods: The Break-Even Calculation

Should your engineering team deploy a dedicated 4x H100 pod running 24/7, or utilize a serverless GPU endpoint (such as Runpod Serverless)?

Latency Metrics Breakdown: TTFT vs. ITL vs. P99 Tail Latency

When monitoring production LLM APIs, average response time is a misleading metric. Production SLAs depend on three distinct metrics:

• Time to First Token (TTFT): The duration from when a client submits an HTTP prompt to when the first token streams back. TTFT measures the efficiency of the prefill phase. It depends on raw tensor core compute power, prompt length, and chunked prefill configuration. An acceptable conversational TTFT is under 250 milliseconds.
• Inter-Token Latency (ITL): The average time elapsed between subsequent emitted tokens during autoregressive decoding. ITL dictates reading comfort; human reading speed is roughly 5 to 8 tokens per second (125ms–200ms per token). Fast models aim for 30 to 80 tokens/sec (12ms–33ms ITL).
• P99 Tail Latency: The slowest 1% of user requests. High P99 latencies occur when a massive batch of long prompts arrives simultaneously, exhausting the KV cache and forcing the engine to preempt or recompute active generation requests.

Speculative Decoding: Accelerating Generation by 65%

One of the most powerful features supported by modern vLLM and TensorRT-LLM builds is Speculative Decoding. Autoregressive token generation is notoriously inefficient because reading 70 billion parameters from HBM yields only a single token per iteration.

Speculative decoding pairs a small, fast draft model (e.g., Llama-3.1-8B) with the massive target model (Llama-3.1-70B):

1. The lightweight 8B draft model speculates and generates $K=5$ tokens rapidly in succession.
2. The heavy 70B target model evaluates all 5 candidate tokens simultaneously in a single, parallel forward pass (which takes the exact same time as evaluating a single token!).
3. The target model accepts the valid tokens (typically 3 to 4 tokens) and rejects incorrect guesses.

This increases average generation speed from 35 tok/s up to 58–65 tok/s on identical hardware, lowering the effective inference cost per token by 40%.

# Launch vLLM with Speculative Decoding enabled
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --num-speculative-tokens 5 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.92 \
  --port 8000

Production Load Balancing Architecture: Nginx Multi-Worker Gateway

In enterprise production, never expose a single vLLM container directly to user traffic. Deploy a dedicated Nginx reverse proxy configured with health checks and least-connection routing across multiple backend GPU workers:

upstream vllm_cluster {
    least_conn;
    server 10.0.1.10:8000 max_fails=3 fail_timeout=10s; # Worker 1 (4x H100)
    server 10.0.1.11:8000 max_fails=3 fail_timeout=10s; # Worker 2 (4x H100)
    keepalive 64;
}

server {
    listen 80;
    server_name api.my-ai-company.com;

    location /v1/chat/completions {
        proxy_pass http://vllm_cluster;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        
        # Disable buffering for Server-Sent Events (SSE) streaming!
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
        chunked_transfer_encoding on;
    }

    location /health {
        proxy_pass http://vllm_cluster/health;
        proxy_connect_timeout 2s;
        proxy_read_timeout 2s;
    }
}

Dynamic Multi-LoRA Serving: Serving 100 Models on 1 GPU

In enterprise software, customers demand custom-tuned models adapted to their specific domain, corporate tone, or database schema. Deploying 50 different full 70B models for 50 clients would require an astronomical 200 NVIDIA H100 GPUs costing over $50,000 per month!

vLLM and SGLang solve this via Dynamic Multi-LoRA Serving. A single base model (e.g. Llama-3-70B FP8) is loaded once into High-Bandwidth Memory. Lightweight Low-Rank Adaptation (LoRA) adapter weights (typically only 50MB to 200MB each) reside in host RAM or NVMe storage. When a user request arrives specifying a specific adapter, vLLM dynamically loads the adapter matrices into the GPU computation pipeline just for that request's batches:

# Start vLLM with dynamic Multi-LoRA enabled
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --enable-lora \
  --max-loras 16 \
  --max-lora-rank 64 \
  --lora-modules legal-advisor=/workspace/adapters/legal \
                 medical-coder=/workspace/adapters/medical \
                 sql-generator=/workspace/adapters/sql \
  --tensor-parallel-size 4 \
  --port 8000

Quantization Showdown: AWQ vs. GPTQ vs. EXL2 vs. Native FP8

To reduce VRAM requirements, engineering teams employ post-training quantization. Here is an empirical evaluation of quantization formats on Llama-3-70B:

Production Python Client: Resilient SSE Streaming with Backpressure

Consuming high-throughput vLLM streams in production web services requires handling network backpressure and client disconnects properly. Here is a production-ready asynchronous Python client using httpx:

import httpx
import asyncio
import json

async def stream_llm_response(prompt: str, model: str = "meta-llama/Meta-Llama-3.1-70B-Instruct"):
    url = "http://localhost:8000/v1/chat/completions"
    headers = {"Content-Type": "application/json"}
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
        "temperature": 0.7,
        "max_tokens": 512
    }

    async with httpx.AsyncClient(timeout=60.0) as client:
        async with client.stream("POST", url, json=payload, headers=headers) as response:
            if response.status_code != 200:
                print(f"[API Error]: HTTP {response.status_code}")
                return

            async for line in response.aiter_lines():
                if line.startswith("data: "):
                    data_str = line[6:].strip()
                    if data_str == "[DONE]":
                        break
                    try:
                        data = json.loads(data_str)
                        delta = data["choices"][0]["delta"].get("content", "")
                        print(delta, end="", flush=True)
                    except json.JSONDecodeError:
                        continue
    print("\n[Stream Complete]")

# Run test client
if __name__ == "__main__":
    asyncio.run(stream_llm_response("Explain how PagedAttention eliminates memory fragmentation in GPU VRAM."))

Speculative Decoding: Doubling Token Generation Speeds for Free

In standard auto-regressive generation, the memory-bandwidth bottleneck forces the GPU to execute a complete forward pass to generate a single token. Speculative Decoding fundamentally changes this arithmetic by pairing a massive target model (e.g. Llama-3.1-70B) with an ultra-fast, lightweight draft model (e.g. Llama-3.1-8B):

• The Drafting Phase: The small 8B model speculatively predicts the next 4 to 6 candidate tokens in rapid succession. Because the 8B model fits into L2/L3 caches and consumes minimal memory bandwidth, this draft generation takes only a few milliseconds.
• The Verification Phase: The large 70B model inspects all 5 drafted tokens simultaneously in a single batched forward pass. It accepts tokens that match its probability distribution and rejects the first hallucinated token.
• The Performance Outcome: On code generation, JSON schema outputs, and structured reasoning, the acceptance rate often exceeds 75%. This allows an 8x H100 cluster to deliver 140+ tokens per second on a 70B model—more than double the throughput of standard decoding—with mathematically identical perplexity and output distribution.

# Launch vLLM with native Speculative Decoding
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --num-speculative-tokens 5 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.92 \
  --port 8000

Chunked Prefill & Automatic Prefix Caching (APC) for Agents

AI agent workflows (such as LangChain, AutoGen, and CrewAI) send lengthy system prompts, tool schemas, and conversation histories repeatedly with every turn. In legacy serving systems, recalculating the KV-cache for identical 4,000-token system prompts on every single API call causes massive Time-to-First-Token (TTFT) latency spikes and wastes compute.

Both modern vLLM and SGLang solve this through Automatic Prefix Caching (APC). By storing KV cache blocks in a Radix-Tree data structure keyed by token hashes, the inference engine instantly reuses precomputed attention tensors for shared system prompts. In long-running multi-turn sessions:

• TTFT drops from 1,200 ms down to under 35 ms (a 97% latency reduction).
• GPU VRAM throughput doubles, as compute units only process newly appended user messages rather than reprocessing identical instructions.

Summary & Next Steps

High-performance LLM serving is no longer about brute-force GPU hardware—it is an algorithmic discipline of KV cache management, continuous scheduling, and optimal precision selection. Review our detailed benchmarks and provider guides to architect your production serving stack:

• Top 10 GPU Cloud Providers Ranked
• Runpod Dedicated & Serverless Specs
• Lambda Cloud Instance Benchmarks
• Cheapest GPU Cloud Server Guide