Speculative Decoding & Fast Inference (2026)

The Inference Bottleneck Problem

Autoregressive language models generate text one token at a time: compute attention over all previous tokens, sample the next token, add it to context, repeat. This sequential process is inherently slow. Even with powerful GPUs, generating a 1000-token response takes seconds. For latency-sensitive applications (chat, code completion, real-time summarization), this latency is unacceptable.

The bottleneck isn't compute—it's memory-bound operations. The KV (key-value) cache for attention grows with sequence length, and memory bandwidth becomes the limiting factor long before the GPU's compute capacity is saturated. Modern inference optimization targets memory efficiency and parallelization.

Speculative Decoding: 2-3x Speedup via Parallel Verification

Speculative decoding is a clever technique that exploits the fact that large models often agree with smaller models on "easy" tokens. Here's how it works:

1. A small, fast draft model (e.g., a 7B model) proposes multiple tokens in parallel (e.g., k tokens lookahead)
2. The large main model verifies all k proposed tokens in a single forward pass
3. If the main model accepts all k tokens, you've effectively generated k tokens with ~1 main model forward pass
4. If there's a disagreement, you keep the main model's decision and continue

The result: 2-3x throughput improvement on many workloads, often with no quality degradation. The main model's output is always used, so generated text quality is unchanged.

Dynamic Speculation Windows: The number of tokens proposed (k) can be adjusted dynamically based on model agreement rates. If the draft model and main model agree frequently, increase k. If they disagree often, decrease k. This adaptive approach outperforms fixed speculation windows.

Hardware Considerations: Speculative decoding works best when the draft model runs on different hardware or streams in parallel with the main model. On a single GPU, the gains are modest; across multiple GPUs or mixed compute (draft on CPU, main on GPU), speedups are substantial.

PagedAttention: Efficient KV Cache Management

The KV cache is a critical bottleneck. Naively, for a 16K-token context, you store 16K key vectors and 16K value vectors for each of the model's layers. As the cache grows, fragmentation occurs: when requests complete, memory is freed in scattered chunks, making it hard to allocate large contiguous blocks for new requests.

PagedAttention solves this by managing the KV cache like virtual memory in operating systems:

• Divide the KV cache into pages (logical blocks)
• Allocate pages on-demand as sequences grow
• Requests don't need contiguous memory; pages are mapped via a page table
• When a request completes, its pages are freed and immediately reusable by other requests

Benefits:

• Reduces memory fragmentation: up to 70% KV cache memory savings
• Increases throughput: more requests fit in memory simultaneously
• Enables batching: serving dynamic batch sizes efficiently

PagedAttention is the core innovation in vLLM and is now standard in production inference servers.

Continuous Batching: Request-Level Scheduling

Traditional batching is static: collect N requests, run them in a single batch, wait for all to finish. This wastes GPU time—if one request needs 50 tokens and another needs 100, the GPU idles after the first request completes.

Continuous batching (iteration-level scheduling) improves utilization:

1. Maintain a queue of in-flight requests
2. At each generation step, schedule which requests to process
3. Remove completed requests immediately; add new requests from the queue
4. The GPU runs at high utilization throughout

For a typical workload with varied request lengths, continuous batching can 2-3x throughput compared to static batching.

Quantization Options for Modern GPUs

Quantization reduces model weights and activations from 32-bit or 16-bit floating point to lower precisions:

INT8 Quantization: 8-bit integer weights and activations. Minimal accuracy loss; supported by NVIDIA Hopper GPUs with Tensor Core acceleration. Recommended for production.

INT4 Quantization: Further compression, but requires careful rounding and post-training quantization. Accuracy loss is more noticeable but still acceptable for many tasks. Extreme memory savings (4x vs. FP32).

FP8: 8-bit floating point (1 sign, 4 exponent, 3 mantissa bits). Balance between INT8 and FP16. Supported on Hopper and newer. Gaining popularity in 2026.

Recommendation: Start with INT8 on Hopper+ GPUs. INT8 is production-proven, fast, and has minimal quality impact. Use INT4 only if memory is severely constrained (e.g., mobile, edge).

Practical Deployment Playbook

When to apply each technique:

1. Start with PagedAttention + continuous batching (foundation)

   • Use vLLM or TensorRT-LLM for serving
   • Enables dynamic batching and efficient memory use
   • 2-3x throughput vs. naive inference

2. Add INT8 quantization (memory + speed)

   • Reduces model size by 4x
   • Preserves accuracy
   • Enables larger batch sizes

3. Implement token streaming (latency for UX)

   • Stream tokens to client as they're generated
   • Perceived latency drops significantly
   • Requires streaming protocol (e.g., Server-Sent Events)

4. Tune max token limits (predictability)

   • Cap max output tokens per request (e.g., 512 tokens)
   • Prevents tail latency from runaway generations
   • Set reasonable defaults in UI/API

5. Optional: Add speculative decoding (advanced optimization)

   • Requires multiple GPUs or mixed compute
   • Additional 2-3x throughput on top of PagedAttention
   • Best for high-volume, latency-critical services

Monitoring and Observability

Key metrics to track:

• p95 latency: 95th percentile time-to-first-token and time-per-token
• Throughput: tokens/second across all concurrent requests
• Batch size: actual batch sizes achieved (continuous batching adapts dynamically)
• Memory utilization: GPU memory used; should be 80-90% for efficiency
• Error budget: track failed requests, timeouts, queue overflow

Tuning knobs:

• Request timeout (balance fairness vs. outlier protection)
• Max batch size (higher = more throughput, higher latency variance)
• Quantization precision (INT8 vs. INT4; accuracy vs. speed tradeoff)
• Speculation window (if using speculative decoding)

Monitoring these metrics helps you understand if your optimization is working and where the next bottleneck is. Most serving teams find that 60% of gains come from PagedAttention + continuous batching, 30% from quantization, and 10% from speculative decoding and fine-tuning.