vLLM v0.34.0 was released on 16 November 2026. It includes 1,061 commits from 412 contributors, and several of its changes affect how much GPU memory vLLM uses and how your server starts. The KV cache is now sized from the memory that’s actually free, the default block size has changed, and the old Model Runner V1 has been removed. This guide explains what changed, which flags restore the old behaviour, and how to upgrade a production deployment without surprises.

All details come from the official vLLM v0.34.0 release notes.

The short version

  • Extensible KV cache is on by default. Most models get a larger cache and more throughput.
  • gpu_memory_utilization now defaults to 1.0.
  • Default --block-size is 64 (it was 16).
  • Model Runner V1 is gone. V2 is the only GPU runner.
  • huggingface_hub >= 2.0.0 is required, plus tokenizers >= 0.23.3 and timm >= 1.0.30.
  • ROCm 7.2 artifacts are dropped (use ROCm 10.0), and some quantisation paths are removed.
  • 10 new models, including Mistral Large 4.

1. Extensible KV cache: bigger cache, one trade-off

Earlier versions sized the KV cache from an estimate made before CUDA graph capture. In v0.34, on CUDA and ROCm, vLLM sizes it from the memory that is actually free after CUDA graph capture and warmup.

The release notes give an example: MiniMax-M3 on an H200 goes from a 433k-token to a 672k-token cache. A larger cache means more concurrent sequences, higher throughput and lower time-to-first-token under load.

The trade-off: per-token decode latency (TPOT) can rise on some models. The notes report +6–18% on some H200 models at concurrency 128. If you serve latency-sensitive traffic, or you share the GPU with another process, restore the old sizing:

# Option 1: go back to the old 90% memory cap
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 --gpu-memory-utilization 0.9

# Option 2: keep the old utilisation default and disable extensible sizing
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 --no-enable-extensible-kv-cache

(Use your own model name. The one above is only an example.)

Shared GPUs warning: because gpu_memory_utilization now defaults to 1.0, vLLM will try to use the whole card. If another process, such as an embedding server, a second vLLM instance or a training job, runs on the same GPU, set --gpu-memory-utilization explicitly.

2. Default block size is now 64

The KV cache block size moved from 16 to 64 tokens. Most users won’t notice, but if you tuned memory or prefix-caching behaviour around 16-token blocks, pin it:

vllm serve <model> --block-size 16

ROCm paged attention still uses 16.

3. Model Runner V1 is removed

Model Runner V2 is now the only GPU runner. If your deployment sets VLLM_USE_V2_MODEL_RUNNER=0, vLLM will fail at startup. Remove it:

grep -rn "VLLM_USE_V2_MODEL_RUNNER" deploy/ k8s/ docker/ .env* 2>/dev/null

If you wrote custom V1 logits processors, port them to vllm.v1.worker.gpu.sample.logits_processor.

Features moving to V2 in this release include dual-batch overlap (also with speculative decoding and DCP) and suffix decoding with a GPU speculator, so you no longer need Arctic Inference for that.

4. Dependency and platform changes

Update the Hugging Face stack in the same environment:

pip install -U "vllm==0.34.0" "huggingface_hub>=2.0.0" "tokenizers>=0.23.3" "timm>=1.0.30"
python -c "import vllm, huggingface_hub; print(vllm.__version__, huggingface_hub.__version__)"

Official install options from the release notes:

Platform Install
CUDA 13.0 (PyPI) pip install vllm
CUDA 13.0 (uv) uv pip install vllm --torch-backend=auto
ROCm 10.0 pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.34.0/rocm100

Docker images:

docker pull vllm/vllm-openai:v0.34.0          # CUDA 13.0
docker pull vllm/vllm-openai-rocm:v0.34.0     # ROCm 10.0
docker pull vllm/vllm-openai-cpu:v0.34.0      # CPU

Other removals to check:

  • ROCm 7.2 images and wheels (-rocm72) are no longer published. Move to ROCm 10.0.
  • QuTLASS / fp_quant is removed, so fp_quant checkpoints no longer load.
  • Marlin W4A8 is removed. Use --quantization humming with VLLM_HUMMING_INPUT_QUANT_CONFIG instead of VLLM_MARLIN_INPUT_DTYPE.
  • Keye-VL and Keye-VL-1.5 model support is removed.

5. A safe upgrade procedure

Don’t swap the image in production and hope for the best. Run a quick before/after comparison.

Step 1: Record your current numbers. On the old version, note the KV cache size printed at startup and run your usual benchmark:

vllm bench serve --model <model> --dataset-name random \
  --random-input-len 1024 --random-output-len 256 --num-prompts 500

Step 2: Start v0.34 with default settings and compare the logged KV cache capacity.

Step 3: Run the same benchmark and compare throughput, TTFT and TPOT. If TPOT got worse than your SLO allows, retry with --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache.

Step 4: Smoke-test the OpenAI-compatible API your apps actually use:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="<model>",
    messages=[{"role": "user", "content": "Reply with the word OK."}],
    max_tokens=5,
)
print(resp.choices[0].message.content)

Step 5: Roll out gradually, for example one replica at a time, while watching GPU memory and latency dashboards.

6. New features worth a look

  • New models: Mistral Large 4 (FP8 and NVFP4), MiniMax-M3.1-Flash, NVIDIA Nemotron-3.5-Transcribe (with /v1/audio/transcriptions), Liquid AI LFM3 family, NVIDIA Canary-1B-v2, Zyphra ZAYA1 and more.
  • Speculative decoding: custom speculators through declarative configs and a register_speculative_method plugin API. The draft KV cache now matches the target’s dtype, so an FP8 target gets an FP8 draft cache.
  • Checkpoint/restore with CRIU and cuda-checkpoint through /checkpoint/prepare, /checkpoint/status and /checkpoint/restore, which is useful for faster cold starts.
  • Fault-tolerant DP+EP scale-down: dead ranks are removed and their experts redistributed without a restart.
  • NUMA binding is on by default (best effort), and cache-aware admission ordering is available as an opt-in.
  • Performance: the release notes report gains on Blackwell and MI355X, for example up to 1.95× decode for Step5 NVFP4 MoE on 4×GB200. As always, measure on your own hardware and models.

Upgrade checklist

  1. Update huggingface_hub, tokenizers and timm alongside vLLM.
  2. Remove VLLM_USE_V2_MODEL_RUNNER=0 and port any V1 logits processors.
  3. Shared GPU? Set --gpu-memory-utilization explicitly.
  4. Compare TPOT before and after; opt out of extensible KV cache if needed.
  5. Pin --block-size 16 only if you depended on it.
  6. Move ROCm deployments to ROCm 10.0 images.
  7. Check that you don’t depend on fp_quant, Marlin W4A8 or Keye-VL.

Takeaway

vLLM 0.34 gives most deployments a larger KV cache and more throughput with no code changes, but it also changes memory defaults and removes the V1 runner. Benchmark before and after, set --gpu-memory-utilization on shared GPUs, and keep the opt-out flags ready if decode latency matters more than throughput.