Skip to content
Neural Daily

ToolsNews

vLLM 0.32: decision endpoint and leaner KV cache

🇭🇺 Olvasd el magyarul →

vLLM 0.32.0, the new release of the popular open-source inference engine, shipped on October 18. It adds an endpoint for structured decisions and runs low-precision KV cache on more GPUs. Check the breaking changes before you upgrade.

By: Mikola Máté Patrik · · 2 min read

GitHub release page of vLLM v0.32.0 with the Highlights section
Image: vLLM · Screenshot · Screenshot of official release page · Source

What's new

The release contains 814 commits from 363 contributors, 121 of them new (GitHub). The main changes:

  • New POST /v1/systemone endpoint: answers choice, noul and score questions over a structured state. It is served by default and runs on Cloudflare/clef-omni, LiquidAI/d1-3B and DiffusionGemma.
  • Generic hf tool and reasoning parser: it reads the checkpoint's response_template, so models that ship one need no model-specific parser.
  • Low-precision KV cache: NVFP4 KV cache now runs on SM8x, SM12x and SM90 GPUs. On SM100, GLM-5.x defaults to an FP8 KV cache, halving KV memory per rank. In the PR's benchmark that gave 2.3 to 5.5% more throughput at a 7 to 11% TTFT cost.
  • Faster startup: vllm download-kernels installs FlashInfer's precompiled kernels, and vllm preload --preload-autotune keeps the autotune table warm across restarts.
  • Load balancing: with --api-server-count above 1, each connection goes to the least-loaded API worker. On short requests the PR's benchmark showed 43 to 85% more req/s (Qwen3-0.6B).
  • New models: EmbeddingGemma 2 (omni-modal embeddings), MiniCPM-V 4.7 and Nemotron 3.5 ASR.

Security

Three advisories are fixed: GHSA-4hhp-h66f-j5j7 (chat template resource exhaustion), GHSA-q97c-p368-75c4 (tokenizer max_token_id) and GHSA-frvg-qw88-hww9 (P2P ZMQ socket exhaustion). Default CORS no longer allows every origin; browser clients need --allowed-origins. Untrusted images are limited to raster formats, and JSON schemas nested deeper than 64 levels are rejected with a 400.

Check before upgrading

  • Python 3.10 is no longer supported; the minimum is 3.11.
  • It requires transformers>=5.16.1,<5.20.0 and tokenizers>=0.23.2.
  • max_tokens is now an upper bound rather than a reservation from the context window, so long prompts are no longer rejected.
  • Cascade attention was removed. --disable-cascade-attn only warns for now and goes away in v0.33.0.
  • Responses API reasoning stream events were renamed to response.content_part.added/done with part.type="reasoning_text".

Who should upgrade

If you serve models on your own GPUs, the KV cache changes and smarter load balancing can bring real gains. Because of the breaking changes, start on staging, especially if browsers call your API (CORS) or you rely on Responses API stream events. The /v1/systemone endpoint also fits the decision model trend we covered with Liquid AI's d1-ultra.

    Produced with the help of AI, reviewed by an editor. How we make our articles

    Reviewed and edited by the Neural Daily desk. About

    Related articles