Skip to content
Neural Daily

ToolsNews

EAGLE-4: one drafter for every batch size

EAGLE-4, the next generation of the speculative decoding method, is out. One trained drafter adapts how much it speculates to the hardware and load, with up to 3.59x speedup for a single request in the team's tests.

By: Mikola Máté Patrik · · 2 min read

Eagle blog cover: Introducing EAGLE-4, One Flexible Drafter for All Batch Sizes and Hardware
Image: Eagle · Press kit · Official blog social image (og:image) · Source

What is new

In the EAGLE family a small drafter proposes tokens ahead and the target model verifies them in one pass. The main addition in EAGLE-4 is EAGLE-Harness: a warm-up profiles draft and target costs across batch sizes, context lengths and speculative token counts, then during generation it combines those costs with draft confidence to pick tree depth, width and verification budget. The same drafter speculates aggressively for one request and holds back under heavy load.

The second change is a two-stage LM head: a low-rank head shortlists candidates across the full vocabulary, and the original head computes exact logits only for those. The team says full-vocabulary logits took 65.4% of a draft forward pass.

Numbers from the team

  • Qwen3.8-27B on 4x B300: 863.8 tokens/s for one request (vanilla: 240.5), 28,340 aggregate tokens/s at 512 requests (vanilla: 21,203).
  • Nemotron 3.5 Super VL: 2.74x for one request, 1.69x at 512.
  • AMD MI350X, single GPU: 3.49x for one request.
  • 4-vCPU server, 4-bit Gemma-4-E2B, llama.cpp: 2.28x to 2.92x.
  • Phone CPU (Snapdragon 8 Elite): HumanEval from 22.8 to 82.6 tokens/s, 53.4% less energy per token.

Headline results use temperature 0. At temperature 1 the Qwen speedup at 512 requests drops to 1.03x, so gains shrink a lot with sampling.

How to use it

The eagle-harness package installs from PyPI and runs as a plugin on SGLang, vLLM and, experimentally, TokenSpeed. Stock engine packages lack the required hooks, so you need the patched wheels or container images listed in the repository. The license is source-available rather than standard open source. Drafters currently exist for Qwen3.8-27B and Nemotron 3.5 Super VL.

If you run your own inference server with these models, it is quick to measure what it gains on your real traffic.

    Produced with the help of AI, reviewed by an editor. How we make our articles

    Reviewed and edited by the Neural Daily desk. About

    Related articles