The EAGLE team has released EAGLE-4, an adaptive speculative-decoding drafter, alongside a pip-installable package called eagle-harness. The drafter measures serving costs and adjusts its speculative tree for the current batch size, hardware, and workload, reducing the need to maintain separate configurations for interactive and high-concurrency inference.

Speculative decoding uses a smaller model to propose several tokens before the target model verifies them in parallel. Correct verification preserves the target model’s output distribution while reducing the number of sequential target-model calls. Performance depends on how quickly the drafter proposes useful candidates and how much work the target spends verifying them.

pip install eagle-harness

The harness integrates with SGLang, vLLM, and the experimental TokenSpeed engine through their registration hooks, with no runtime patching. The reference pairing uses Qwen3.8-27B and a published EAGLE-4 checkpoint.

Runtime-sized speculation

EAGLE-4 profiles draft generation and target verification at the active batch size, then selects the depth and branching factor expected to produce the highest throughput. A top-k value of 1 creates a speculative chain, while larger values create a tree of candidate continuations.

The drafter combines several techniques intended to limit its own compute and memory costs:

• Sliding-window draft layer: Keeps a bounded window of recent hidden states instead of attending across the full sequence.
• Low-rank candidate head: Factors token prediction into smaller matrices, reducing the cost of proposing candidates.
• Append-only draft cache: Extends the draft KV cache between rounds instead of rebuilding it.