Speculative Decoding
An inference optimization technique where a small, fast 'draft' model generates candidate tokens ahead of the main model, and the main model verifies them in parallel. When the draft model's predictions match what the main model would have produced, tokens are accepted instantly—reducing latency by 2–3x without changing output quality. Speculative decoding is particularly valuable for AI agents where response latency directly affects user experience, especially voice agents and live-chat support agents.
Written by Max Zeshut
Founder at Agentmelt · Last updated Sep 9, 2026
Frequently asked questions
- Does speculative decoding change the model's outputs?
- No. Speculative decoding is mathematically equivalent to standard decoding—the main model has the final say on every token. The draft model only proposes candidates; any token the main model would not have generated is rejected. The output distribution is identical; only the speed improves.