--- title: "Speculative Decoding: Accepted Draft Tokens Are Not a Throughput Result" description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20." canonical: "https://www.leviathansystems.co/articles/speculative-decoding-draft-acceptance-versus-output-throughput" last-updated: "2026-10-20" --- # Speculative Decoding: Accepted Draft Tokens Are Not a Throughput Result Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20. A speculative-decoding acceptance ratio describes how much proposed work survives verification. It does not measure how quickly the system delivers output. Compare emitted output over an equally defined interval that includes the work of proposing and verifying candidates, then check the required response time and task quality. More accepted tokens per verification step can be useful diagnostic evidence without establishing a faster serving offer. ## Follow candidates through verification Speculative decoding uses a proposer to supply candidate tokens for target verification. In the standard linear mechanism in the 2023 paper by Leviathan, Kalman and Matias, the target evaluates positions together, retains the prefix before the first rejection and discards the remaining candidates. An extra sampled token corrects that rejection or extends a fully accepted block. With stochastic sampling, acceptance and correction use proposal and target probabilities to preserve the target distribution under the algorithm's assumptions. They are more than word matching. Proposed candidates, accepted drafts and emitted output count different work. The extra token is output without being an accepted draft. References: [P1](https://proceedings.mlr.press/v202/leviathan23a/leviathan23a.pdf) ## Read the dated vLLM interface as accounting vLLM's Per-Request Acceptance Metrics page, dated August 20, 2026 and checked October 20, labels this interface experimental and subject to change. Its draft_acceptance_rate is accepted draft tokens divided by proposed draft tokens. Its mean_acceptance_length is 1 + accepted draft tokens / verification steps, including the bonus token. The accepted-draft count excludes bonuses; the proposed count subtracts drafts invalidated by structured-output constraints. These fields cover single-sequence requests; requests for multiple sequences return null. Server aggregates also count multiple-sequence requests, so summing all reporting requests reconciles only for an entirely single-sequence workload. For a comparison, match the request population and observation interval. Missing coverage is not a measured zero. Neither ratio supplies elapsed time: tokens per step cannot establish tokens per second. References: [V1](https://docs.vllm.ai/en/stable/features/speculative_decoding/acceptance_metrics/) ## SD-201: twenty proposals produce fifteen output tokens Fictional case SD-201 stipulates five complete verification steps, each proposing four candidates. The accepted draft counts are zero, one, two, three and four, respectively. No step is truncated and no early termination removes output; the stipulated accounting emits one additional bonus token per step. The proposal total is 5 × 4 = 20 candidates. Accepted drafts total 0 + 1 + 2 + 3 + 4 = 10 tokens. Adding five bonus tokens gives 15 emitted output tokens. The acceptance fraction is 10 / 20 = 50%. Mean emitted length is 1 + 10 / 5 = 3 tokens per verification step. Those figures describe the same invented generation with different denominators. Fifty percent measures the fraction of proposals retained. Three per step measures progress per verification cycle. It does not mean three times faster: a speculative cycle and a target-only decoding step have no stipulated equality of elapsed cost. SD-201 is an accounting example. Its counts and the timings below are invented, with no named model, hardware ranking or observed benchmark result. ## SD-201: two labels say 200 tokens/s but count different work Offer A stipulates a complete generation interval of 100 milliseconds, or 0.1 seconds, for SD-201. Timing starts immediately before the first proposal and ends when the final output token is produced. It includes all proposal, verification and intervening scheduling time. Prompt processing precedes this boundary and client delivery follows it, so this is a generation-only interval. Over that same interval, proposed-draft throughput is 20 / 0.1 = 200 draft tokens/s. Accepted-draft throughput is 10 / 0.1 = 100 accepted draft tokens/s. Emitted-output throughput is 15 / 0.1 = 150 output tokens/s. Internal proposals cannot replace output in the last calculation. Fictional offer B emits the same stipulated fifteen-token task output in 75 milliseconds, or 0.075 seconds, over an equally defined generation interval that includes its complete generation work. B therefore reports 15 / 0.075 = 200 emitted output tokens/s. A's 200 and B's 200 describe different populations. The example stipulates a common target tokenizer, output-counting convention and task result. It does not predict identical stochastic text from real runs. B has higher emitted throughput within the example's boundary; neither offer supplies evidence of aggregate capacity under concurrent traffic or complete client response time. ## Find the elapsed cost that acceptance leaves out The original paper's walltime analysis includes proposal-model cost and assumptions about available parallel computation. vLLM's method guidance likewise makes gains dependent on model, traffic, hardware and sampling. The buyer implication is that a higher acceptance figure can accompany a longer generation interval. Producing a better candidate continuation can take more time; checking a larger candidate block also consumes resources. Work on candidates that are discarded still belongs to the serving arrangement's cost. Under busy traffic, account for the proposal and verification work alongside the other requests sharing those resources. Keep acceptance as a diagnostic. If output throughput changes, it can help the serving team examine whether fewer proposals survived or whether more time was spent obtaining them. Acceptance alone cannot distinguish those explanations. Even unchanged acceptance leaves the elapsed cost unresolved. Use the complete measured interval rather than adding isolated stage timings that may overlap. No acceptance threshold or speculative depth can settle the purchase without the output result. References: [P1](https://proceedings.mlr.press/v202/leviathan23a/leviathan23a.pdf), [V2](https://docs.vllm.ai/en/stable/features/speculative_decoding/) ## Keep losslessness, task quality and repeatability separate vLLM's losslessness discussion, on the page dated September 25, 2026 and checked October 20, distinguishes mathematical sampling guarantees, implementation validation and numerical behavior. It qualifies theoretical losslessness by hardware precision, does not guarantee stable token log probabilities, and notes that floating-point and batch differences can affect outputs. A target distribution can include incorrect answers. Preserving that distribution does not demonstrate factual correctness, a better answer or identical text across runs. Evaluate the offered configuration against the required task quality; treat stochastic comparisons as comparisons under the agreed sampling and evaluation method. Retain the exact target revision and tokenizer, proposer or draft identity, numerical representation, sampling settings and runtime/backend version. If the offer uses a separate draft model, identify its revision as well. A changed target or sampling policy changes the comparison even when both reports use the same acceptance metric. References: [V2](https://docs.vllm.ai/en/stable/features/speculative_decoding/) ## Require output evidence before accepting the optimization An offer supported only by acceptance counts is an optimization hypothesis. To support a performance decision, it needs emitted-output evidence against the target-only reference or other proposed serving arrangement, using equivalent task, quality and traffic requirements. Retain the actual output count and the start and finish events for the timed interval, including the proposer wherever it runs. Distinguish faster generation for an individual request from greater serving capacity at the required load. A quiet-request result leaves the busy-period decision open. Likewise, a generation-only interval leaves prompt processing, queueing before generation and client delivery outside its claim. The linked GPU benchmark comparison guide provides the broader method for comparing complete systems and response-time requirements. Use that guide for the full measurement record. For this narrower optimization claim, the decisive question is whether the complete offered arrangement produces comparable useful output within the required time. Acceptance explains part of the mechanism; it does not answer that question by itself. ## Separate any new equipment work from the software decision Speculative decoding can remain a software decision on existing equipment. If a selected offer separately identifies an additional server with a defined mounting and cabling requirement, bring that equipment package to the physical project discussion. Leviathan's Complete Physical Infrastructure Deployment service can combine agreed receiving, rack integration, onsite installation, IT cabling and physical records. Use the related service and project inquiry for that separate installation need, with the equipment, location and connections identified. Speculative-serving evaluation and performance acceptance remain with the appointed software team; this article offers no inference, benchmarking or monitoring service. References: [L1](https://www.leviathansystems.co/services/turnkey-gpu-deployment) ## Primary sources - [P1 — Yaniv Leviathan, Matan Kalman and Yossi Matias — Fast Inference from Transformers via Speculative Decoding (ICML 2023)](https://proceedings.mlr.press/v202/leviathan23a/leviathan23a.pdf). Checked 2026-10-20. Sections 2 and 3 explain sampling and walltime assumptions; historical experiments do not predict an offered system. - [V1 — vLLM — Per-Request Acceptance Metrics](https://docs.vllm.ai/en/stable/features/speculative_decoding/acceptance_metrics/). Checked 2026-10-20. Experimental interface, page dated August 20, 2026; definitions and coverage apply to this documented interface. - [V2 — vLLM — Speculative Decoding](https://docs.vllm.ai/en/stable/features/speculative_decoding/). Checked 2026-10-20. Page dated September 25, 2026; workload dependence and qualified losslessness, without a performance promise. - [L1 — Leviathan Systems — Complete Physical Infrastructure Deployment](https://www.leviathansystems.co/services/turnkey-gpu-deployment). Checked 2026-10-20. Defined physical delivery scope; a software optimization creates no installation requirement by itself. ## Related Leviathan resources - [GPU benchmarks: compare complete systems and response-time requirements](/articles/which-gpu-is-fastest-workload-benchmarks) - [Complete Physical Infrastructure Deployment for a separate equipment project](/services/turnkey-gpu-deployment) - [Discuss a separately identified equipment installation](/contact?service=turnkey-gpu-deployment&from=%2Farticles%2Fspeculative-decoding-draft-acceptance-versus-output-throughput) - [The Same Seed Does Not Establish the Same AI Result](/articles/pytorch-seed-deterministic-algorithm-reproducibility) - [Why Sampled AI Traces Do Not Establish the Request Error Rate](/articles/ai-observability-trace-sampling-versus-request-metrics) ## Have a separate equipment installation requirement? For an identified physical project, share a nonconfidential equipment summary, site, rack locations and required IT connections to discuss installation scope and availability. [Discuss the scope](/contact?service=turnkey-gpu-deployment&from=/articles/speculative-decoding-draft-acceptance-versus-output-throughput)