The Tiny Draft Model Hidden Inside Qwen

I had already spent two posts pulling apart Qwen3.6's attention stack, first Gated DeltaNet, then partial RoPE. I expected the next oddity to be another attention detail. It was sitting at the very end of the checkpoint:

mtp.layers.0.*

That path belongs to a whole extra transformer layer. Its job is to take what the 64-layer model was about to say and guess one token farther ahead.

A language model normally writes one tiny chunk of text, runs the full model again, then writes the next. Qwen's extra layer gives vLLM and SGLang a cheaper option. They can run the small layer a few times, build a draft, and let the full model check several tokens at once. The draft never gets the final say, so the output still comes from Qwen's main model.

Three lines in the Qwen3.6 config describe the arrangement:

"num_hidden_layers": 64,
"mtp_num_hidden_layers": 1,
"mtp_use_dedicated_embeddings": false

Qwen has 64 language-model layers and one Multi-Token Prediction layer. mtp_use_dedicated_embeddings = false tells the predictor to use the main model's token embeddings. The checkpoint index lists an mtp.fc projection, one full-attention block under mtp.layers.0, and three normalization weights. It contains no second embedding table.

For the dense 27B model, the matrices in that module add up to roughly 0.39 billion parameters, about 1.5 percent of the language model. That number made the title feel slightly dishonest. "Tiny" still means hundreds of millions of weights here.

One step deeper than the main model

Suppose the prompt ends at token $x_t$. The main Qwen stack produces a hidden state $h_t$, and its output head turns that state into a probability distribution for $x_{t+1}$.

Once $x_{t+1}$ has been chosen, the MTP module receives two pieces of information: the main model's final hidden state and the embedding of that new token. It normalizes them, concatenates them, and projects the result back to Qwen's hidden width. One transformer block then predicts $x_{t+2}$ through the same vocabulary head:

I like this arrangement because the predictor does not begin from raw tokens. It inherits the 64-layer model's compressed reading of the prompt in $h_t$, combines it with the latest token, and performs a small continuation of the main computation.

I initially read Multi-Token Prediction as several parallel output heads. Qwen3.6 uses a sequential design in which its single MTP block predicts one additional depth per call. The serving engine can feed the predicted token and the new MTP hidden state through the block again to guess farther ahead. Recent vLLM releases warn that setting more than one speculative token runs the same MTP layer multiple times, which may reduce acceptance as errors accumulate.

During training, the future token is known and supplies an auxiliary prediction target. At inference, the module has to consume its own guesses after the first extra step. I would be cautious with long speculative chains because every wrong guess leaves the next call with a worse input.

How the draft becomes real output

The full model first produces a token and the hidden state behind it. The MTP layer extends that state into a short candidate sequence. Then Qwen runs once over all candidate positions with causal masking, producing its own distribution at every position in the draft.

The verifier accepts the longest prefix that agrees with the target model. At the first rejection, the server drops the remaining draft and uses the verifier's corrected replacement distribution. Greedy decoding therefore returns the same tokens as ordinary greedy decoding. With the proper rejection sampler, stochastic speculative decoding preserves the target distribution up to hardware-level numerical differences. The server uses MTP outputs as proposals and derives final output from the target probabilities.

This can be faster because decoding a single token often leaves a GPU waiting on memory. The hardware must read a large set of weights to perform little arithmetic. Verification gives the same weight load several token positions to process. When two or three draft tokens survive, one expensive target pass replaces several ordinary decode steps.

I would not enable this and assume an automatic win. Acceptance rate controls the gain. Every rejected suffix consumes MTP work and target verification capacity without advancing the sequence. Larger batches can also push the target model from memory-bound toward compute-bound operation, leaving less spare arithmetic for verification. vLLM describes speculative decoding as a latency optimization for medium-to-low-QPS, memory-bound workloads and recommends benchmarking the actual model, hardware, sampling settings, and traffic pattern.

What vLLM and SGLang do with the weights

Current vLLM MTP documentation uses a generic configuration:

--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Qwen's model card still shows the older family-specific name, qwen3_next_mtp. In either form, vLLM creates a proposer from the MTP tensors inside the target checkpoint. Its Qwen MTP implementation normalizes the token embedding and target hidden state, concatenates them through mtp.fc, runs mtp.layers.0, and applies the language-model head.

SGLang routes the same idea through its EAGLE-style speculative machinery. Qwen recommends --speculative-algo NEXTN, three draft steps, topk=1, and a four-token verification cap. NEXTN tells SGLang to use the checkpoint's future-token predictor, while the remaining flags determine draft depth and tree width. No external draft checkpoint appears in either command.

Native MTP avoids the separately trained small model used in a conventional speculative-decoding setup. Qwen ships the proposer weights beside the target weights and trains them against the same hidden-state space and vocabulary.

I called the draft model hidden because ordinary Hugging Face Transformers generation currently ignores those weights. Its Qwen3.5/3.6 model class lists ^mtp.* among unexpected checkpoint keys to skip. vLLM and SGLang can load the same tensors as a proposer, so enabling MTP changes which checkpoint weights run without changing the checkpoint itself.

References

• Qwen3.6-27B model card and serving recipes
• Qwen3.6-27B configuration
• Qwen3.6-27B checkpoint weight index
• vLLM Qwen3.5 MTP implementation
• vLLM speculative-decoding documentation
• SGLang speculative-decoding documentation
• Better & Faster Large Language Models via Multi-Token Prediction
• DeepSeek-V3 Technical Report, Section 2.2
• Qwen3.5/3.6 implementation in Transformers