POSITIONAL ENCODING
Why Qwen Rotates Only a Quarter of Each Attention Head
What Qwen's 25% rotary fraction changes and what it leaves untouched.
SeriesQwen Hybrid ArchitecturePart 2 of 3 ↗Suppose you have five notebooks with 256 boxes on each page. When you file a note, you add a location mark to the first 64 boxes. The other 192 keep only what you wrote in them. Move the same notes to different shelves and the marked boxes compare differently because their location changed; the unmarked boxes can still compare by their contents.
Qwen does something close to that inside each full-attention head. I went back through Qwen3.6's config after writing about Gated DeltaNet and got stuck on two numbers: head_dim = 256 and partial_rotary_factor = 0.25. RoPE touches the first 64 dimensions of every query and key. The other 192 pass through unchanged.
The architecture summary says "uses RoPE." The implementation does this:
rotary_dim = cos.shape[-1]
q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
q_embed = torch.cat([apply_rope(q_rot), q_pass], dim=-1)
k_embed = torch.cat([apply_rope(k_rot), k_pass], dim=-1)The released Qwen3.6 config gives head_dim = 256 and partial_rotary_factor = 0.25. The Transformers implementation makes the split before attention. Calling the remaining 192 dimensions positionless would go too far. RoPE does not transform them directly, but earlier layers can write position information into them.
The attention score has two terms
RoPE rotates pairs of query and key coordinates by an angle determined by token position. If a query occurs at position $m$ and a key at position $n$, their rotated dot product depends on the relative offset $n-m$.
With partial RoPE, split each query and key into a rotated part and a pass-through part:
Qwen computes the score as
The first term changes with relative distance. The second stays fixed when two tokens move farther apart. A single head can therefore combine a relative-position score with a direct similarity score.
For a 256-dimensional head, the split is not a separate miniature attention mechanism. Qwen still forms one dot product and applies one softmax across the available keys. The learned query and key projections choose what to place in the first 64 coordinates and what to place in the other 192. They can also change the scale of both contributions. A head might put a delimiter's spacing signal in the rotated part while putting its token identity in the pass-through part, but the architecture does not assign those jobs in advance.
That distinction avoids an easy mistake. The 192 pass-through coordinates do not vote independently and then get averaged with the rotated coordinates. They contribute to the same score. A query can use both kinds of feature when it decides whether one prior token is more useful than another.
Qwen applies per-head QK normalization before RoPE. Its queries and keys come from residual states that have already passed through ordered, causal computation. "Direct similarity" here means RoPE-free coordinates, not position-blind ones.
The pass-through subspace
RoPE makes position part of every similarity calculation it touches. That helps with a nearby modifier or the distance between paired delimiters. Retrieval based on identity alone has a different requirement. A function name 80,000 tokens back is still the same function name, while its rotary phase differs sharply from that of a nearby match.
The pass-through term gives the model a place to encode features whose match should survive displacement. The rotated term can resolve order and relative location. Qwen's learned projections decide which features occupy the 64 rotated coordinates and which occupy the remaining 192.
The split also leaves 192 coordinates untouched by context-extension changes to RoPE itself. Qwen uses a RoPE base of 10,000,000 and recommends YaRN when extending its native 262,144-token context toward one million tokens. YaRN changes rotary frequencies, not the 192 pass-through coordinates.
This is my reading of the score decomposition. Qwen has published no Qwen3.6 ablation that attributes long-context performance to the 25% setting.
A 2026 study, Fractional Rotation, Full Potential?, trained models with different rotary fractions. Fractions around 10% and above reached similar final loss to full RoPE across the tested settings. NoPE runs were less stable, while even a small rotated subspace restored stable training. The experiments stopped at 8B parameters and did not test Qwen3.6-27B. They support partial RoPE as a general design choice, not Qwen's exact ratio.
Partial RoPE predates this Qwen generation. GPT-J, GPT-NeoX and Pythia also used a 25% rotary fraction; several later families apply RoPE across the full head.
Partial RoPE does little to the KV cache
Rotating 64 coordinates takes one quarter of the elementwise rotation work, and a cached cosine and sine table can be one quarter as wide. The saving is small beside the model's main memory costs.
Partial RoPE does not shrink Qwen's KV cache by 75%. Keys and values still use the full 256-dimensional head width. Qwen gets larger cache savings from four KV heads shared across 24 query heads, and from using full attention in only 16 of its 64 language-model layers. The other 48 layers use Gated DeltaNet and keep recurrent state instead of a standard attention KV cache.
The partial rotation changes how Qwen scores full-width keys. Their stored width remains 256.
The 64 dimensions also carry three kinds of position
Qwen3.6 is a native vision-language model. Its 64 rotary dimensions form 32 coordinate pairs, and the config divides those pairs into sections of [11, 11, 10]. The implementation interleaves temporal, height and width frequencies across them. Text tokens use the same machinery with their position IDs aligned across the three axes.
This leaves a compact positional subspace carrying sequence order and visual coordinates, alongside a 192-dimensional pass-through subspace. A full-attention head can compare image or video tokens using spatial position in one part of its score and learned content similarity in the rest.
One useful test would log each score term before softmax and plot it against token distance. Repeating the measurement for text and video could show whether the 192 pass-through dimensions carry long-range content matches or mostly support the multimodal rotary channel.