Attention Should Be Allowed to Say No

Qwen's Gated Attention is a targeted change to how an attention head writes its result. The head still retrieves information from the context in the usual way, but Qwen adds a learned gate that can dampen or suppress that result before it enters the residual stream.

The point is not that Qwen replaces attention or makes every head sparse. It is that standard attention has an awkward blind spot: it can decide which source is best, but it cannot directly say that no source is useful enough to write forward. Qwen gives each head that additional control.

Why attention needs a write control

In ordinary attention, a head scores earlier tokens, normalizes those scores with softmax, and combines the corresponding value vectors. A simplified distribution might look like this:

Token	Attention weight
Token A	60%
Token B	25%
Token C	10%
Token D	5%

Those weights must sum to one:

\sum_j \alpha_{ij} = 1

That is sensible when at least one visible token contains information the head needs. The difficulty is that the distribution still has to choose a destination when every candidate is poor. Softmax can rank options; it does not preserve a clean signal for whether the winning option was useful in absolute terms.

Consider the logits 10, 9, 8 and -100, -101, -102. Their relative gaps are the same, so softmax produces nearly the same distribution for both. Yet the second set can reasonably be read as a head finding nothing compelling in the context. Standard attention has no separate control that lets the head act on that distinction.

The Qwen paper frames this as a missing write decision. Attention already answers where to read and what values to combine. It does not explicitly answer whether the result deserves to affect the model state.

The Qwen mechanism

Qwen's intervention is applied after the attention operation:

\widetilde{y}_h = g_h \odot \operatorname{Attention}_h(Q,K,V)

The gate is computed from the current token representation:

g_h = \sigma(xW_h^g)

The sigmoid keeps each gate value between zero and one. A value close to one preserves the head's output; a value close to zero makes the head contribute little at that token. The attention pattern can still identify Token A as the best available source, while the gate can independently decide that the resulting update is too weak to write forward.

This is a modest addition rather than a different attention algorithm. Qwen still performs the attention computation, so the design is not a direct reduction in full-attention cost. It is a write-control mechanism: compute the result, then decide how much of it survives.

Why the placement matters

The Qwen team did not assume that any gate would work equally well. It evaluated more than 30 configurations that varied the gate's input, whether it was shared across heads, whether it was additive or multiplicative, and where it appeared in the attention block.

The strongest reported setup was:

• query-dependent
• specific to each attention head
• multiplicative sigmoid gating
• placed immediately after scaled dot-product attention

These choices give the mechanism a specific meaning. A gate on a value vector asks whether a source token offers useful information in general. An output gate instead asks whether the information retrieved for this query is worth using now. A function definition can matter while producing a call site and be irrelevant while completing a comment; the gate can make that decision separately for each token and head.

The query-dependent design also avoids treating a head as simply on or off. The same head can be active at one position and quiet at the next. That is closer to how attention heads are normally used: their value depends on the local computation, not only on their identity.

Why attention sinks matter

The motivation becomes clearer through attention sinks. In many transformers, some heads send an unusually large share of their probability to the first token in a sequence, even when that token has little semantic relationship to the current prediction. The first token can become a stable place for probability the model does not want to use.

That behavior is not necessarily a bug in the learned model. Under a softmax constraint, routing probability to a relatively harmless token is one way to approximate a null result. But it is indirect: the model is using source selection to solve an output-suppression problem.

The output gate gives the model a direct alternative. In one reported baseline, average attention to the first token dropped from 46.7 percent to 4.8 percent after output gating. The result does not prove that every attention sink has one cause, but it supports the intended interpretation: when the head has an explicit suppression path, it relies less on a sink token as a workaround.

This is why the change is interesting beyond the percentage itself. It separates two jobs that had been entangled in standard attention. The distribution decides where to read. The gate decides whether that read should write.

What the experiments show

The gates were not inert parameters. In the reported models, many gate values were close to zero, and different heads were active for different tokens and contexts. That is consistent with the design goal: a head need not contribute uniformly at every position just because it was computed.

The experiments also report a training effect. Gated models produced smaller activation spikes and tolerated more aggressive learning rates in runs where conventional models became unstable. The paper is appropriately cautious about the exact explanation. The gate may limit extreme residual updates, reduce indirect cancellation through attention sinks, add a helpful nonlinearity, or combine several of these effects. The experiments establish that the behavior changed; they do not settle a single causal story.

The design also did not turn attention into computationally sparse attention. The model still retrieves and combines values before gating them. Any future speedup would require showing that gate values can be predicted or exploited early enough to skip work. The reported result is narrower: selective writes are useful even when every head still executes.

What happens at longer contexts

The most striking result in Qwen's paper appears when models are evaluated beyond their training context length. Within the trained range, the baseline and gated versions performed similarly on the reported long-context benchmark. When the context was extended, the gated version degraded more slowly: at 64K tokens, the baseline scored roughly 38 while the gated model scored about 67; at 128K, the scores were roughly 32 and 59.

Those numbers belong to specific model configurations and a particular context-extension method, so they should not be treated as a universal guarantee. Gated Attention is not a complete long-context solution. But the result fits the mechanism being tested. A sink-based workaround depends on token positions and attention patterns that may shift outside the training regime. An output gate represents suppression directly, so it has less reason to depend on a particular token continuing to play the same role.

Where this appears in Qwen3-Next

Qwen later uses the mechanism in Qwen3-Next as part of a hybrid architecture rather than as an isolated patch. Most token mixing is handled by recurrent Gated DeltaNet layers, while periodic full Gated Attention layers provide direct retrieval across the context. Mixture-of-experts routing controls feed-forward capacity, and the output gate regulates whether a full-attention result should pass forward.

That division of labor is useful context for the gate. Qwen is not arguing that all computation should be replaced with a single selective mechanism. It assigns different roles to recurrent state, full attention, routing, and output control. Gated Attention is the part responsible for making a retrieved signal optional.

The practical takeaway

The Qwen contribution is small enough to look obvious after the fact, but it names a concrete choice that ordinary attention leaves implicit. A head can make three distinct decisions:

1. Where should it read from?
2. What representation should it retrieve?
3. Should that retrieved result be written at all?

Standard attention handles the first two. Qwen adds the third with a learned, per-head output gate. The reported improvements in sink behavior, optimization, and context extension make that choice worth paying attention to, but the core claim remains deliberately narrow: when an attention result is not useful, the architecture should be able to represent that directly.

References

1. Qiu et al., "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free", 2025.
2. Xiao et al., "Efficient Streaming Language Models with Attention Sinks", 2023.
3. Qwen Team, Qwen3-Next model documentation.