The point is not that Qwen replaces attention or makes every head sparse. It is that standard attention has an awkward blind spot: it can decide which source is best, but it cannot directly say that no source is useful enough to write forward. Qwen gives each head that additional control.

Why attention needs a write control

In ordinary attention, a head scores earlier tokens, normalizes those scores with softmax, and combines the corresponding value vectors. A simplified distribution might look like this:

Those weights must sum to one:

That is sensible when at least one visible token contains information the head needs. The difficulty is that the distribution still has to choose a destination when every candidate is poor. Softmax can rank options; it does not preserve a clean signal for whether the winning option was useful in absolute terms.

Consider the logits 10, 9, 8 and -100, -101, -102. Their relative gaps are the same, so softmax produces nearly the same distribution for both. Yet the second set can reasonably be read as a head finding nothing compelling in the context. Standard attention has no separate control that lets the head act on that distinction.

The Qwen paper frames this as a missing write decision. Attention already answers where to read and what values to combine. It does not explicitly answer whether the result deserves to affect the model state.

The Qwen mechanism

Qwen's intervention is applied after the attention operation:

The gate is computed from the current token representation:

The sigmoid keeps each gate value between zero and one. A value close to one preserves the head's output; a value close to zero makes the head contribute little at that token. The attention pattern can still identify Token A as the best available source, while the gate can independently decide that the resulting update is too weak to write forward.

This is a modest addition rather than a different attention algorithm. Qwen still performs the attention computation, so the design is not a direct reduction in full-attention cost. It is a write-control mechanism: compute the result, then decide how much of it survives.

Why the placement matters

The Qwen team did not assume that any gate would work equally well. It evaluated more than 30 configurations that varied the gate's input, whether it was shared across heads, whether it was additive or multiplicative, and where it appeared in the attention block.

The strongest reported setup was:

• query-dependent
• specific to each attention head
• multiplicative sigmoid gating
• placed immediately after scaled dot-product attention

These choices give the mechanism a specific meaning. A gate on a value vector asks whether a source token offers useful information in general. An output gate instead asks whether the information retrieved for this query is worth using now. A function definition can matter while producing a call site and be irrelevant while completing a comment; the gate can make that decision separately for each token and head.

The query-dependent design also avoids treating a head as simply on or off. The same head can be active at one position and quiet at the next. That is closer to how attention heads are normally used: their value depends on the local computation, not only on their identity.

Why attention sinks matter

The motivation becomes clearer through attention sinks. In many transformers, some heads send an unusually large share of their probability to the first token in a sequence, even when that token has little semantic relationship to the current prediction. The first token can become a stable place for probability the model does not want to use.

That behavior is not necessarily a bug in the learned model. Under a softmax constraint, routing probability to a relatively harmless token is one way to approximate a null result. But it is indirect: the model is using source selection to solve an output-suppression problem.

The output gate gives the model a direct alternative. In one reported baseline, average attention to the first token dropped from 46.7 percent to 4.8 percent after output gating. The result does not prove that every attention sink has one cause, but it supports the intended interpretation: when the head has an explicit suppression path, it relies less on a sink token as a workaround.

This is why the change is interesting beyond the percentage itself. It separates two jobs that had been entangled in standard attention. The distribution decides where to read. The gate decides whether that read should write.

What the experiments show

The gates were not inert parameters. In the reported models, many gate values were close to zero, and different heads were active for different tokens and contexts. That is consistent with the design goal: a head need not contribute uniformly at every position just because it was computed.

The experiments also report a training effect. Gated models produced smaller activation spikes and tolerated more aggressive learning rates in runs where conventional models became unstable. The paper is appropriately cautious about the exact explanation. The gate may limit extreme residual updates, reduce indirect cancellation through attention sinks, add a helpful nonlinearity, or combine several of these effects. The experiments establish that the behavior changed; they do not settle a single causal story.

The design also did not turn attention into computationally sparse attention. The model still retrieves and combines values before gating them. Any future speedup would require showing that gate values can be predicted or exploited early enough to skip work. The reported result is narrower: selective writes are useful even when every head still executes.

What happens at longer contexts

The most striking result in Qwen's paper appears when models are evaluated beyond their training context length. Within the trained range, the baseline and gated versions performed similarly on the reported long-context benchmark. When the context was extended, the gated version degraded more slowly: at 64K tokens, the baseline scored roughly 38 while the gated model scored about 67; at 128K, the scores were roughly 32 and 59.

Those numbers belong to specific model configurations and a particular context-extension method, so they should not be treated as a universal guarantee. Gated Attention is not a complete long-context solution. But the result fits the mechanism being tested. A sink-based workaround depends on token positions and attention patterns that may shift outside the training regime. An output gate represents suppression directly, so it has less reason to depend on a particular token continuing to play the same role.

Where this appears in Qwen3-Next

Qwen later uses the mechanism in Qwen3-Next as part of a hybrid architecture rather than as an isolated patch. Most token mixing is handled by recurrent Gated DeltaNet layers, while periodic full Gated Attention layers provide direct retrieval across the context. Mixture-of-experts routing controls feed-forward capacity, and the output gate regulates whether a full-attention result should pass forward.

That division of labor is useful context for the gate. Qwen is not arguing that all computation should be replaced with a single selective mechanism. It assigns different roles to recurrent state, full attention, routing, and output control. Gated Attention is the part responsible for making a retrieved signal optional.

The practical takeaway

The Qwen contribution is small enough to look obvious after the fact, but it names a concrete choice that ordinary attention leaves implicit. A head can make three distinct decisions:

1. Where should it read from?
2. What representation should it retrieve?
3. Should that retrieved result be written at all?

Standard attention handles the first two. Qwen adds the third with a learned, per-head output gate. The reported improvements in sink behavior, optimization, and context extension make that choice worth paying attention to, but the core claim remains deliberately narrow: when an attention result is not useful, the architecture should be able to represent that directly.

References

• Qiu et al., "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free", 2025.
• Xiao et al., "Efficient Streaming Language Models with Attention Sinks", 2023.
• Qwen Team, Qwen3-Next model documentation.

The appeal is obvious. There is one model, one transcript, and one apparent centre of intelligence. The system is easy to explain and surprisingly easy to prototype.

I am increasingly unsure that this should be the default architecture for serious coding agents.

*Diagram: Monolithic Agent versus Planner–Executor Architecture*

A coding agent does not feel to me like a single act of intelligence stretched across many tokens. It feels more like a small organization—or, less generously, a small distributed system. There is planning, retrieval, observation, implementation, verification, and revision. These activities depend on one another, but they are not quite the same kind of work.

Some parts require broad judgment. Others require patience and mechanical discipline. Some require deciding what evidence would be useful; others require turning noisy output into a compact account of what happened. It seems wasteful to assume that the same model should perform all of them simply because it can.

The architecture I currently find most convincing is some version of a planner–executor system. A frontier model handles the parts of the loop where judgment matters most: understanding the task, forming hypotheses, deciding what to inspect, and determining whether the evidence supports the current approach. Smaller or more specialized models handle much of the operational work around it, while tests, linters, browsers, and other tools keep the process tied to reality.

This is partly a cost argument, but I do not think cost is the interesting part. The more important question is where model capability has the highest marginal value.

My intuition is that the strongest model should spend its attention deciding what matters, not reading every line emitted by the terminal.

Consider a bug report: “The checkout button disappears on mobile after applying a coupon.” An agent can begin by searching the repository for the checkout button and gradually collect whatever looks relevant. Sometimes this works. But in a large codebase, search is not a neutral activity. Every file opened and every plausible fragment retrieved changes the context in which the model makes its next decision.

The danger is not only that the agent wastes time. It may begin treating whatever happens to be present in context as evidence that it matters. A few loosely related files become the working theory of the bug simply because they were found first.

A planner can begin at a slightly higher level. Is this likely to be a responsive layout issue? Does applying the coupon change some cart-validity state? Is the frontend rendering stale derived data? Is the apparent UI failure actually downstream of a product-state transition?

The initial theory does not need to be correct. Its value is in determining what evidence to seek next. Good planning is less about predicting the answer immediately and more about choosing investigations that make the space of possible answers smaller.

Once that direction has been set, much of the next work is comparatively routine. Open the relevant files. Reproduce the failure. Run a targeted test. Extract the meaningful part of a trace. Record what changed and what did not.

This is where I think smaller models are underrated. A model does not need to understand the entire repository to report that the coupon request succeeded, a cart-state flag changed, and the button’s visibility depends on that flag. It needs a narrow interface, a clear task, and some discipline about what it returns.

The point is not to replace reasoning with cheap models. It is to protect reasoning from everything that does not deserve to occupy it.

Raw tool output is usually a poor form of memory. Test logs contain repeated stack frames, warnings, timing noise, and environmental debris. Browser traces contain hundreds of events that are technically real but strategically useless. Repository search produces fragments that are individually plausible and collectively confusing.

Giving all of this to the strongest model can feel like the safest option because nothing has been hidden. In practice, though, a large context window is not the same thing as a faithful memory. Information can remain present while its relevance becomes increasingly difficult to judge.

A distillation layer makes that compression explicit. Instead of carrying the entire history of the run forward, the system carries a compact account of what it currently believes, what it has observed, and what remains uncertain. The planner can then reason from a cleaner state rather than repeatedly reconstructing the investigation from terminal exhaust.

There is an engineering benefit here as well. When the system has explicit boundaries, its failures become easier to discuss. Did it inspect the wrong part of the repository? Did it omit an important observation? Did it generate a patch that violated the intended constraint? Did it accept weak verification?

In a monolithic agent, all of these often collapse into the same diagnosis: the model failed. That is not a particularly useful way to improve a system.

I think a similar argument applies to code generation, although less cleanly. There are changes where the architectural reasoning and implementation are too closely connected to separate: subtle concurrency bugs, difficult migrations, or refactors where the shape of the patch is itself the main design decision. In those cases, the frontier model may need to remain deeply involved.

But a large amount of coding work becomes local once the problem has been framed properly. If the relevant files, expected behaviour, constraints, and verification target are known, a capable coding model can often produce the patch. The quality of the result then depends heavily on the quality of the instruction it receives.

“Fix the bug” is not meaningful delegation. “Change this behaviour without altering this interface, avoid a broad refactor, and add a regression test for this case” is much closer.

This resembles, in a loose way, how strong engineering teams already operate. The most experienced engineer is not valuable because they personally type every line. Their value often lies in framing the problem, spotting bad assumptions, preserving conceptual integrity, and deciding whether an implementation actually solves what it claims to solve.

A planner–executor agent gives the frontier model a similar role. It shapes the work, delegates a constrained change, reviews the result, and decides whether the evidence is strong enough to stop.

The obvious weakness is that every handoff can lose information. A smaller model may omit the one important line in a log. An executor may misunderstand the planner. A compressed state may hide a clue that would have changed the investigation.

I do not think this is a minor objection. Poorly designed decomposition can easily perform worse than a coherent single-model loop.

But the alternative is not perfect information. A monolithic agent also compresses its history, only less visibly. It relies on attention to preserve relevance across an expanding transcript and on the model to maintain a stable internal account of what has happened. Compression still occurs; it is simply implicit and difficult to inspect.

Planner–executor systems make the compression boundary part of the design. They force us to decide what counts as evidence, what state should survive, what an executor is allowed to change, and what verification is required before the system can claim success.

Those choices may matter more than the names of the models involved.

I would hesitate to claim that planner–executor systems are the final form of coding agents. Models are improving quickly, and some of the distinctions that seem important today may become unnecessary. For small repositories and short tasks, a single capable model may remain the simplest and best option.

My thesis is narrower. As coding agents take on longer tasks in larger repositories, I expect the separation between judgment and execution to become more useful, not less. The hard part is often not producing code. It is maintaining a coherent view of the problem while deciding what should be done, what should be ignored, and when the available evidence is good enough.

The strongest model is not the agent. The agent is the entire loop: models, tools, memory, tests, constraints, and review.

The goal should not be to keep the largest model busy. It should be to spend intelligence deliberately.