08 · Sliding-Window Attention

Look nearby often. Look everywhere only sometimes .

Most language patterns are local: nearby words explain one another. Sliding-window attention lets each token read only a recent neighbourhood. Stacked layers pass information farther, and occasional global layers can reconnect the entire context.

moving local window + local/global layer stacks · mistral & gemma

Read the paper

part 1 of 5

Each token gets a moving spotlight

Instead of comparing with the whole past, a local attention layer reads only the nearest earlier tokens. Move the current token and resize its spotlight.

current token11

window size4

The spotlight moves with the current token. Tokens outside it are not compared in this layer.

part 2 of 5

Local messages travel through depth

One layer reaches only nearby tokens. But its output already contains their information, so the next layer can carry that information another step.

Stack more local layers

A token can receive information from a neighbour that already received information from its neighbour.

layers stacked3

After 3 layers, information can travel about 6 tokens through this toy window.

part 3 of 5

A distant fact can fall outside the light

A small window saves work, but one layer cannot jump directly to an old clue. Try widening the window until the final question can see the fact.

Can the question still see the old fact?

The project code appears near the beginning. The question arrives much later.

window size5

ORION

code?

The fact is outside the window.

Local attention needs deeper layers or an occasional global layer to recover it.

part 4 of 5

Occasional global layers reconnect everything

Modern models can mix many cheap local layers with a few layers that read the full context. Arrange the stack and see how often long-range access returns.

Place a global layer every few local layers

Local layers handle nearby patterns cheaply. A global layer periodically reconnects the whole sequence.

1local

2local

3local

4local

5global

6local

7local

8local

9local

10global

11local

12local

13local

14local

15global

16local

17local

18local

local layers

global layers

long-range reset

every 5 layers

part 5 of 5

Pay for the neighbourhood, not the whole city

A local layer’s work depends on window size rather than total context length. Compare the reads, then notice the trade-off: smaller windows delay distant information.

Set the local attention budget

Compare how many earlier tokens each layer reads.

context

window

global layer reads65,536

local layer reads1,024

local-layer saving

64× fewer

Only 2% of the context is read directly.

trade-off

Small windows are cheap but delay distant information. Global layers restore direct long-range access at a higher cost.

canonical papers

Longformer: The Long-Document Transformer Beltagy et al. · 2020 Mistral 7B Jiang et al. · 2023

Loading the interactive visualization…

04 · Reference notes

← All visual guides

Sliding-Window Attention Explained

Sliding-window attention limits each token to a local context window, reducing attention cost while stacked or occasional global layers preserve longer-range information.

01 · Definition

What is Sliding Window Attention?

It is a structured sparse-attention pattern: rather than compare every token with every earlier token, each layer gives a token a fixed local neighborhood to read.

02 · Mechanism

Why use a local window

Full attention grows quadratically with context length because every token can compare with every other token. A fixed window makes each token attend to only a bounded number of neighbors, so the attention work grows much more gently with sequence length.

03 · Mechanism

How information travels farther

A single local layer cannot connect distant tokens directly. But stacking local layers expands the receptive field: information can pass from one neighborhood to the next. Some architectures also add global or full-attention layers to reconnect the entire context at selected depths.

04 · Mechanism

What the window changes

The model gets reliable nearby context at every layer and gives up immediate access to arbitrary far-away tokens. Window size, layer depth, and the placement of global layers determine the balance between cost and long-range recall.

05 · Mechanism

When sliding windows help

They are useful when local syntax, recent instructions, or nearby evidence carry most of the signal. They reduce compute and cache pressure for long documents, while architectural choices preserve enough long-range communication for the task.

Source · Original paper