03 · Grouped-Query Attention

Let many readers share a few notebooks .

A model reads text in parallel using several attention heads. Think of them as readers with different jobs. Normally every reader keeps its own copy of the past. Grouped-Query Attention puts readers into teams and gives each team one shared copy instead.

the shared copies are called key/value heads · ainslie et al. 2023

Read the paper

part 1 of 5

One sentence, many readers

A model does not read a sentence just once. Several attention heads read it at the same time, each looking for something different. Pick a job below and watch the focus move.

Give each reader a job

Choose a question. The same sentence lights up differently.

Thesmallredfoxquicklycrossedthefield

Answer: fox

Reader 1 looks for the main character.

These readers are attention heads. Each head asks its own kind of question about the same text.

part 2 of 5

The notebooks pile up

To answer the next word, every reader needs the earlier text. If every reader keeps a private notebook, a longer conversation gets copied again and again.

Make the conversation longer

Every new word adds one saved page to every notebook. Drag the conversation length and watch the piles grow.

words remembered6

one notebook per reader

8 notebooks

6 words × 8 notebooks = 48 saved pages

four readers share each notebook

2 notebooks

6 words × 2 notebooks = 12 saved pages

The notebooks are the KV cache. GQA makes the cache smaller by keeping fewer notebooks, not by forgetting words from the conversation.

part 3 of 5

Give each team one notebook

GQA puts nearby readers into teams. Readers still have different jobs, but everyone on a team reads the same saved copy of the past. Build a few team layouts and trace any reader to its notebook.

Build the teams

First choose the number of notebooks. Then tap any reader to see which notebook it uses.

tap a reader

Notebook 1

Q1–Q4

Notebook 2

Q5–Q8

Reader Q4 → Notebook 1

The reader keeps its own question, but reuses this notebook’s saved text.

part 4 of 5

Choose how much to share

There is no single fixed amount of sharing. One notebook per reader uses the most memory. One notebook for everyone uses the least. GQA deliberately lives between those two ends.

Find the useful middle

Move from one shared notebook to one notebook per reader. More notebooks preserve more separate views; fewer notebooks save more memory.

sharing level2 notebooks

current choice

GQA · group sharing

8 readers · 2 notebooks

memory used25%

room for separate stored viewsbalanced

part 5 of 5

Smaller memory, same number of readers

The payoff grows with longer conversations. Choose a conversation length and a notebook count to see how much saved memory the model must carry while it generates each new word.

Try it on a real-size model

This example has 32 readers. Choose how much text it remembers and how many shared notebooks it keeps.

conversation length

shared notebooks

one notebook per reader

32 GB

32 readers · 32 stored notebooks

your shared setup

8.0 GB

32 readers · 8 stored notebooks

without sharing32 GB

with sharing8.0 GB

You kept all 32 question-askers, but the saved past is 4× smaller. In model language: 32 query heads share 8 key/value heads.

canonical paper

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Ainslie et al. · 2023

Loading the interactive visualization…

02 · Reference notes

← All visual guides

Grouped-Query Attention Explained

Grouped-query attention (GQA) reduces KV-cache size by letting groups of query heads share key and value heads, making LLM decoding more memory-efficient.

01 · Definition

What is Grouped-Query Attention?

Each query head can still ask a different question about the context. The sharing happens only in the saved key and value representations that must be read repeatedly while a model generates tokens.

02 · Mechanism

Why the KV cache matters

During autoregressive decoding, a model stores key and value vectors for every earlier token. With many attention heads and a long conversation, that cache can dominate memory bandwidth and limit how many requests a server can run at once.

03 · Mechanism

How GQA groups heads

Multi-head attention gives every query head its own key and value head. Multi-query attention shares one KV head across all queries. GQA is the middle ground: several query heads form a group and read from one shared key/value head.

04 · Mechanism

What changes and what stays separate

The query projections remain separate, so heads can retain distinct attention patterns. Only the cache is shared. That is why GQA can deliver most of the memory benefit without collapsing all heads into one reader.

05 · Mechanism

When GQA helps

The benefit is strongest at inference, especially for long contexts and batch-heavy serving. Fewer KV heads mean less cache storage and less data moved from memory for each generated token; the trade-off is choosing enough groups to retain quality.

Source · Original paper

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints ↗Ainslie et al. · 2023