The Query-Head Count Does Not Set a Model's KV-Cache Population

A query uses retained state without becoming another retained KV head

In scaled dot-product attention, a query is compared with keys. Softmax turns the scaled scores into weights, and those weights combine the corresponding value vectors. Multiple attention heads perform these operations before their outputs are combined by an output projection. Query/key width and value width are distinct dimensions in the original attention formulation.

During autoregressive generation, previously processed positions supply reusable keys and values. Those vectors form the KV cache, separately for each applicable layer. Processing another position adds its state; the current query uses the available state to produce an attention result. A conventional KV cache does not need to retain a new historical query vector alongside every historical key and value.

The learned projection weights and the cached vectors also describe different populations. The weights define transformations; the cache contains state computed for a particular sequence. An architecture's head counts help describe that state, but the number of query heads alone leaves its retained KV-head dimension unresolved.

References: [AT1][HC1]

MHA, GQA and MQA change who shares a key/value pair

Use eight query heads to follow the distinction. In conventional multi-head attention, or MHA, each query head has its own corresponding key head and value head: eight query heads, eight key heads and eight value heads. Here, eight KV heads means eight heads of each type, not eight vectors split between K and V.

Grouped-query attention, or GQA, divides query heads into groups. Each group shares one key head and one value head. With two equal groups, query heads 1 through 4 use one KV-head pair and heads 5 through 8 use another. There are two groups of four query heads, so the KV-head count is two. Confusing group size with group count would produce the wrong cache calculation.

Multi-query attention, or MQA, shares a single key head and value head across all eight query heads. Sharing applies to the head dimension: each retained position still has its own vectors in each applicable layer. In every case the eight queries can produce different attention weights and eight head outputs, because their query vectors can differ even when their keys and values are shared.

This comparison follows the GQA paper's Figure 2 and Section 2.2 in arXiv:2305.13245v3, revised December 23, 2023. Shazeer's 2019 MQA formulation separately shows the query and output head dimensions remaining while the K/V head dimension is removed.

References: [GQ1][MQ1]

Count layers, positions, KV heads, width and bytes separately

For one sequence, assume every counted layer retains T positions, has Hkv key heads and Hkv value heads, and uses width d for both K and V. Assume each scalar occupies b bytes. Storing each distinct element once gives raw KV bytes = 2 × layers × T × Hkv × d × b. The factor two counts K plus V; it does not count two model replicas.

This is a populated-state calculation with tightly packed, uniformly sized values. It excludes model weights, activations, attention workspace, pages and rounding, allocator pools, replicas and sharding. It is neither a whole-model footprint nor a per-GPU allocation forecast.

Keep the inputs independent. Hq is the query-head count; Hkv is the head count used in this storage calculation. T is the number of retained positions at the instant being described, including processed prompt and generated positions that remain cached. It is not automatically the advertised maximum context or a requested output allowance that has yet to be processed.

The width d counts scalar elements per head, while b specifies their storage size. A two-byte assumption applies to both cached K and V here; a weight-precision label cannot supply it. If key and value widths or precisions differ, count their bytes separately and add them. If layers or sequences retain different lengths, sum those contributions instead of forcing them into one common T.

References: [HC1][PT1]

Eight query heads, three different raw cache sizes

Consider an invented arithmetic example with one sequence, four cached self-attention layers, 1,024 retained positions per layer, head width 64 and two bytes per scalar in both K and V. All three cases have eight query heads. Every distinct KV element is stored once, with the exclusions stated above.

In the MHA case, Hkv = 8. The raw count is 2 × 4 × 1,024 × 8 × 64 × 2 = 8,388,608 bytes. That is 8 MiB, using 1 MiB = 1,048,576 bytes.

In the two-group GQA case, Hkv = 2. The count is 2 × 4 × 1,024 × 2 × 64 × 2 = 2,097,152 bytes, or 2 MiB. Each group contains four query heads, but it contributes one key head and one value head to the retained state.

In the MQA case, Hkv = 1. The count is 2 × 4 × 1,024 × 1 × 64 × 2 = 1,048,576 bytes, or 1 MiB. All eight query heads use the shared KV-head pair.

All three cases have eight query heads and produce eight head outputs before output projection. They stipulate different KV-head structures, not three runs of an unchanged model. The fourfold reduction from 8 to 2 MiB applies to raw KV state; it does not establish a matching reduction in total GPU memory or four times as many concurrent requests.

The same arithmetic also exposes a context comparison trap. Keeping the GQA case's other assumptions fixed but retaining 4,096 positions gives 2 × 4 × 4,096 × 2 × 64 × 2 = 8 MiB. A smaller KV-head count can be offset by more retained positions. Neither case demonstrates that an application can use that longer context successfully.

A smaller KV-head count belongs to a model structure and checkpoint

Changing eight independently learned KV-head pairs into two shared pairs changes the architecture and its key/value projection weights. It is not a free runtime switch that turns the original model into a smaller-cache equivalent with identical outputs.

The December 2023 GQA paper describes checkpoint conversion by averaging key/value projection weights within groups, followed by additional pretraining. Its experiments use T5.1.1 encoder-decoder models; the stated limitations explicitly leave decoder-only evaluation outside those experiments. Its reported quality and timing comparisons belong to that study's models, tasks and training conditions.

For a buyer, the consequence is to identify the exact offered model and checkpoint before comparing memory. A converted or differently trained GQA artifact needs quality evidence for the intended task. The historical conversion method explains an architectural tradeoff; it is not a buyer conversion procedure or a guarantee that any selected model retains acceptable quality.

References: [GQ1]

A valid attention shape does not prove compact physical storage

PyTorch 2.14's scaled_dot_product_attention reference, read October 20, 2026, distinguishes query-head count Hq from key/value-head count H. Its output carries Hq. The documented GQA constraints require Hq to be divisible by H and the key and value head counts to match. Thus eight query heads and two KV heads meet those head-count constraints; that alone does not establish support for every input or execution path.

The same reference marks GQA experimental and documents backend and tensor limitations. Its illustrative equivalent calculation repeats keys and values along the head dimension. That describes mathematical behavior; it does not establish whether a selected kernel materializes repeated tensors or how a serving engine retains its cache.

Consequently, neither the API's GQA option nor an input shape proves zero physical replication. Distinguish persistent cache storage from temporary attention inputs and workspace, and identify the actual implementation when interpreting a memory report.

References: [PT1]

Translate distinct state into the allocation actually offered

Growth policy changes the relationship between populated state and reserved capacity. Hugging Face's Cache strategies documentation, retrieved October 20, 2026, describes dynamic cache growth and static reservation of a maximum size. It also explains that sliding-window or chunked-attention layers stop growing at their respective limits. The retained position count can therefore differ by layer, and a reserved slot need not contain useful retained state.

Allocation granularity is another distinction. The September 12, 2023 PagedAttention paper describes fixed-token KV blocks allocated as needed. A partially filled final block can occupy more storage than its live elements alone require. Its block-based sharing mechanism is separate from GQA's sharing between query heads; one concerns storage management, the other model head structure.

Parallelism adds placement and duplication questions. The GQA paper notes replication of a single MQA KV head under the model-partitioning arrangement it discusses. Do not generalize either that replication pattern or a perfectly even split to every engine. Ask which layers and KV heads each device retains, and whether state is partitioned, replicated or shared. Independently served model replicas also bring their own state requirements.

For a long-context offer, connect the raw count to the actual cache representation, retained lengths, reserved capacity and per-device allocation. Keep runtime and allocator memory outside the raw-state arithmetic visible. The relevant evidence is the complete serving configuration under the required sequence lengths and simultaneous requests. Dividing free memory by the example's raw bytes would skip those dependencies.

References: [HS1][PA1][GQ1]

Use the raw count to test the explanation behind a capacity claim

Suppose a fictional supplier says that two models each have eight attention heads and therefore require the same KV storage. The worked example shows the missing field: eight query heads can coexist with eight, two or one KV heads. Request the architecture's separate head counts, widths, layers, cache precision and retained lengths before accepting the comparison.

Now suppose the supplier identifies two KV heads and claims four times as many simultaneous long-context users. The raw population may support a narrower statement about fewer cached values. Concurrency still needs evidence for the complete offered model, allocation, implementation, traffic and required response times, together with acceptable task outputs. The head-count ratio cannot settle that purchase.

The related guides retain the other decisions: weight quantization concerns the model representation, prefill/decode separation concerns placement of serving phases, and FlashAttention concerns attention intermediates. None supplies the missing KV-head count. Use the raw count to interpret retained state, then evaluate the supported serving arrangement with the appointed software provider.

Connect a separately selected equipment project to physical installation

If the selected system also needs physical installation, Leviathan's rack-and-stack service covers agreed equipment mounting, IT cabling, labels and installation records. Use the related service and project inquiry with a nonconfidential equipment list, site and released rack/cable package. Model selection, cache implementation and serving-capacity evaluation remain with the appointed software team.

References: [LS1]