A Vocabulary Table and a Request's Token Embeddings Count Different Memory

Follow one token ID to its learned row

Start after the compatible tokenizer has supplied integer token IDs. A token ID identifies an entry in a model's vocabulary; it is not itself the learned vector. PyTorch's word-embedding tutorial explains the table as V rows of D components, with index i selecting row i. The row values are model parameters learned through training.

At this lookup boundary, the same ID selects the same row of an unchanged table. The ordered input IDs determine which rows appear at which request positions. Later model operations can add position information and use surrounding context; the initial row lookup alone supplies no complete contextual interpretation.

A one-hot representation is a different object: a V-component selector with one entry equal to one and the others zero. Mathematically, multiplying that selector by a V x D table selects a row. This does not mean an indexed embedding lookup must construct and store a one-hot vector for every token. The PyTorch 2.14 Embedding interface accepts integer indices and returns their corresponding vectors.

An embedding service used for semantic answer caching represents a whole query for similarity matching. That is a different object from the per-token lookup output here: L input positions supply L vectors, rather than one query-level search vector. The related cache guide follows that query vector into a prior-answer match.

The related chat-template guide explains how message roles become compatible serialized input IDs. The vision guide follows flattened image patches through a learned projection. Here the input already consists of token IDs, and the operation selects stored rows.

References: [EM1][T1][Q1]

Keep vocabulary rows, vector width and request positions separate

V is the number of entries in the table. D is the number of numerical components in each entry. L is the number of positions in the request sequence being embedded. D describes vector width, rather than the number of tokens or the vocabulary size.

PyTorch 2.14 documents a weight shape of (num_embeddings, embedding_dim). Its output appends the embedding dimension to the input-index shape. For a single unbatched sequence of L IDs, that means an L x D output alongside the V x D table.

For the dense representations considered below, table-value bytes equal V x D x bytes per weight element. Materialized request-output bytes equal L x D x bytes per output element. Weight and output element widths are stated separately, even when the example makes them equal. These formulas count numerical payloads, without establishing their device placement or the allocator's actual reservation.

References: [EM1][B1]

EM-217: a 256 KiB table supplies an 8 KiB request output

Original fictional example EM-217 stipulates one unchanged, fully resident dense table with V = 1,024 entries and D = 64 components per entry. The request has L = 32 valid token-ID positions. Both table values and the separately materialized lookup output use FP32 at four bytes per component. There is no batching, padding, compression or shared output storage in this example. One KiB means 1,024 bytes.

The table contains 1,024 x 64 = 65,536 learned scalar values. At four bytes each, its value payload is 262,144 bytes, or 256 KiB. Selecting rows for a request does not create another learned table.

The request output contains 32 x 64 = 2,048 scalar values. Its payload is 2,048 x 4 = 8,192 bytes, or 8 KiB. Here, output means the embedding lookup's output tensor, before later model processing. It does not mean generated text or next-token scores.

Stipulate exactly 16 distinct IDs, each appearing twice across the 32 positions. Each occurrence selects an existing row. The table still has 1,024 learned rows, and the materialized output still has 32 positions. Counting distinct selected rows would describe 16 rows, but it would omit half the positions in this request representation.

While both stipulated payloads coexist, their sum is 256 + 8 = 264 KiB. That is the combined value payload of exactly these two objects. The integer input IDs, other weights, later activations, output logits, any training gradients and optimizer state, and allocator overhead are separate populations. EM-217 supplies arithmetic rather than a GPU fit, execution, speed, semantic-quality or billing result.

References: [B1]

Doubling vocabulary rows and doubling request positions change different objects

Consider a separately invented alternative table with V = 2,048, still D = 64 and FP32. Its learned-value payload is 2,048 x 64 x 4 = 524,288 bytes, or 512 KiB. If its request still has 32 positions and the same output width and element size, its materialized embedding output remains 8 KiB. The additional vocabulary entries enlarge the table, rather than adding request positions.

For a different comparison, return to EM-217's original 1,024-row table and stipulate a 64-position request. The table remains 256 KiB. That materialized output contains 64 x 64 x 4 = 16,384 bytes, or 16 KiB. More occurrences request more output vectors; they do not add learned entries to the fixed vocabulary.

These comparisons deliberately hold dimensions constant to isolate the counts. They do not establish that the same text would have the same token length under another vocabulary. A real vocabulary change needs compatible learned weights and token-ID mapping, followed by model evaluation.

Check whether the output projection shares the learned weights

The historical Transformer paper, Attention Is All You Need, distinguishes embeddings from the learned transformation and softmax that produce next-token probabilities. Section 3.4 states that its architecture shares one weight matrix between the two embedding layers and the pre-softmax linear transformation. This article uses version 7, revised August 2, 2023, of the 2017 paper.

Multiple uses of a matrix do not automatically imply multiple independent sets of learned values. Conversely, that paper's sharing choice does not establish that another model ties its input embeddings and output projection. Before adding a second full vocabulary-sized matrix to a parameter calculation, establish the actual architecture's sharing and runtime storage.

Sharing weights also does not turn next-token scores into the input embedding sequence. Scores over vocabulary choices and D-component input vectors describe different intermediate objects. Their lifetimes and materialization need their own accounting. EM-217 includes no output projection, logits tensor or probability tensor in its 264 KiB subtotal.

References: [EM2]

Sparse gradients do not establish compressed resident weights

PyTorch 2.14 defines sparse=True on Embedding as making the gradient with respect to the weight matrix a sparse tensor. That describes a gradient representation during training. It is not a declaration that the resident learned table uses compressed sparse row storage, commonly called CSR, or that only the IDs in this request occupy weight memory.

Nor does selecting 16 distinct rows change EM-217's stipulated dense table into a 16-row allocation. A claim about a compressed, sharded or otherwise different weight arrangement would require evidence about that arrangement. A sparse-gradient option alone cannot supply it.

References: [EM1]

Identify the counted object before comparing memory claims

Suppose one fictional offer reports 256 KiB of embedding weights and another reports 8 KiB of embeddings per 32-position request. Under EM-217's assumptions, both numbers can describe the same arrangement. Treating the second as a 32-fold reduction in learned-weight storage would compare a request output with a model parameter table.

Ask the software owner which object the figure measures, its shape, element representation and storage location. Establish whether repeated positions remain represented and whether reported parameter totals account for genuine weight sharing. Those answers determine whether the numbers can be compared at all.

A vocabulary table's dimensions do not rank models by semantic usefulness, and a shorter sequence does not establish a token bill or a speedup. For hardware capacity, carry the identified table and request representation into the complete memory assessment. The related quantization article covers other weights, cache state and runtime memory; the padding article covers the separate question of how multiple sequence lengths become a batch layout.

Connect selected equipment to a separate physical installation

If a separately selected hardware project needs mounting and physical IT connections, Leviathan's rack-and-stack service can scope the agreed equipment installation, cabling, labels and handoff records. Use the related service and project inquiry with a nonconfidential equipment list, rack layout and site requirements. Model selection, embedding implementation and software capacity evaluation remain with the appointed software team.

References: [L1]