A 4-Bit Model Does Not Set the GPU Memory Requirement

Read what the 4-bit label actually covers

Weights are the model's learned numerical parameters. Quantization represents selected values using fewer bits. A four-bit value has sixteen possible bit patterns; the specified format determines their meaning. In a weight-only offer, that description does not establish four-bit activations, the intermediate values produced during computation.

Hugging Face's Transformers v5.20.2 bitsandbytes documentation illustrates the distinction. Quantized linear layers replace standard linear layers, while other modules, such as LayerNorm, have a separately controlled dtype. Its four-bit example also specifies a separate compute dtype, including a bfloat16 option. Storage width therefore does not describe every module or every calculation.

For a buyer, the useful identity is the exact model revision and supported quantized representation, including what remains at another precision. Two offers saying only 4-bit have not yet established that they supply the same representation.

References: [H1]

Keep the 16 GB versus 4 GB arithmetic inside its assumptions

Consider a fictional model with exactly eight billion stipulated scalar weight values. Assume every value uses the same width and ignore everything except those values. At sixteen bits each, eight billion times sixteen divided by eight gives 16,000,000,000 raw bytes. At four bits each, the same calculation gives 4,000,000,000 raw bytes.

Those are 16 GB and 4 GB in decimal units, where one GB is one billion bytes. They are ideal raw-weight totals. They are neither measured checkpoint sizes nor free-memory requirements, and they do not prove fit on any named GPU. An actual representation can include different widths and additional data.

The example reduces one term by 12 GB, or a factor of four. Even if every other memory population stayed unchanged, adding those unchanged populations would make the reduction in the combined total less than fourfold. In a real runtime, allocation policies and working memory can also change. The raw-weight ratio alone cannot establish the peak-memory ratio or the price of completing the requested work.

Count the representation around the packed weights

Packing values is only part of a quantized representation. Hugging Face's quantization concept guide explains that two INT4 values can be packed into one byte. It also describes scales and, for asymmetric affine schemes, zero-points used to interpret quantized values. Parameters can be shared across a tensor or retained for smaller channels or groups, changing the additional storage required. These are examples of format-dependent metadata, not a claim that every four-bit format uses the same scheme.

The complete model footprint must include the packed values, applicable metadata and modules retained at another precision. File packaging and runtime representation also need their own accounting. Multiplying a total parameter count by half a byte omits these distinctions.

Quantization approximates numerical values. Its resource benefit must therefore be considered alongside evidence that the offered representation preserves adequate quality for the buyer's task.

References: [H2][H1]

Follow memory beyond the model footprint

NVIDIA's TensorRT-LLM memory reference separates weights, internal activation tensors and input/output tensors, with the KV cache a major part of the last category. Its TensorRT-engine discussion relates activation requirements to the graph and supported tensor shapes. A checkpoint file contains no measurement of those live populations.

TensorRT's architecture documentation adds execution-context memory, intermediate activations and temporary scratch space used by layer implementations. It describes reusing memory where tensor lifetimes permit, and notes that CUDA infrastructure and device code also consume memory. Adding every tensor's individual maximum would therefore be a poor substitute for observing the complete runtime.

The buying implication is to request the relevant peak over the required operating stages, with the metric and units identified. Distinguish successful loading, idle residency and serving the agreed requests. Record any startup requirement on the purchased allocation as well. A quiet loaded model cannot demonstrate that prompt processing and generation will fit at the requested load.

References: [N1][N2]

Give the KV cache its own representation and workload

The attention key-value cache retains computed state from processed tokens for reuse during autoregressive generation. It is a separate population from the learned weights. Transformers v5.20.2 describes a dynamic cache that grows as generation advances and a static cache that reserves a maximum size. Sliding-window or chunked-attention layers can limit growth, so context length alone is not a universal memory formula.

The same documentation presents quantized KV caching as a separate choice. A weight-only four-bit model does not establish a four-bit cache. The offer should identify the cache representation and allocation strategy actually used by the supported configuration.

For comparison, keep prompt lengths, generated lengths, concurrent requests and any assumed cache reuse consistent. More active sequence state can consume capacity released by smaller weights. That may be a useful use of the saving, but it provides no automatic basis for buying less memory.

References: [H3]

Compare an allocation offer with evidence for the requested use

Consider two fictional offers for the same requested base-model revision and inference task. The buyer needs long-document questions during busy periods with several simultaneous users. Both sellers propose a smaller GPU allocation on the strength of four-bit weights.

Offer A cites a small checkpoint file and the ideal raw-weight calculation. It supplies no complete representation description or relevant runtime evidence. The file size is useful storage information, but the proposed allocation remains unsubstantiated for this use.

Offer B identifies its exact quantized artifact, converted and retained module types, compute dtype, cache representation, runtime/backend versions and GPU allocation. It also supplies a report of measured peak memory for the agreed prompt and output lengths, concurrency and cache conditions, together with task-quality results against the agreed reference. These are stipulated contents of the fictional offer, not reported test outcomes.

B gives the buyer a basis to assess the smaller allocation. Its evidence still has to match the proposed configuration and satisfy the buyer's capacity and quality requirements. A short single-request run would leave the busy-period requirement open. A different quantized artifact would leave the offered representation unverified. If capacity is measured across multiple devices, retain the per-device results rather than treating their sum as one allocation.

The decision is conditional: accept the smaller allocation when relevant evidence supports the required use and the project has allowed for its actual operating variability. Otherwise retain a larger option or obtain the missing evaluation through the appointed software provider. There is no universal reserve percentage that repairs an incomplete comparison.

Price the same useful result. A fourfold raw-weight reduction establishes neither fourfold cheaper inference nor equal output quality. Compare the complete offered resources and supported operating conditions, including the required response time and completed work. A smaller memory line item has value when the resulting service still meets the buyer's need.

Bring the selected hardware to the physical deployment inquiry

For customer-furnished hardware, carry the selected OEM system, GPU configuration and quantities into the installation package. Leviathan's rack-and-stack service covers agreed equipment mounting, physical IT cabling, labels and installation records at customer sites or authorized colocation spaces. The related service page and project inquiry provide the next step for discussing that physical work.

Bring the rack layout, rail and mounting details, cable endpoints, site authorization and required handoff. Keep model evaluation and inference-capacity acceptance with the appointed software team. Software commissioning is not a standard rack-installation inclusion; the physical scope does not establish application performance or approve a model representation.

References: [L1]