PyTorch Allocated and Reserved Memory: Two Views of the Same Pool

Read the accounting boundary before comparing numbers

PyTorch 2.14 defines memory_allocated as the current GPU memory occupied by tensors. It defines memory_reserved as the current GPU memory managed by the caching allocator. Both API references specify bytes and a selected device; when the device argument is omitted, the statistic concerns the current device.

In the caching-pool account considered here, tensor-occupied storage is within the managed amount. Allocated and reserved therefore answer different questions about overlapping memory. Reserved does not mean an additional requirement alongside the tensors already inside it. It also describes a runtime allocator, rather than a cloud capacity booking.

Keep the exact metric name and units beside a supplied value. A line labeled model memory could refer to tensor occupancy, a pool total or a broader device report. The label alone leaves the comparison unresolved.

References: [P1][P2][P3]

Follow storage beyond the tensor's lifetime

PyTorch's caching allocator retains memory for reuse. Releasing tensor storage can reduce tensor occupancy while leaving that storage under allocator management. Its CUDA memory discussion explains why unused cached memory can continue to appear as used in nvidia-smi. A tensor's lifetime and its former storage's stay in the pool need not end together.

Attach the PyTorch version and allocator backend to the report. PyTorch 2.14 cautions that some memory statistics are specific to the native backend and have no meaningful interpretation with cudaMallocAsync. Check the applicable metric documentation before transferring an interpretation between backends.

References: [P3]

Place the reported pool inside the device account

The memory_allocated reference identifies retained cache and GPU context as reasons its value can be lower than nvidia-smi's. PyTorch's memory-usage guide adds a coverage distinction: its profiler normally sees device allocations managed through the PyTorch allocator. Direct CUDA allocations outside that allocator are absent from that view; NCCL is a documented example of a library making such allocations.

A whole-device figure can therefore include memory outside the pool being discussed. It is a broader observation, not a third independent demand to add to the tensor and pool figures. A complete reconciliation needs the same device and instant, a defined pool boundary and nonoverlapping outside allocations. Subtracting readings from different times or coverage scopes does not establish that outside amount.

References: [P1][P4]

AM-214: 12 GiB inside 20 GiB, plus 4 GiB outside

Fictional accounting example AM-214 stipulates one device, one instant and one caching pool. Tensors occupy 12 GiB within 20 GiB managed by the allocator. Separately, stipulate 4 GiB covering every other device-memory allocation outside that pool, with no overlap. These are invented observations, rather than measurements from a named GPU or a recommended operating margin. All example values use GiB; the cited APIs return bytes.

The portion of the pool not occupied by those tensors is 20 - 12 = 8 GiB. The complete stipulated usage is 20 + 4 = 24 GiB. Adding 12 + 20 + 4 = 36 GiB counts the tensor storage twice, because the first 12 GiB is already included in the 20 GiB pool.

An equivalent nonoverlapping breakdown is 12 GiB occupied by tensors, 8 GiB of the pool not occupied by those tensors, and 4 GiB outside the pool: 12 + 8 + 4 = 24 GiB. This alternative is valid because the example explicitly defines complete, disjoint populations at one instant.

The 8 GiB difference supplies no block layout or allocation-availability evidence. It cannot establish a contiguous 8 GiB allocation available to every request. The two totals alone do not describe how retained storage can satisfy a particular next allocation.

References: [P1][P2][P4]

Lower tensor occupancy can coexist with unchanged device usage

At a later stipulated instant in AM-214, tensor occupancy falls to 6 GiB. The managed pool remains 20 GiB and the complete outside allocation remains 4 GiB. Now 20 - 6 = 14 GiB of the pool is not occupied by those tensors. The complete usage is still 20 + 4 = 24 GiB.

The example has a changed tensor population and retained pool storage. Its unchanged 24 GiB device total alone does not establish a leak. Equally, this constructed explanation does not prove that a real application has no leak or that its future requests will fit. A larger next request needs evidence for its actual allocation and runtime conditions.

Preserve the time basis of supplied evidence. The largest tensor value from one instant and the largest pool value from another are neither disjoint populations nor necessarily simultaneous observations. Even peaks of genuinely separate populations cannot be summed and presented as an observed simultaneous peak without evidence that they coincided.

Reconcile two reports claiming that the same model fits

Suppose one fictional evaluation summary quotes AM-214's initial 12 GiB tensor value, while another quotes its 24 GiB complete usage. Both summaries say the model fits. Those values can describe the same stipulated state. The smaller number does not itself demonstrate a more memory-efficient implementation; it omits retained pool storage and outside allocations counted by the broader figure.

Ask each report's owner to identify the device or assigned allocation, PyTorch version, allocator backend, exact metrics, units and coverage. Establish whether the numbers concern the same model and workload at loading, active execution or a later quiet stage. Keep an instantaneous value distinct from a peak over a stated interval. This is the information needed to decide whether the reports disagree or simply answer different questions.

For the buyer, the useful conclusion is a reconciled account tied to the proposed allocation and relevant operating stages. A tensor-only figure can be useful evidence, but cannot stand in for the complete requirement without explaining what else coexists. Neither the pool total nor its gap above tensor occupancy guarantees the next request will succeed.

Use the related quantized-memory guide for the workload populations that must be considered, and the training-state guide for replicas and shards. Those questions remain separate from the allocator lifetime and overlap explained here.

References: [P1][P2][P3]

Connect a selected system to a separate physical installation need

A corrected memory report helps the platform and application owners evaluate the selected system. It does not itself create an equipment-installation requirement. Model operation, allocator tuning, training and software acceptance remain with the appointed software providers.

If the selected equipment also needs receiving, rack integration, onsite installation or IT cabling, use Leviathan's Complete Physical Deployment service and project inquiry below to discuss that physical package. Share a nonconfidential equipment summary, site, required physical stages and receiving team. The agreed engagement defines the work, resources and handoff records.

References: [L1]