GPU Platforms
PyTorch Tensor Views and Strides: Shared Storage and Copy Costs
A PyTorch view is a tensor that interprets an existing tensor's data through its own shape, strides and storage offset. Creating that view does not copy the underlying values. Different tensor objects and shapes can therefore share one storage, while a later layout conversion can create another. To assess a memory or data-movement claim, follow distinct live storages and the operations that consume them, rather than adding every tensor's logical size.
A tensor describes both data and a way to address it
This explanation uses ordinary dense strided tensors in PyTorch 2.14. A tensor's shape gives its length along each dimension. Its dtype defines the element representation. The underlying storage contains the data bytes; strides and a storage offset determine which elements a logical index addresses.
Object identity asks whether two references name the same tensor object. Storage identity asks whether tensors refer to the same underlying data container. A view can be a different tensor object with different metadata while sharing its base's storage. Comparing shapes or counting objects cannot establish the number of independent data allocations.
That sharing also affects meaning. If a permitted edit changes a value through one view, the change is visible through other tensors addressing that same storage location. A view does not supply an independent snapshot. If an application needs an independent snapshot of the same elements, confirm that the copy does not alias the original data.
TV-209: transpose the addressing, retain the 48 bytes
In worked example TV-209, assume a contiguous 4 × 3 FP32 base tensor at storage offset zero, with exactly twelve stored elements containing the values 0 through 11 in row order. FP32 uses 32 bits, or four bytes, per element, so this storage contains 12 × 4 = 48 raw bytes. These are invented values and layout assumptions, not an executed workload or a GPU allocation measurement.
The base has strides (3, 1). Moving one row advances three storage elements; moving one column advances one. Its rows contain 0, 1, 2; then 3, 4, 5; then 6, 7, 8; then 9, 10, 11.
Swapping the two dimensions gives a 3 × 4 transpose view with strides (1, 3) and the same zero offset. Its rows read 0, 3, 6, 9; then 1, 4, 7, 10; then 2, 5, 8, 11. The values stay where they were in storage.
For zero-based indices (i, j), the storage element position is offset + i × stride[0] + j × stride[1]. View index (2, 1) therefore addresses 0 + 2 × 1 + 1 × 3 = element 5. Its first byte is 5 × 4 = byte offset 20 from the storage start, and its value is 5. The corresponding base index is (1, 2). Two shapes now describe the same 48 bytes.
Noncontiguous describes the traversal through that storage
TV-209's transpose is noncontiguous in the ordinary contiguous format. Successive columns in one logical row address storage positions three elements apart. The base storage itself remains a contiguous byte container: the view's noncontiguity concerns how its logical dimensions traverse that container.
A consumer must still access values to compute with this view. Its accepted layouts and implementation determine whether it can use that representation directly or introduces a layout conversion. Contiguity may affect performance, but the metadata alone establishes neither a compulsory copy nor a universal slowdown. Evidence for the consuming operation is needed to turn a view-creation claim into a claim about the complete computation.
A contiguous materialization changes the storage count
Now stipulate that TV-209's noncontiguous transpose is materialized as an independent contiguous 3 × 4 tensor in the ordinary format, while the base and transpose view remain alive. The new tensor has strides (4, 1). Its storage holds the transpose's logical rows in sequence: 0, 3, 6, 9, then 1, 4, 7, 10, then 2, 5, 8, 11.
The new storage contributes another twelve FP32 elements, or 48 raw bytes. The distinct live storages total 48 + 48 = 96 raw bytes. There are three tensor objects, each describing twelve elements, but adding their three logical extents gives 144 bytes and counts the shared base storage twice.
The copy preserves the transpose's indexed values while changing their physical positions. Index (2, 1) now reaches element 2 × 4 + 1 = 9, starting at byte offset 36 in the new storage, and still reads value 5.
PyTorch's contiguous operation is conditional on the requested memory format. If the input already satisfies that format, it returns the input itself. An already-contiguous input therefore does not establish another storage merely because contiguous appears in an operation list.
References: [P6]
A reshape label does not establish a copy contract
For the shape-changing form of view, equal element counts are necessary but insufficient. The requested dimensions must also be compatible with the existing sizes and strides. PyTorch's view reference describes a compatibility condition for combining dimensions: each preceding stride must equal the following stride multiplied by that following dimension's size.
In TV-209, collapsing the transpose's two dimensions into one would require 1 = 3 × 4, which is false. Its flattened logical order cannot be expressed as one constant-stride walk through that original storage. The base's dimensions satisfy 3 = 1 × 3. Both tensors have twelve elements, but their addressing differs.
PyTorch 2.14 documents reshape as returning a view when possible and a copy otherwise. Flatten can return the original object, a view or a copy. Neither name alone proves independent storage or freedom from copying. A report needs the applicable operation behavior and the actual input layout.
A small logical extent can retain a larger storage
Consider a separate view selecting the base's second row in TV-209. It describes three FP32 elements: 3, 4 and 5. Its shape is (3,), stride is (1,), and storage offset is three elements, placing its first value at byte offset 12.
Its logical extent is 3 × 4 = 12 bytes, yet it still shares the original 48-byte storage. Keeping that view alive can retain the larger storage after the base tensor object is no longer referenced elsewhere. The smaller shape does not split off a separately owned twelve-byte allocation or release the other bytes. This row view is already contiguous in the ordinary format, so requesting that same format with contiguous would return the view itself and preserve its sharing relationship.
PyTorch's serialization documentation illustrates the distinction with a small view whose saved storage contains the larger base's values. That documented example concerns serialization; the lifetime point here follows from the view retaining shared storage. For memory interpretation, count the underlying storage once and distinguish its size from the portion addressed by the surviving view.
Ask which storage survives and which operation moves values
Suppose two descriptions of TV-209 each report three twelve-element tensors. One retains only views of the original storage; the other retains the base, transpose view and independent materialization above. Equal object counts and logical element totals leave their unique raw-storage totals unresolved. The all-view case needs 48 raw bytes under the stated assumptions; the materialized case needs 96.
Ask the software owner to identify the transformation, input and output shapes, dtype, strides, offsets and sharing relationship. Establish which storages coexist at the point being reported, including any larger storage kept alive by a small view. For a no-copy claim, establish whether it covers only view creation or also subsequent layout conversions and consumers.
These raw-byte sums omit allocation rounding, tensor metadata, gradients and other buffers. PyTorch's CUDA documentation places tensor occupancy inside a caching allocator with its own managed-memory account. Use the related allocated-versus-reserved guide for that separate accounting layer and the inference-memory guide for the wider workload requirement.
The bandwidth guide addresses bytes transferred across a named boundary. Bytes retained in storage and bytes transferred during execution are different quantities. Neither TV-209's 48-byte sharing nor its 96-byte coexistence establishes GPU fit, measured traffic or elapsed time.
Connect selected equipment to its physical installation scope
If the selected GPU system also needs physical installation, use Leviathan's rack-and-stack service and related project inquiry to discuss agreed equipment mounting, IT cabling, labels and physical installation records. Share a nonconfidential equipment list, rack plan, site window and required handoff. Tensor-layout decisions and application performance acceptance remain with the appointed software team.
References: [L1]
Primary sources
- [P1] PyTorch 2.14 — Tensor ViewsChecked 2026-10-20. Created and updated May 7, 2026. Explains shared data, edits visible through views, transpose noncontiguity and the absence of value movement during view creation.
- [P2] PyTorch 2.14 — torch.StorageChecked 2026-10-20. Updated May 6, 2026. Defines storage and tensor metadata, sharing between shapes and separate gradient storage for ordinary tensors.
- [P3] PyTorch 2.14 — torch.Tensor.strideChecked 2026-10-20. Defines movement along each dimension and shows element-stride examples; TV-209's dimensions and calculations are original.
- [P4] PyTorch 2.14 — torch.Tensor.storage_offsetChecked 2026-10-20. Specifies the offset in storage elements, rather than bytes, and illustrates a slice starting within its storage.
- [P5] PyTorch 2.14 — Tensor AttributesChecked 2026-10-20. Updated June 15, 2026. Defines float32 as 32-bit floating point and explains dense strided layouts; the example uses the ordinary contiguous format.
- [P6] PyTorch 2.14 — torch.Tensor.contiguousChecked 2026-10-20. Defines the requested memory format and return of the original tensor when it already satisfies that format.
- [P7] PyTorch 2.14 — torch.Tensor.viewChecked 2026-10-20. The shape-changing overload requires compatible sizes and strides; the dtype-reinterpreting overload is outside this explanation.
- [P8] PyTorch 2.14 — torch.reshapeChecked 2026-10-20. Documents view-or-copy behavior according to the input's shape and strides.
- [P9] PyTorch 2.14 — torch.flattenChecked 2026-10-20. Documents return of the original object, a view or a copy while preserving logical element order.
- [P10] PyTorch 2.14 — Serialization semantics: preserving viewsChecked 2026-10-20. Shows how a small view can reference storage larger than its logical extent. Its file example illustrates sharing, rather than measuring runtime memory.
- [P11] PyTorch 2.14 — CUDA semantics: memory managementChecked 2026-10-20. Distinguishes tensor occupancy from allocator-managed memory and describes native allocation rounding; no allocator settings are prescribed.
- [L1] Leviathan Systems — Rack & Stack / ColocationChecked 2026-10-20. Defines physical mounting, IT cabling, labels and installation records, with software bring-up and full commissioning outside the standard installation scope.