A Sparse Matrix Can Use More Storage Once Its Indices Are Counted

Identify the supplied layout before interpreting its zero percentage

An ordinary dense 4×4 matrix retains sixteen scalar positions, including positions whose values are zero. A scalar compressed sparse row representation, or CSR, instead uses a values array, a column-index array and an array of row offsets. The offsets identify each row's span in the other two arrays. PyTorch 2.14 names these components values, col_indices and crow_indices.

The examples below concern one two-dimensional matrix with scalar values, no batch dimensions, no blocks and zero-based indexing. Both index arrays have the same stipulated element width. The dense comparison is a contiguous payload with the same shape and value type. These assumptions make the raw byte comparison explicit.

CSR and 2:4 Tensor Core operands are different representations. The 2:4 eligibility and execution question belongs to the related structured-sparsity article; its metadata rules cannot supply a CSR byte count.

References: [P2][N1][P1]

CS-214: recover four rows from three arrays

CS-214 is an invented storage illustration. Its row offsets are [0,1,2,3,4], column indices are [1,0,3,2], and values are [5,7,9,2]. The declared matrix shape is 4×4. Array positions and matrix coordinates both start at zero.

For row 0, the first two offsets are 0 and 1. Read array positions starting at 0 and stopping before 1: that selects value 5 and column index 1. Thus row 0 has 5 in column 1. Row 1 uses positions from 1 up to, but excluding, 2: value 7 belongs in column 0. Row 2 uses the next range, from 2 to before 3, placing 9 in column 3. Row 3 uses positions from 3 to before 4, placing 2 in column 2.

The logical rows are therefore [0,5,0,0], [7,0,0,0], [0,0,0,9] and [0,0,2,0]. Every unspecified position is zero in this ordinary matrix interpretation. The final offset, 4, marks the end of the last row and equals the number of specified entries.

Five offsets are needed for four rows because every row needs a beginning and an end, with adjacent rows sharing a boundary. Equal adjacent offsets would describe an empty row; that row would still require its boundary entry. The offsets compress row membership, while the column indices retain the missing horizontal positions.

References: [P2][N1]

Four entries need 52 bytes; eight need 84

Stipulate FP32 values at four bytes each and 32-bit integer indices at four bytes each. The dense payload is 4 × 4 × 4 = 64 bytes. CS-214's four values occupy 16 bytes, its four column indices occupy another 16 bytes, and its five row offsets occupy 20 bytes. The raw CSR total is 16 + 16 + 20 = 52 bytes.

Twelve of the sixteen logical positions are zero: 75% zeros. Nevertheless, the representation saves only 64 - 52 = 12 raw bytes, or 18.75% of the dense payload. The four values alone are not the representation.

For a second invented 4×4 case, stipulate eight distinct specified entries, all nonzero, and eight unspecified zero positions. Its values occupy 8 × 4 = 32 bytes; column indices occupy another 32 bytes; row offsets still occupy 20 bytes. The total is 32 + 32 + 20 = 84 bytes.

This second matrix is half zero yet needs 20 more raw bytes than its 64-byte dense payload. These are invented component-count examples, rather than measured tensor or allocator usage.

Row count and index width move the storage crossover

For this scalar, nonbatched representation, let rows and columns describe the logical shape, nse count specified entries including any explicitly stored zeros, indexBytes be the width shared by both index arrays, and valueBytes be the width of each scalar value. Counting the three arrays gives rawCSRbytes = (rows + 1) × indexBytes + nse × (indexBytes + valueBytes). The dense payload is rows × columns × valueBytes.

CSR uses fewer raw bytes only when that first expression is strictly smaller than the dense payload. With CS-214's shape and widths, this becomes 20 + 8 × nse < 64. Since nse is an integer, at most five specified entries fit below the dense total: five use 60 bytes and six use 68 bytes. That crossover belongs to this example, not to every sparse matrix.

Changing the shape also changes the row-offset term. Stipulate the same sixteen logical FP32 positions and four specified entries arranged as a 16×1 matrix. Its seventeen 32-bit row offsets need 68 bytes, giving 68 + 16 + 16 = 100 raw CSR bytes against the same 64 dense bytes. This is a separate invented matrix, not a recommendation to reshape data.

Index width matters even without changing the matrix. PyTorch 2.14 documents 32-bit and 64-bit CSR indices. If CS-214 instead has 64-bit indices, its values remain 16 bytes, but column indices require 32 and row offsets require 40: 88 bytes altogether. The original 52-byte result therefore depends on the explicit 32-bit assumption; it is not a default PyTorch allocation claim.

References: [P1][P2][N1]

A specified entry need not be a nonzero logical value

Use nse here to mean stored, specified scalar entries. A value slot containing zero still occupies space and retains its column index. PyTorch's sparse MaskedTensor tutorial explicitly illustrates stored zeros. A report labeled nnz therefore needs its counting definition before it can be read as the number of nonzero logical values.

Duplicate coordinates need a separate format qualification. PyTorch permits repeated coordinates in uncoalesced COO tensors and interprets their contributions by summation. Two stored contributions can consequently describe one logical position, and their sum can be zero. This COO behavior is not an assumption of the CSR examples, whose coordinates are unique.

Ask the provider to identify the actual layout and, for CSR, the declared shape, lengths and dtypes of all three component arrays, together with the definition used for the reported entry count. Stored entries, distinct coordinates and nonzero logical values are different counts. Substituting one for another can understate storage or misstate sparsity.

References: [T1][P1]

Keep the raw array total inside its memory boundary

The formula counts component-array payloads only. It excludes tensor objects and other metadata, allocator overhead or retained capacity, any dense and sparse copies that coexist during conversion, operator workspace and other tensors. It is not a serialized-file-size formula or a complete device-memory requirement.

For example, if the stipulated 64-byte dense payload and 52-byte CSR payload coexist, those two payloads alone total 116 bytes before any other costs. A smaller final representation can therefore coexist with a higher temporary requirement. This is an accounting illustration, not a conversion procedure.

A capacity claim needs evidence for the supplied artifact and relevant operating stages on the proposed allocation. Use the related inference-memory article for the other workload populations and the allocator article for the meaning of allocated and reserved reports. Neither question is answered by subtracting one matrix's raw array counts.

A smaller CSR representation does not establish faster execution

A direct scalar CSR matrix-vector calculation uses each column index to locate the corresponding vector element. In CS-214, the stored column indices identify vector positions 1, 0, 3 and 2. This illustrates why fewer stored values need not mean a simple contiguous stream of operand reads. Irregular accesses and any layout conversion can change the execution cost; the byte formula contains no timing model.

PyTorch 2.14 warns that sparse execution can take longer and that sparse operator coverage does not match dense coverage. Its sparse matrix-multiplication reference supplies concrete restrictions: mixed COO/CSR inputs are unsupported, and its optional reduction mode is implemented for CSR on CPU. A general sparse-support label cannot establish the required operator, value dtype, index type and backend combination.

The buyer can accept a raw storage calculation when the actual arrays, widths and shape support it. A speed or smaller-allocation claim needs separate evidence for the required operations and complete workload on the offered configuration. Keep the model artifact, framework/backend versions and hardware target attached to that evidence. A zero percentage or smaller matrix representation supplies no speed multiplier.

References: [N1][P1][P3]

Connect selected hardware to an actual installation need

If the software evaluation leads to selected hardware that needs physical installation, Leviathan's rack-and-stack service and related project inquiry are the relevant next step for agreed mounting, physical IT cabling, labels and installation records. Share the nonconfidential equipment list, rack and cable plans, site and target window. Sparse-format selection and application evaluation remain with the appointed software team.

References: [L1]