Sparse GPU FLOPS Need the Right Zeros in the Right Places

Read the sparse qualifier before using the number

NVIDIA's H100 specification page, checked October 20, 2026, lists 1,979 teraFLOPS for FP16 Tensor Core arithmetic in the H100 SXM column. That row carries the footnote “With sparsity.” It is a GPU specification for that variant and arithmetic mode, not a dense FP16 figure or a measured model-serving result.

Retain the qualifier wherever the figure travels: a comparison slide, capacity proposal or purchase discussion. Removing it silently changes the workload conditions attached to the number. Nor does a Tensor Core rating establish the rate of every operation in an application.

The existing GPU benchmark guide explains how to compare complete systems and their results. The question here comes earlier: what would make a sparse arithmetic capability relevant to the model implementation being offered?

References: [N3]

The mechanism needs both values and their positions

NVIDIA's explanation of Ampere Sparse Tensor Cores describes a compressed operand holding nonzero values and metadata identifying their original positions. The hardware uses those positions to select corresponding values from the other operand and skip zero multiplications.

The position information matters. Two numbers without locations would not identify the entries they multiply. Likewise, zeros in an ordinary dense representation do not establish use of a compressed sparse path.

This is why supported hardware can cover more of the equivalent dense calculation while skipping selected zero work. The cited July 2021 explanation concerns A100/Ampere; current runtime requirements are considered separately below.

References: [N2]

The operation determines which four entries belong together

TensorRT 11.4.0's current sparsity documentation specifies at least two zeros per four-entry group. For Convolution weights [K,C,R,S], C must be divisible by four; groups run along input channels C at each fixed output channel and kernel position. For MatrixMultiply with a Constant-produced input, groups run along reduction dimension K. Its described path requires FP16 or INT8 execution.

The two uses of K describe different dimensions. A check must follow the operation's rule, rather than flattening a tensor and counting arbitrary neighboring entries. Tensor layout, operation and numerical format therefore belong beside any eligibility finding.

References: [N1]

Two half-zero vectors produce different pattern answers

Consider two original toy vectors, with consecutive non-overlapping groups of four stipulated along a reduction axis. A = [1,0,2,0,3,0,4,0]. B = [1,2,3,0,4,0,0,0]. Each contains four zeros among eight entries: both are globally 50% zero.

A's first group, [1,0,2,0], has two zeros; its second, [3,0,4,0], also has two. Both groups meet the stipulated test. B's first group, [1,2,3,0], has only one zero. Its second group has three, but those extra zeros cannot repair the first group's failure. The rule applies to each group.

A total-zero report would give the same percentage for A and B and miss the decisive difference. Likewise, a whole-model percentage can conceal a failing group inside an otherwise sparse population. A's result establishes only this toy pattern. These vectors are neither actual model weights nor a recommendation to change weights; they establish no supported deployment or speed.

Ask which sparse implementations were actually selected

TensorRT separately reports pattern-qualified layers and layers selected for sparse tactics, meaning implementation choices. Its documentation explicitly allows dense tactics to win for some problem sizes. An eligible-layer count therefore cannot stand in for the selected implementation of the offered engine.

Ask the software provider for three connected findings: a pattern report tied to the exact model representation; evidence from the corresponding engine/build record connecting the selected sparse implementation to the deployed artifact, GPU target, runtime release, numerical mode and required input shapes; and application results against the agreed quality and performance criteria. A report at one step cannot supply the missing result at the next. Until that evidence is connected, leave assumed sparse uplift out of application sizing.

References: [N1]

Sparsity and lower precision change different properties

Quantization maps numerical values into a more limited set of representable levels; structured sparsity constrains zero locations. NVIDIA's May 2023 explanation discusses them as separate techniques that can be combined. A low-bit label does not establish the required sparse pattern, and a sparse label does not establish lower-bit storage.

Compression also concerns a particular operand representation. Position metadata remains, and the complete application has other memory populations. TensorRT's memory documentation describes intermediate activations, persistent execution-context memory and temporary scratch storage in addition to weights. A reduced sparse operand cannot establish a halving of their combined peak.

Use the related quantized-model memory guide for the full capacity question. Use the Roofline guide to distinguish arithmetic capacity from traffic at a named memory boundary.

References: [N4][N5]

A twofold arithmetic opportunity is not a twofold application result

An original timing illustration shows the missing step. Stipulate a serial application taking 100 milliseconds: 60 milliseconds in one affected region and 40 elsewhere. Now suppose the affected region alone becomes exactly twice as fast, with no added overhead or change to the rest. Its time becomes 30 milliseconds; total time becomes 70 milliseconds. The resulting speed ratio is 100 / 70, about 1.43, despite the stipulated twofold improvement in that region.

Those are invented timings, not a prediction from an OEM peak or a test of any model. If no sparse implementation is selected, the example has supplied no reason for the region to improve at all. Even when selected, its actual speed must be established separately from the hardware arithmetic rating.

Quality needs its own comparison too. NVIDIA warns that forcing weights to zero can alter accuracy. The software owner must evaluate the supplied model representation against the application's agreed quality criteria. A valid zero pattern does not show that changed weights preserve the required answers.

Request evidence for the exact model artifact, representative inputs, required load and response-time conditions on the offered system. Keep the reference representation and numerical mode visible. If both precision and sparsity change, a measured difference concerns that combined configuration unless the evaluation separates their effects.

References: [N2]

Carry the selected equipment into the installation scope

Accept the sparse-performance case only when the offered representation, selected execution and application results are connected to the required use and meet the agreed criteria. Missing evidence remains an open software-selection question.

Once hardware is selected, Leviathan's rack-and-stack service is the relevant next step for agreed equipment mounting, physical IT cabling, labels and installation records. Use the related service and project-inquiry links with the equipment list, released rack and cable plans, site and target window. Model changes, runtime selection and application evaluation remain with the appointed software team.

References: [L1]