An FP32 Tensor Can Be Used with TF32 Matrix Arithmetic

Follow the value from FP32 storage through TF32 arithmetic

A matrix multiplication forms output values by combining products of input elements. The stored input, the values supplied to multiplication, the running accumulation and the stored output each need their own description.

NVIDIA's January 27, 2021 Ampere/A100 explanation describes FP32 inputs rounded to TF32 immediately before multiplication. Products of those rounded operands are computed without further precision loss, then accumulated into an FP32 output. Lossless multiplication at that stage does not mean the original FP32 inputs survived rounding.

Its TF32 description uses one sign bit, eight exponent bits and ten fraction bits. For normal values, an implicit leading binary one adds another significant bit. The exponent width and the precision of the fraction answer different questions: retaining the exponent width does not retain every fine distinction between FP32 values.

In that historical mode, tensor storage stays FP32. Its operation scope covers matrix multiplication and convolution, rather than every operation using an FP32 tensor.

References: [TF1][F1]

Read PyTorch's precision policy separately from tensor dtype

PyTorch 2.14 documents torch.set_float32_matmul_precision as controlling internal float32 matrix-multiplication precision. It leaves the output dtype unchanged and does not control convolution precision.

The precision-choice description identifies highest with internal float32 computation. High permits TF32 or an algorithm representing each input using two bfloat16 values when suitable fast algorithms are available; otherwise it falls back to highest. Medium describes internal bfloat16 computation when available, with fallback to high.

Consequently, a recorded high policy alone does not universally identify TF32 execution. A float32 output alone does not identify full-significand float32 internal arithmetic. For a particular comparison, retain the software versions, relevant policy and evidence identifying the implementation actually used.

An available hardware capability, a permitted software choice and an observed execution path are three different pieces of evidence.

References: [TF2]

TF-302: the input remains 4 MiB

Constructed example TF-302 stipulates one unchanged FP32 input matrix containing 1,048,576 elements. Each element occupies four bytes. The raw value payload is 1,048,576 x 4 = 4,194,304 B. With 1 MiB defined as 1,048,576 bytes, that is exactly 4 MiB.

Now change only the permitted internal matrix-multiplication precision. The stipulated input array still contains the same FP32 elements, so its raw payload remains 4 MiB. Adding the historical TF32 sign, exponent and fraction fields to obtain nineteen numerical bits provides no basis for calculating a nineteen-bit packed allocation.

TF-302 accounts only for that stipulated input array. Outputs, temporary allocations, other tensors and allocator overhead require their own accounting. It supplies neither a peak-memory measurement nor evidence that two implementations have identical total memory use.

References: [TF1]

A wider output cannot restore a rounded input

Consider a separate constructed numerical example. FP32 can distinguish 1 from 1 + 2^-12, which is 1.000244140625. Assume round-to-nearest conversion to a representation with ten fraction bits. Between 1 and 2, adjacent representable values are spaced by 2^-10, or 0.0009765625. The second input lies only one quarter of a step above 1, so both inputs round to 1.

Multiplying either rounded operand by 1 now gives the same product. Accumulating that product into FP32 and storing an FP32 result does not reveal whether the original input was 1 or 1.000244140625. The original distinction disappeared before multiplication. More output bits cannot reconstruct information that no longer reaches the arithmetic.

The numbers and rounding assumption here are an educational construction, not a device test or a universal error bound. PyTorch 2.14's NVIDIA TF32 numerical discussion gives the related warning that multiplication by an identity matrix can differ from the original input. It does not establish the resulting error or task quality for a buyer's workload.

References: [F1][TF3]

Keep the later CUDA TF32 type in its own scope

The historical description of TF32 as a math mode does not establish that no later API has a TF32 type. NVIDIA's CUDA Math API archive 13.4.2 documents __nv_tf32, an explicit datatype restricted there to tile code.

That reference specifies 32-bit storage and four-byte alignment. It describes at least nineteen value bits, with precision dependent on the hardware target and the arrangement of value bits within storage unspecified.

This later datatype is a separate API fact. It neither establishes nineteen-bit packed storage nor shows that PyTorch's float32 matmul policy changes a tensor into __nv_tf32. Identify the software interface before interpreting a type or mode label.

References: [TF4]

Decide what two float32 offers have actually established

Imagine two software offers for the same requested matrix workload. Both identify float32 input and output tensors on the same GPU. Offer A presents a timing result and those dtype labels. Offer B also identifies its internal arithmetic path, software versions, relevant operation shapes and numerical comparison against the agreed reference. These are invented offer descriptions, with no reported test outcomes.

A leaves the internal-arithmetic comparison unresolved. B makes that question inspectable, but its fuller description does not itself demonstrate acceptable results. The supporting evidence must correspond to the configuration being offered, including the path that actually executed. A supported TF32 mode or a framework policy is insufficient evidence of that path.

Agree what numerical success means before using the timing result to select an offer. The application owner may require an output comparison against a specified reference, a task-quality evaluation, or both. Keep those requirements attached to the results. Matching dtype labels cannot establish bitwise equality, acceptable error, equal convergence or equal application quality.

Different arithmetic policies can be valid candidates when the task permits them. Compare their completed work under the same acceptance requirements, then assess timing. A faster result that lacks the required numerical evidence leaves the purchase decision open. Keep measured peak memory as another quantity; TF-302 cannot substitute for it.

Use the related GPU benchmark guide for whole-workload comparisons. The FP8 guide explains mixed-precision training roles, while the quantized-weight guide explains inference memory populations. This decision adds the missing question within a float32 label: which matrix arithmetic produced the result?

References: [TF2][TF3]

Carry selected equipment into the physical installation package

If the selected equipment needs installation, Leviathan's related Rack-and-Stack / Colocation service and project inquiry provide a route to discuss mounting, agreed IT cabling, labels and physical installation records.

Share a nonconfidential equipment list and quantities, site, rack layout, rail details, cable endpoints and target window. The physical package needs the selected hardware configuration, rather than a float32 benchmark label. Numerical-policy evaluation and software acceptance stay with the appointed software team; software commissioning is not a standard rack-installation inclusion.

References: [L1]