The Same FP32 Sum Can Change When Its Additions Are Grouped Differently

An FP32 reduction needs an arithmetic description

FP32 identifies a floating-point representation. A sum reduction also needs a description of how its intermediate values are formed. Mathematically, parentheses can move without changing a sum. In finite precision, moving them changes which exact intermediate values must be rounded.

NVIDIA's Floating Point and IEEE 754 documentation distinguishes an individual operation with a specified format and rounding mode from a sequence organized in different ways. Each addition can obey its rule while the completed expressions disagree. This failure of associativity is enough to explain the possibility of a difference before considering random state or a different device.

References: [NVFP]

SUM-24: keep the inputs and rounding rule fixed

Constructed example SUM-24 uses a = 16,777,216, which is 2^24; b = 1; and c = -16,777,216. All three inputs are exactly representable in IEEE 754 binary32. Their exact real-arithmetic sum is 1. There is no decimal parsing approximation in these input values.

Stipulate that every addition is separately rounded to binary32 using round-to-nearest, ties-to-even. There is no mixed precision, extended intermediate accumulation or fused multiply-add. Overflow and subnormal behavior do not affect these values. Compare the two explicit groupings (a + b) + c and a + (b + c) under those same assumptions.

References: [NVFP][OR32]

The left grouping loses the unit before cancellation

For (a + b) + c, the exact first sum is 16,777,217. Binary32 has 24 significant binary digits for normal values: the leading one plus 23 fraction bits. From 2^24 up to just below 2^25, adjacent representable values are two apart. The neighbors around this first sum are therefore 16,777,216 and 16,777,218.

The exact 16,777,217 is halfway between them. Ties-to-even chooses the candidate whose least significant retained significand bit is zero. That is 16,777,216; the upper candidate has a one in that bit. Both candidates are even decimal integers, so decimal integer parity alone would not explain this choice.

The rounded intermediate is now 16,777,216. Adding c, which is -16,777,216, gives exactly 0. The contribution from b disappeared at the first rounding boundary. The later cancellation exposes that loss; it does not restore the missing unit.

References: [NVFP][OR32]

The right grouping preserves the unit

For a + (b + c), the exact inner sum is 1 - 16,777,216 = -16,777,215. Integers with magnitudes from 2^23 up to just below 2^24 are spaced one apart in binary32. This intermediate is exactly representable, so rounding leaves it unchanged.

The final addition is 16,777,216 + (-16,777,215) = 1, also exactly representable. The right grouping therefore returns 1.

The input sequence, input format, number of additions, per-addition format, rounding rule and output format are identical in both cases. Only the grouping changed. Relative to this example's exact real-arithmetic reference of 1, the left result has absolute error 1 and the right result has absolute error 0. Correctly rounded individual additions do not ensure that the completed reduction equals the exact mathematical sum.

References: [NVFP][OR32]

Repeatable groupings can still disagree

Each specified SUM-24 grouping can return its own result on every repetition. There is no random draw to reseed. Repeatability within an implementation and agreement between implementations are separate questions. The related seed and deterministic-algorithm guide develops the repeatability decision.

NVIDIA illustrates serial and tree organizations of reductions and explains how concurrent participation can rearrange additions. A tree can also execute on one thread. More parallelism is unnecessary to demonstrate the mechanism, and a changed device count alone does not establish that a particular library changed its reduction tree.

PyTorch 2.14's Numerical accuracy note, updated June 1, 2026, provides no general bitwise-equality guarantee for mathematically identical floating computations, including across platforms after randomness is controlled. Its batched matrix example also illustrates how mathematically equivalent requests can take different implementation paths.

SUM-24 specifies hand-parenthesized arithmetic. It does not predict the output or execution order of torch.sum, NCCL or a CUDA kernel. Interpreting a real difference requires evidence of the arithmetic that actually ran.

References: [NVFP][PT214]

Separate accumulator precision from final storage

Suppose the left grouping is completed as stipulated, and its result is then stored in a wider floating-point format. The value being converted is already 0. A wider representation can store that zero exactly, but it receives no record of the unit lost earlier.

Keeping intermediate additions in higher precision would change the arithmetic premise. It is a different candidate to describe and assess against the application's requirement, rather than an automatic guarantee of correctness for every reduction. Ask where values are rounded, including conversions before the final output.

The related TF32 guide covers a different source of information loss: operand precision within matrix arithmetic. SUM-24 needs no reduced-precision operands to produce its disagreement.

Define equivalent results before comparing timing

Two offers saying same FP32 inputs and FP32 result may describe different valid sequences of rounded additions. To compare them, ask the producing software team to identify the reduction operation and input shape, actual grouping or permitted grouping variability, accumulator representation, rounding boundaries, software versions and participating resources represented by its evidence. Keep the corresponding output evidence attached. Those details make the numerical claim interpretable before a timing comparison ranks it.

For exact replay, define the output boundary and representation whose bits must match, along with the execution and environment scope over which replay is required. Both SUM-24 groupings are repeatable, but their unequal outputs fail a requirement that the two implementations return identical bits for these inputs. A seed or dtype label supplies no missing equality evidence.

For agreement with a reference, identify the reference's purpose and the supported comparison rule. For an application outcome, identify the actual outcome requirement. In SUM-24, 1 matches the exact sum and 0 does not. That establishes this example's numerical comparison; it does not establish model quality, safe application behavior or which result a separate workload should accept. No universal tolerance follows from the example.

A possible rounding explanation is a reason to investigate the arithmetic path, not permission to accept unexplained disagreement or declare a GPU defective. If a submitted result includes an NCCL generated-result check, use the related NCCL acceptance guide to understand what that check covered. Here the decision is whether the compared reductions establish the intended numerical relationship. Timing becomes useful when it refers to work that meets that relationship.

Discuss installation when selected equipment needs physical work

If the selected equipment also needs physical installation, Leviathan's rack-and-stack service and project inquiry provide a route to discuss agreed mounting, physical IT cabling, labeling and handoff records. Share a nonconfidential equipment, rack and site outline to discuss scope and availability. Numerical evaluation and application acceptance remain with the appointed software team; software commissioning is not a standard rack-installation inclusion.

References: [L1]