All-Reduce, All-Gather or Reduce-Scatter: Different Results on the Same GPUs

Define the result before the hardware

Two proposals can name the same GPUs and still promise different work. The first question is what the next computation needs: a combined value at each position, every participant's separate contribution, or only one share of a combined result.

NVIDIA's NCCL 2.32.3 documentation, checked October 20, 2026, defines all-reduce as a reduction across matching positions, with the full result at every rank. All-gather concatenates contributions in rank order. Reduce-scatter reduces matching positions and distributes equal-sized blocks of that result among ranks. Sum is the reduction used throughout the example below; the word reduction alone does not specify sum.

A rank is a participant's index in the collective group. For this illustration, stipulate two ranks, one per GPU. These logical labels need an explicit mapping to devices. The example explains mathematical outputs and ownership; it does not select a communication algorithm or prescribe software execution.

References: [N1]

Follow the same two inputs through three collectives

In this hypothetical integer example, rank 0 contributes [1, 2, 3, 4], and rank 1 contributes [10, 20, 30, 40]. Read each array from left to right as four positions. No GPU run or measured result is represented.

Sum all-reduce adds corresponding positions: 1 + 10 = 11, 2 + 20 = 22, 3 + 30 = 33, and 4 + 40 = 44. Both rank 0 and rank 1 receive [11, 22, 33, 44]. Each participant now has the complete four-position reduction. It does not receive the two original arrays separately.

All-gather of those original inputs gives both ranks [1, 2, 3, 4, 10, 20, 30, 40]. The first four elements came from rank 0 and the next four from rank 1. None of the pairs has been added. A consumer can identify the separate contributions, but it has not been handed the sum array.

Sum reduce-scatter starts with those same four-element inputs. Rank 0 receives [11, 22], and rank 1 receives [33, 44]. Each returned value already includes both ranks' contribution at its position. The split concerns ownership of the reduced output, not a choice to ignore one participant's input. Neither rank receives all four reduced values in its own output.

References: [N1][N2]

Gathering reduced shards uses different inputs

Now consider all-gather after that reduce-scatter. Its inputs are rank 0's [11, 22] and rank 1's [33, 44]. Gathering these two-element shards gives [11, 22, 33, 44] at both ranks. It reconstructs the same mathematical sum result as the example's all-reduce.

This second all-gather does not consume the original [1, 2, 3, 4] and [10, 20, 30, 40]. Gathering those four-element inputs produced eight elements. Gathering the reduced two-element shards produces four. Identifying the input to the gathering step resolves the apparent contradiction.

NVIDIA documents the logical equivalence of reduce-scatter followed by all-gather to all-reduce. Here that relationship explains how distributed ownership can become a complete result at every participant. It establishes neither equal elapsed time for two proposals nor the internal implementation of a particular all-reduce.

References: [N1]

Count the returned data before reading a message-size label

Stipulate that every element in this example occupies four bytes. The all-reduce output contains four elements per rank: 4 × 4 = 16 B. All-gather of the original inputs returns eight elements per rank: 8 × 4 = 32 B. Reduce-scatter returns two elements per rank: 2 × 4 = 8 B. All-gather of the reduced shards returns four elements per rank, or 16 B.

These are logical output payloads per rank. They count neither bytes transferred on links, all input and temporary buffers, elapsed time nor peak GPU memory. In particular, the 8 B shard does not show that the operation communicated half as many bytes as all-reduce or used half its peak memory. The arithmetic describes which result is retained locally.

The NCCL reference also uses different count conventions. All-reduce's count describes each rank's input and output length. All-gather's sendcount describes its contribution; the output length is the number of ranks times that count. Reduce-scatter's recvcount describes its returned shard; its input length is the number of ranks times that count.

For the three operations on the original arrays, those counts are respectively 4, 4 and 2. The later all-gather of shards has sendcount 2. A bare label such as four elements or sixteen bytes therefore needs its operation and input/output meaning before it can support a comparison.

References: [N2]

Rank order determines layout and ownership

All-gather's ordering and reduce-scatter's block assignment follow rank indices. A rank label does not by itself identify a rack position, physical GPU number or network port. Keep the rank-to-device mapping attached to any explanation of where the data ends up.

Suppose device A initially hosts rank 0 with [1, 2, 3, 4], while device B hosts rank 1 with [10, 20, 30, 40]. In a second hypothetical mapping, A becomes rank 1 and B becomes rank 0, with each device retaining its original contribution. All-gather now returns [10, 20, 30, 40, 1, 2, 3, 4]. The values are unchanged, but their rank-ordered placement has changed.

For sum reduce-scatter under that second mapping, B receives [11, 22] because it is rank 0; A receives [33, 44] because it is rank 1. The combined sum is still [11, 22, 33, 44]. A downstream consumer that assumes A always owns the first shard would misidentify the data. This is why an array layout needs the rank mapping even when the installed devices are unchanged.

References: [N1][N2]

Compare the result the consumer actually needs

Consider a proposed computation whose next stage requires the complete four-position sum on both ranks. The example's sum all-reduce supplies that result. Reduce-scatter alone supplies only a shard to each participant. All-gather of the original inputs supplies both contributions without adding them. Those are three different completion states for the same downstream requirement.

A proposal using reduce-scatter followed by all-gather can describe the required full result, but the comparison must include both stages. If the next computation instead needs only the assigned reduced shard, retaining that distributed output may be the intended result. If it needs each original contribution separately, the gathered eight-element array answers that different requirement. A larger output is not automatically more useful, and a smaller output is not evidence of faster equivalent work.

Before comparing performance claims, settle the participating rank group, what each input position represents, the reduction operation where applicable, and which values each downstream consumer needs locally. Then read the output count in that context. The linked NCCL bandwidth guide explains algbw and busbw; the acceptance-packet guide addresses completed-run evidence. For training-state placement and runtime memory, use the separate DDP/FSDP comparison.