---
title: "MoE Token Dispatch: Why an All-Reduce Result Does Not Describe the Exchange"
description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20."
canonical: "https://www.leviathansystems.co/articles/moe-all-to-all-token-dispatch-versus-allreduce-evidence"
last-updated: "2026-10-20"
---

# MoE Token Dispatch: Why an All-Reduce Result Does Not Describe the Exchange

Leviathan Systems — Technical content
Published 2026-10-20; updated 2026-10-20.

MoE token dispatch sends individually identifiable token representations toward selected experts; all-reduce combines corresponding values and returns the reduced result to every participant. An all-reduce score cannot establish how routed expert traffic behaves. Evaluate evidence for the actual routing distribution, participating groups and complete dispatch/return boundary, including the work concentrated at each expert.

## Identify the result the expert needs

A cluster proposal may include a strong all-reduce result and say that every GPU can reach every other GPU. Neither statement identifies which token representations reach which experts, how unevenly work is assigned, or when the required expert outputs become available.

NVIDIA's NCCL 2.32.3 Collective Operations page, checked October 20, 2026, defines all-reduce as reduction across matching input positions, with the result at every rank. Sum, for example, combines values at the same array position. The related collective-output article explains that operation in detail.

For routed expert work, each selected expert needs the representations assigned to it. A token here means a model position whose intermediate vector is being processed, rather than a text string sent over the fabric. The January 8, 2024 Mixtral paper describes expert parallelism as sending tokens to the GPU holding an expert and returning the expert's output to the token's original location.

Adding unrelated token vectors position by position would lose the separate inputs needed for that routing decision. Returning the same reduced array everywhere also answers a different ownership question. The buyer needs evidence of the exchange the application actually uses.

References: [N1](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html), [P1](https://arxiv.org/html/2401.04088v1)

## Follow one documented dispatch and return path

NVIDIA's Megatron Core 0.17.0 API documents MoEAlltoAllTokenDispatcher as one implementation. Preprocessing derives per-expert token counts and communication splits from the routing map. Dispatch preparation rearranges tokens and probabilities; token_dispatch exchanges them across expert-parallel ranks. After expert computation, token_combine returns results to their originating ranks, and postprocessing restores order and shape.

Its documented workflow also includes tensor-parallel all-gather and reduce-scatter stages and sorting for multiple local experts. The module lists AllGather and Flex dispatchers too. A training or inference offer needs its actual execution path and versions identified.

NCCL's AlltoAll definition sends destination-specific chunks and illustrates equal N-value chunks between ranks. That definition does not specify how the unequal routing populations below would be implemented. The example supplies no native NCCL call, padding scheme or unequal-split implementation.

References: [M1](https://docs.nvidia.com/megatron-core/developer-guide/0.17.0/apidocs/core/core.transformer.moe.token_dispatcher.html), [N1](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html)

## Eight assignments can leave one destination with five

Consider a fictional group of four ranks, R0 through R3, with one GPU and one expert per rank. R0 originates tokens A and B; R1 originates C and D; R2 originates E and F; R3 originates G and H. Each token has exactly one expert assignment. All eight assignments are retained, with no token dropping.

Stipulate this routing map: A, B, C, E and G go to R0's expert; D goes to R1's expert; F goes to R2's expert; H goes to R3's expert. Destination assignment counts are therefore 5, 1, 1 and 1.

Only C, E and G cross rank boundaries on the outward dispatch: C starts at R1, E at R2 and G at R3, and each goes to R0. A and B are already at R0. D, F and H remain at their originating ranks. This gives three remote vectors and five local assignments.

All eight assignments still require expert processing. Counting only the remote vectors would hide the two local assignments contributing to R0's five-token population.

## An even destination count can require more remote vectors

Keep the origins and one-assignment rule, but stipulate a second routing map. B and C go to R0; A and D go to R1; F and G go to R2; E and H go to R3. The destination counts are now 2, 2, 2 and 2.

The remote vectors are A from R0 to R1, C from R1 to R0, E from R2 to R3 and G from R3 to R2. B, D, F and H remain local. This map has four remote vectors and four local assignments.

The total remains eight assignments, but both the destination distribution and the remote population have changed. An average of two assignments per destination describes both maps while concealing the first map's five-assignment concentration at R0.

These invented maps compare distributions. They do not establish interchangeable outputs for a trained model or recommend changing its expert selections. Which expert processes a token is part of the computation being evaluated.

## Six KiB versus eight KiB leaves completion time unresolved

Now stipulate that every logical token vector occupies exactly 2 KiB, where 1 KiB is 1,024 bytes. The first map's outward remote payload population is 3 x 2 = 6 KiB, or 6,144 bytes. The second map's is 4 x 2 = 8 KiB, or 8,192 bytes.

These totals count each outward remote vector once. They exclude local rearrangement, padding, metadata, routing probabilities, protocol overhead, physical-link traversal and the return/combine traffic. No execution durations or physical paths have been supplied.

The first map has the smaller counted remote payload, while R0 has five assignments to process instead of two. If completion requires every assigned expert output, the work at that destination matters alongside communication. The second map spreads assignment counts evenly, but counts alone do not establish equal computation times either.

Consequently, neither 6 KiB nor 8 KiB identifies the faster completed exchange. The first map's smaller total cannot establish a speed advantage; the second map's even counts cannot establish one either. A claim about complete layer time needs evidence covering the actual work and dependencies.

The example also shows why total tokens, average assignments per expert and total outward payload are insufficient summaries on their own. Each removes a different part of the distribution.

## Read all-to-all separately from the physical connection plan

An all-to-all communication description names relationships among logical participants. It does not require a dedicated physical cable between every GPU pair. A claim that all devices are reachable still leaves the carrying paths, shared resources and simultaneous traffic unresolved.

For the two routing maps, the same four ranks can have the same reachability while presenting different source-to-destination demands. In the first, all three remote vectors converge on R0. In the second, the four remote vectors are exchanged within two pairs. Reachability alone cannot distinguish these workloads.

The platform owner needs to connect the participating rank groups to actual devices and their supported fabric. The physical installation package separately identifies equipment, ports, assemblies and routes. Neither a logical rank name nor a payload total produces that package.

Use the tensor/pipeline article for operation and stage partitions, and the active-parameter article for model-weight inventory. Those decisions can coexist with expert routing; neither describes this exchange's destination distribution.

## Ask what the submitted evidence actually represents

Start with the claimed result. Does the report time only the communication step, dispatch preparation plus communication, or the full path through expert processing and returned outputs? A timing boundary ending before outputs return cannot establish the time to that later completion state. Ask the evidence owner to identify the included work and the endpoints of the reported interval.

Next, identify the routing population. Request the represented token batch, expert assignments, destination counts and local versus remote assignments, together with the actual participating groups and device mapping. If only evenly distributed traffic is represented, behavior under a concentrated distribution remains unresolved. A single favorable batch does not describe every routing pattern the intended workload can produce.

Keep the model revision, numerical representation, dispatcher/backend versions and hardware arrangement attached to that explanation. Have the producer identify whether padding, capacity limits or dropped assignments change the work represented. The example retained all eight assignments; an actual result covering fewer retained assignments would need that difference exposed.

Finally, connect reported performance to correctly associated returned outputs for the represented work. Dispatch reaching an expert and an output reaching its intended consumer are different completion events. This is a request to explain supplied evidence, rather than an execution or configuration recipe.

An all-reduce report remains evidence for its recorded operation. Use the NCCL bandwidth guide to interpret its metrics and the all-reduce packet guide for its provenance. Neither supplies the missing routed-work observations. The useful conclusion is precise: which dispatch/return claim the evidence supports, and which intended routing conditions remain unrepresented.

## Bring selected equipment and connections to a physical inquiry

Once the platform team has selected the equipment and supported connections, Leviathan's Complete Physical Deployment service can discuss agreed receiving, rack integration, onsite installation, IT cabling and physical handoff records.

Use the related service and project-inquiry links with a nonconfidential equipment summary, proposed locations, connection requirements and receiving team. These inputs make the physical need discussable without turning a software benchmark into an installation scope.

Expert-routing implementation, performance evaluation and workload acceptance remain with the appointed platform and application owners. The physical engagement defines its own work, resources and handoff.

References: [L1](https://www.leviathansystems.co/services/turnkey-gpu-deployment)

## Primary sources

- [N1 — NVIDIA NCCL 2.32.3 — Collective Operations](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html). Checked 2026-10-20. AllReduce semantics and equal-chunk AlltoAll definition; useful for distinguishing reduced values from destination-specific exchange.
- [M1 — NVIDIA Megatron Core 0.17.0 — MoE token dispatcher API](https://docs.nvidia.com/megatron-core/developer-guide/0.17.0/apidocs/core/core.transformer.moe.token_dispatcher.html). Checked 2026-10-20. Named dispatcher workflow, counts, splits and return path, with AllGather and Flex alternatives. Scope is implementation-specific.
- [P1 — Jiang et al. — Mixtral of Experts, arXiv 2401.04088v1](https://arxiv.org/html/2401.04088v1). Checked 2026-10-20. January 8, 2024, Section 2.1: expert-parallel token movement and returned outputs. The four-rank examples are original stipulated mappings.
- [L1 — Leviathan Systems — Complete Physical Deployment](https://www.leviathansystems.co/services/turnkey-gpu-deployment). Checked 2026-10-20. Engagement-defined physical receiving, integration, installation, cabling and handoff scope, with separately assigned software responsibilities.

## Related Leviathan resources

- [MoE active parameters and resident model weights](/articles/moe-active-parameters-versus-resident-model-weights)
- [All-reduce, all-gather and reduce-scatter outputs](/articles/gpu-collectives-allreduce-allgather-reduce-scatter)
- [Read NCCL algbw and busbw in their operation scope](/articles/nccl-bandwidth-validation)
- [Review the provenance of an all-reduce evidence packet](/articles/nccl-all-reduce-validation-acceptance)
- [Tensor and pipeline partitions within model execution](/articles/tensor-versus-pipeline-parallel-model-execution)
- [Complete Physical Deployment for selected equipment](/services/turnkey-gpu-deployment)
- [Discuss the physical deployment and handoff scope](/contact?service=turnkey-gpu-deployment&from=%2Farticles%2Fmoe-all-to-all-token-dispatch-versus-allreduce-evidence)

## Scope physical deployment for the selected system

Share a nonconfidential equipment, site, connection and receiving-team outline to discuss the physical work, handoff requirements and availability.

[Discuss the scope](/contact?service=turnkey-gpu-deployment&from=/articles/moe-all-to-all-token-dispatch-versus-allreduce-evidence)