--- title: "Tensor or Pipeline Parallelism: The Same GPU Count Can Split Different Work" description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20." canonical: "https://www.leviathansystems.co/articles/tensor-versus-pipeline-parallel-model-execution" last-updated: "2026-10-20" --- # Tensor or Pipeline Parallelism: The Same GPU Count Can Split Different Work Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20. Tensor parallelism divides compatible operations within a model layer among cooperating participants. Pipeline parallelism assigns successive parts of model execution to stages. Both can distribute one model across the same number of GPUs, with the same counted weight payload per participant, while creating different dependencies and communication. Compare the actual partition and its workload evidence before treating a parallel degree as a capacity claim. ## Ask which part of the model each GPU executes A proposal saying two-way model parallelism leaves a purchasing question unanswered: do the participants share work inside layers, or do they own successive parts of the model? NVIDIA's Megatron Core guide distinguishes tensor parallelism across individual layers from pipeline parallelism across model depth. Its current latest page was checked October 20, 2026; that moving reference is not a fixed software release. PyTorch 2.14 describes tensor parallelism through a module-specific plan and marks its APIs experimental. Its pipelining package is alpha. The tensor reference was updated May 12, 2026, and the pipeline reference July 24, 2026. A real proposal still needs support for the named model and software version. The distinction below concerns execution of one model. The related DDP/FSDP article owns training-state replication and sharding; the checkpoint-file article owns saved packaging. Neither a retained-state share nor a numbered weight file specifies the execution split. References: [N1](https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html), [P1](https://docs.pytorch.org/docs/2.14/distributed.tensor.parallel.html), [P2](https://docs.pytorch.org/docs/2.14/distributed.pipelining.html) ## TP-223: two arrangements each retain 16 payload units per participant Fictional example TP-223 stipulates one sequential model with four layers, L1 through L4. Each layer contains 8 abstract weight-payload units, giving 4 x 8 = 32 logical units for the model. Each participant denotes one GPU in this example. These invented units have no assigned byte conversion. Pipeline arrangement P places all of L1 and L2 at participant A, its first stage, and all of L3 and L4 at participant B, its second stage. A holds 8 + 8 = 16 units. B also holds 16 units. Their assigned layers differ. Tensor arrangement T instead stipulates a compatible two-way partition of every layer. A holds one half of each layer's weight payload, and B holds the complementary half. Each half contains 4 units. Each participant therefore holds 4 units per layer x 4 layers = 16 units. Both participate in the stipulated tensor-parallel work within each layer. P and T each contain one model and 32 logical weight-payload units, with 16 units assigned to each participant. Those matching totals conceal a different assignment: complete consecutive layers in P, compatible portions of every layer in T. Equal division is an assumption of this toy model, not a property established for arbitrary real layers. This accounting counts only the stipulated weights. It excludes activations, gradients, optimizer state, temporary storage and communication buffers. It supplies no runtime-memory requirement or demonstrated fit. ## Follow two microbatches through P's stage boundary For an illustrative forward ordering, give P two input microbatches, M1 and M2. A microbatch here means an input portion passed through the model. First A executes L1 and L2 for M1. B cannot execute L3 and L4 for M1 until the required output from A arrives. After that handoff, B can work on M1 while A works on M2, if the schedule and available resources permit the overlap. The participants then execute different model depths for different microbatches. They are not executing independent complete model copies. B can process M2 only after M2's first-stage output is available and B is ready for it. Finishing M1 at A does not make M2 ready at B. Each microbatch keeps its own dependency through the sequence. This is an ordering example without durations. It identifies an opportunity for overlap and an initial dependency wait; it establishes no achieved utilization, balanced stage times or universal pipeline-bubble fraction. Training also requires backward work and its communication to be scheduled. This forward illustration defines no optimizer-update boundary; use the related gradient-accumulation article for that separate counting decision. References: [P2](https://docs.pytorch.org/docs/2.14/distributed.pipelining.html) ## Follow cooperating portions of a layer operation in T In T, A and B cooperate on the stipulated partitioned work within L1, then within the later layers. The split is inside those operations rather than a handoff of the complete L2 output to a participant owning L3 and L4. PyTorch 2.14 makes tensor layouts explicit. ColwiseParallel defaults to replicated input and output sharded on the last dimension. RowwiseParallel defaults to input sharded on the last dimension and replicated output. A plan can specify other supported layouts. These defaults illustrate why a tensor-parallel degree alone leaves the operation's input and output ownership unresolved. Communication follows the selected partition and the layouts required by its consumers, including any needed combination or redistribution of results. It cannot be described by assuming that every tensor-parallel operation exchanges the same data in the same way. TP-223 supplies no activation shapes or detailed operation plan. Its 16-unit weight assignment therefore cannot identify message sizes, exchange counts or a suitable interconnect. The related collective-output article explains how communicated results can differ; naming a collective still needs the actual tensors and participating group. References: [P1](https://docs.pytorch.org/docs/2.14/distributed.tensor.parallel.html) ## A pipeline stage can itself contain a tensor group Construct a third toy arrangement using four distinct participants, A, B, C and D. The first pipeline stage contains A and B, which hold complementary compatible halves of L1 and L2. The second stage contains C and D, which hold complementary compatible halves of L3 and L4. Each participant holds 4 + 4 = 8 stipulated weight-payload units; all four together hold the same 32 units. This construction has two tensor groups and two successive pipeline stages within one model. Cooperation inside A/B or C/D and the stage-to-stage handoff describe different relationships. The four-participant count follows from naming those four distinct participants; it is not a general formula for counting every parallelism dimension. No additional data-parallel model replicas have been introduced. A proposal that adds them needs to identify those separate replica groups and their device assignments. PyTorch's pipeline reference describes composition with tensor and data parallelism, but the availability of composition does not establish support for an arbitrary model or arrangement. References: [P2](https://docs.pytorch.org/docs/2.14/distributed.pipelining.html) ## Match the evidence to the problem the split addresses A working-set problem inside one layer operation and a need to distribute model depth raise different questions. Moving whole neighboring layers to another stage does not itself divide the operation that remains on the first participant. A tensor split, in turn, does not mean that each participant owns only half the layer sequence. Ask which constraint the proposed partition is intended to address. For P, request the actual stage boundaries and the tensors crossing them. Equal weight payload at A and B gives no evidence that their processing times match. Ask for the representative microbatch schedule and evidence showing stage execution, handoffs and waiting. For T, request the compatible operation partitions, tensor-group membership, input/output layouts and the communication those choices require. In either case, connect the logical participants to the actual GPUs, servers and connections. A physical count of two GPUs cannot distinguish P's stage handoff from T's cooperating operation paths. The same applies to the two relationships inside the four-participant construction. Keep the model revision, representative inputs, numerical formats, software versions and hardware mapping attached to the provider's fit and performance evidence. A revised partition needs evidence for that revised arrangement. Use the linked DDP/FSDP comparison for the broader peak-memory question rather than treating TP-223's weights as a complete allocation. The useful purchasing result is an explained execution layout with relevant workload evidence. The example's arithmetic supplies neither throughput nor a required link speed, and it selects no real model configuration. ## Take separately selected servers and connections into installation Once the platform and application owners select the supported servers and connections, Leviathan's rack-and-stack service is relevant to the agreed physical installation. It covers equipment mounting, physical IT cabling, labeling and installation records. Use the related service and project-inquiry links to share a nonconfidential equipment summary, released rack and endpoint plans, site, target window and required handoff. Parallelism design, training implementation, software tuning and workload acceptance remain with the appointed platform and application teams; software commissioning is outside the standard physical installation. References: [L1](https://www.leviathansystems.co/services/rack-and-stack) ## Primary sources - [P1 — PyTorch 2.14: Tensor Parallelism](https://docs.pytorch.org/docs/2.14/distributed.tensor.parallel.html). Checked 2026-10-20. Experimental module-planning API and column-wise/row-wise input-output layouts. Applicability depends on compatible operations and the model-specific plan. - [P2 — PyTorch 2.14: Pipeline Parallelism](https://docs.pytorch.org/docs/2.14/distributed.pipelining.html). Checked 2026-10-20. Alpha package describing stage dependencies, microbatch scheduling and composition. Its execution model does not establish a particular workload's fit or efficiency. - [N1 — NVIDIA Megatron Core: Parallelism Strategies Guide](https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html). Checked 2026-10-20. Current latest guide observed October 20, 2026. Used for the individual-layer versus model-depth distinction, without adopting its sizing recommendations. - [L1 — Leviathan Systems: Rack-and-stack and colocation installation](https://www.leviathansystems.co/services/rack-and-stack). Checked 2026-10-20. Agreed equipment mounting, IT cabling, labeling and physical records for separately selected systems. Software bring-up is outside standard physical installation. ## Related Leviathan resources - [DDP replicas and FSDP shards: assess training-state ownership](/articles/gpu-training-ddp-replicas-versus-fsdp-shards) - [Checkpoint files and GPU ranks: separate packaging from execution](/articles/model-checkpoint-file-shards-versus-gpu-parallel-ranks) - [GPU collectives: identify the returned values and their owners](/articles/gpu-collectives-allreduce-allgather-reduce-scatter) - [Gradient accumulation: identify the optimizer-update batch](/articles/gpu-training-microbatch-gradient-accumulation) - [Rack-and-stack installation for the selected equipment](/services/rack-and-stack) - [Send a physical installation project inquiry](/contact?service=rack-and-stack&from=%2Farticles%2Ftensor-versus-pipeline-parallel-model-execution) ## Selected the servers and need physical installation? Share the customer-furnished equipment, released rack and cable plans, site and required physical handoff to discuss rack-and-stack scope and availability. [Discuss the scope](/contact?service=rack-and-stack&from=/articles/tensor-versus-pipeline-parallel-model-execution)