Replicas or Shards: Why More Training GPUs Do Not Automatically Fit a Larger Model

Identify the training state being counted

A model's saved weights are only one part of a training-memory discussion. Parameters hold learned values. Gradients describe the derivatives used to update them. An optimizer can retain additional state associated with those updates. PyTorch's training-memory tutorial identifies parameters, gradients and optimizer state separately from activations saved for backward computation and intermediate tensors created during execution.

For this comparison, persistent training state means the parameter, gradient and optimizer-state population being compared across layouts. It is an accounting boundary, not a claim that every gradient remains allocated throughout every step. Its size depends on the actual training recipe. Ask the software owner what the estimate includes before comparing it with local GPU memory; a weights-only figure cannot answer the larger question.

References: [P3]

Replicas preserve a local fit question

PyTorch's DDP tutorial, updated September 23, 2025, describes a model replica in each participating process and collective gradient synchronization. A rank identifies a process in the distributed group. Processing different data across ranks does not itself partition their model copies.

Consider a simple layout with one rank and its complete replica on each GPU, with ordinary local optimizer state and no additional sharding or offload. Adding another such participant adds another copy of that state. It does not divide the first participant's requirement.

That layout is an explicit assumption. DDP also works with model parallelism, where a replica spans multiple devices. Ask how replicas map to devices before applying the simple comparison.

References: [P1]

Shards change state ownership and require gathering

PyTorch's FSDP2 tutorial, updated September 2, 2025, describes sharding parameters, gradients and optimizer state across ranks. Parameter shards are gathered into the unsharded parameters needed for computation. Backward computation reduces and scatters gradients into shards, and the optimizer updates sharded parameters using sharded gradients.

The distinction is ownership over time: a participant retains a share of the distributed state, but computation temporarily needs more than its retained parameter shard. A proposal should identify the sharding group and which state it distributes. The word sharded alone does not specify the complete memory behavior.

References: [P2]

Compare four 16 GiB GPUs with a stipulated 24 GiB state population

Fictional example TS-24 uses four GPUs, each with 16 GiB of local memory. Stipulate a 24 GiB persistent training-state population for one model and its chosen training recipe, before activations and runtime extras. These are invented accounting inputs, not specifications for a named GPU or estimates derived from a model's parameter count.

Offer R uses the simple replicated layout described above: one rank per GPU, with a complete 24 GiB population at each participant. Each would need 24 GiB against 16 GiB locally. That stipulated persistent requirement alone fails the local capacity screen. Four complete populations would total 96 GiB; installing 64 GiB across four devices does not make those replicas fit.

Offer S instead stipulates that the same 24 GiB population is evenly sharded across all four participants. The retained share is 24 / 4 = 6 GiB per participant. This arithmetic changes the persistent-state comparison, but leaves the runtime fit question open.

The difference between 16 GiB and 6 GiB is not a measured 10 GiB operating margin. It must accommodate the other allocations that coexist at the busiest point, and the usable allocation still needs confirmation. Nothing supplied here establishes their size. Nor does the 64 GiB installed total become one device allocation. S has a plausible state-distribution explanation to evaluate; it has not demonstrated that the job runs.

Evaluate the runtime peak beyond the retained shard

The 6 GiB line omits the unsharded working sets needed during computation. It also omits activations retained for backward computation and temporary intermediate buffers. Those objects do not disappear because the state accounting is divided by four. The relevant question is which allocations coexist on each device, rather than the size of a shard in isolation.

PyTorch's training-memory example shows allocations changing through forward, backward and optimizer work, including optimizer state initialized during the first optimizer step. A measurement taken only before training, or at a quiet point between operations, can miss the requirement that matters.

Keep measurement boundaries visible. PyTorch 2.14 distinguishes memory occupied by tensors from memory reserved by its caching allocator. Its memory-profiling documentation also notes that NCCL can allocate GPU memory outside that allocator, beyond the profiler's default view. A tensor-only trace therefore needs its coverage stated; it is not automatically a complete device-memory account. Ask the evaluating team to reconcile its reported peak with the allocation actually available on every participating device.

References: [P2][P3][P4][P5]

Compare the evidence behind the capacity claim

Give the proposed training providers one common workload description: model and revision, which parameters are trained, numerical formats, optimizer, representative input or sequence sizes, and both per-participant and total batch settings. Record the framework version, GPU variant and count, ranks, replica or sharding groups, and mapping to devices and nodes. Identify any offload or other memory-saving method included in the proposal. Otherwise an apparent hardware comparison may actually compare different training recipes.

Request per-device peak-memory evidence covering initialization and representative complete training steps, with the measurement's scope and operating conditions stated. Ask which state remains replicated, what is gathered for computation, what remains outside the estimate and how the proposed operating margin was established. An average across four devices cannot demonstrate that the most heavily loaded device fits.

Then compare useful training performance under the same agreed outcome and quality requirement. Gradient synchronization and parameter gathering require communication; a lower retained-state figure does not predict the time taken by the complete job. Keep the actual interconnect and software configuration attached to the result. The related GPU benchmark guide explains the broader comparison of completed work.

For TS-24, R needs a changed layout or requirement before it can clear the stipulated capacity problem. S needs the missing runtime and performance evidence. If a supplier changes the batch settings, precision or state placement to make its proposal fit, record that changed basis and reassess the comparison. A capacity estimate, a demonstrated fit and an acceptable training result are separate findings.

References: [P1][P2]

Take the selected system into physical planning

Use the related memory-capacity guide to confirm the exact GPU variant and local capacity. Once the workload and platform owners select a supported system and distribution layout, translate that choice into the physical equipment and connection package. A software decision changes installation needs only through the actual servers, placement and connections it requires.

Leviathan's rack-and-stack service covers agreed equipment mounting, IT cabling, labeling and physical installation records. The service and project-inquiry links below are useful when the selected customer-furnished equipment needs that work. Share a nonconfidential equipment summary, site, released rack and endpoint plans, target window and receiving team.

Training architecture, sharding implementation, software tuning and workload acceptance remain with the appointed platform and application providers. Define the physical handoff to those teams alongside the equipment selection.

References: [L1]