Twelve Training Slots Can Cover Only Ten Distinct Examples

Follow records through the index schedule

Start with a fixed dataset whose records have stable identities. An index identifies the record to retrieve. A sampler supplies index occurrences to participating ranks, where a rank identifies a process in the distributed group. A loader retrieves the assigned records and groups occurrences into local batches. Selecting index 0 twice creates two occurrences of the same record; grouping them into batches does not create another distinct record.

This article counts those identities, not whether two different records happen to contain identical text or images. PyTorch 2.14 documents DistributedSampler for a constant-size dataset whose instances return the same elements in the same order. A changing stream or a different definition of record identity needs its own population account.

Keep this question separate from the adjacent training articles. Replicas or Shards explains model-state ownership. The gradient-accumulation guide explains which computations contribute to an optimizer update. Neither boundary determines how many distinct dataset records an index schedule covers.

References: [P1]

Work through ten records and three ranks

Fictional example SC-241 stipulates ten distinct records numbered 0 through 9 and three data-parallel ranks. There is no shuffle, no augmentation that changes record identity, and no loader-level dropping. Every assigned occurrence is processed. These are declared schedules for reasoning, not an observed training run or a claim about an unspecified pipeline.

The padding policy extends the ordered indices 0 through 9 by appending 0 and 1. This creates twelve positions, allowing four per rank. Strided assignment gives each rank every third position, starting at its rank number. The resulting assignments appear below.

Across all ranks, records 0 and 1 each occur twice; the other eight records each occur once. The total is 2 + 2 + 8 = 12 occurrences covering ten distinct records. Every rank has four locally distinct records, yet adding those three local distinct counts would overstate global coverage. The shared identities must be counted once across the whole group.

The alternative tail-dropping policy retains indices 0 through 8 and assigns three per rank. It produces nine occurrences covering nine distinct records. Record 9 is omitted. Equal allocation has been achieved under both policies, with different consequences for coverage.

Understand why divisibility changes the count

Ten does not divide evenly by three. Equal whole-number assignments can round up to four occurrences per rank or down to three. The first choice needs twelve positions; the second retains nine. Those totals explain the two extra occurrences and the one omitted record in SC-241.

The PyTorch v2.14.0 DistributedSampler implementation makes this mechanism explicit: it can extend the index list from its beginning or truncate its tail, then select strided positions for each rank. Its shuffled path first changes the index order. That supports the mechanism illustrated here without making SC-241 a description of every training framework or configuration.

A supplier saying one epoch covered the dataset has therefore left a question open. Did it mean exhaustion of the scheduled occurrences, coverage of every distinct record, or something else? In the padded schedule, all ten records are covered with repeats. In the tail-dropping schedule, completing every assigned occurrence still leaves record 9 untouched.

References: [P2]

Keep the two drop_last boundaries separate

The same parameter name can conceal different boundaries. DistributedSampler's drop_last concerns divisibility across participating ranks. BatchSampler's drop_last concerns whether the last local batch contains the requested number of occurrences.

Consider a separate variation of the padded SC-241 schedule. Each rank still receives four occurrences, but local batching now groups them in threes and drops each incomplete final batch. Rank 0 drops its final 9, rank 1 drops its final 0, and rank 2 drops its final 1. Only nine occurrences are processed. Records 0 and 1 remain in earlier batches, while record 9 is now absent.

This changes the baseline assumption that every assigned occurrence is processed. A sampler assignment can therefore describe twelve positions without proving twelve were consumed. Read both the rank-allocation policy and the local-batch policy before interpreting the claimed epoch population. A bare statement that drop_last is enabled or disabled does not identify which boundary the offer means.

References: [P2][P3]

Name what the rate counts

For exactly one completed padded SC-241 pass taking T seconds, the occurrence rate would be 12 / T occurrences per second. Distinct coverage over that same pass would be 10 / T distinct records per second. For the tail-dropping pass, both numerators would be nine, divided by that pass's own elapsed time. SC-241 supplies neither time, so it establishes no speed ranking.

State the interval over which identities are counted. Ten distinct records per pass, summed over two passes, is twenty record appearances counted once within each pass. It is still only ten distinct identities across both passes if the population is unchanged and both cover it. The uniqueness count cannot silently reset at every epoch while being described as new records across the complete run.

For an offered examples-per-second result, ask whether the numerator counts processed occurrences or distinct identities within a specified window. Retain the timing boundary too: a rate for a timed portion needs the population consumed during that portion. Do not replace an occurrence rate with a unique-record rate by assuming every scheduled position is new data.

Preserve shuffle state and intended repetition

An epoch description also needs its ordering policy. PyTorch 2.14 documents a shared shuffle seed and epoch-dependent sampler state, warning that unchanged epoch state reuses the ordering. With shuffling, the identities repeated or omitted can depend on that epoch's order. An epoch number alone does not demonstrate cumulative coverage.

In SC-241, shuffle is deliberately absent. Repeating the same padded schedule repeats 0 and 1 again. Repeating the same tail-dropping schedule continues to omit 9. Changing the order could change those identities, but no unprovided later schedule can be assumed to recover an omitted record.

Repetition can also be intentional. The versioned PyTorch sampler source includes weighted sampling with replacement, which permits repeated indices. A buyer should compare the intended frequency and treatment of records, rather than treating the largest distinct count as an automatic quality advantage. Frequency, any loss weighting and the training objective belong together. Matching occurrence totals or distinct totals alone does not prove equivalent gradients or model quality.

References: [P1][P3]

Compare the population behind the offer

Give competing training providers the same dataset revision and definition of a record. Have the appointed software team reconcile the scheduled indices with what is actually consumed, including repeats, omissions, local-batch dropping, shuffle state and intended weighting. A nonconfidential identity-count summary can support this discussion without exposing the training records.

For SC-241, an interpretable statement would be: ten fixed records, three ranks, twelve processed occurrences in one padded pass, ten distinct records covered, with 0 and 1 each processed twice. The tail-dropping offer instead reports nine processed occurrences, nine distinct records and omission of 9. Neither statement establishes the time or quality of the other offer.

Once those populations are comparable, use the GPU performance guide to evaluate completed training work against its required outcome. Matching GPU counts and epoch labels cannot repair different workload populations. If the provider changes the sampling policy to improve a reported rate, the revised population belongs beside that rate.

Carry the selected system into physical installation

After the platform and application owners select the training system, Leviathan's rack-and-stack service and the related project inquiry provide a next step for agreed physical mounting, IT cabling, labels and installation records. Share a nonconfidential equipment list, site, released rack and endpoint plans, target window and receiving team.

Training-data preparation, sampler implementation, software tuning and model-quality evaluation remain with the appointed software providers. Software commissioning is outside the standard physical installation scope. The useful installation input is the selected equipment and its physical work package.

References: [L1]