---
title: "Six Checkpoint Files Do Not Specify Six GPU Ranks"
description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20."
canonical: "https://www.leviathansystems.co/articles/model-checkpoint-file-shards-versus-gpu-parallel-ranks"
last-updated: "2026-10-20"
---

# Six Checkpoint Files Do Not Specify Six GPU Ranks

Leviathan Systems — Technical content
Published 2026-10-20; updated 2026-10-20.

Treat logical model contents, file packaging and supported execution as separate deliverables. A checkpoint's numbered files can describe storage partitions without specifying a distributed process count. Before purchasing GPUs, require the appointed software provider to connect the exact artifact to a supported runtime layout and relevant workload evidence.

## Read the index as a description of stored weights

In Hugging Face Transformers v4.47.1, a sharded checkpoint divides saved weights among files. Its illustrated weight_map index connects parameter names to storage files. The six numbered weight files describe that saved package. Device placement appears separately on the page.

An index can locate a stored weight while leaving its eventual process or device unresolved. The edition's shard-by-shard loading discussion supplies neither a GPU-rank specification nor a universal runtime peak-memory formula.

Ask which meaning of shard the supplier intends: a storage grouping or ownership during execution.

References: [H1](https://huggingface.co/docs/transformers/v4.47.1/big_models)

## FS-211: four files and eight files contain the same 12 GiB

Fictional model FS-211 contains twenty-four uniquely identified tensors, T01 through T24. Each is stipulated to contain 0.5 GiB of weight payload. One GiB means 1,073,741,824 bytes. The total stipulated weight payload is 24 × 0.5 GiB = 12 GiB.

Package A puts six tensors in each of four payload files. The groups are T01–T06, T07–T12, T13–T18 and T19–T24. Each file contains 6 × 0.5 GiB = 3 GiB of weight payload. Four such files contain 12 GiB.

Package B puts three consecutive tensor identities in each of eight payload files, starting with T01–T03 and ending with T22–T24. Each contains 3 × 0.5 GiB = 1.5 GiB of weight payload. Eight such files also contain 12 GiB.

Both packages are stipulated to preserve exactly the same tensor identities, shapes, numerical representation and values, with one stored copy of each tensor. Each has a complete index mapping all twenty-four identities to its own files. Repackaging changed the file population and the index's file references, while preserving the logical weights.

Only this payload arithmetic excludes headers, index data and other metadata. Those extras remain part of a real delivery's storage requirement. Neither A's four files nor B's eight files establishes a four-GPU or eight-GPU execution. The 12 GiB figure also supplies no runtime fit result.

## Describe two-rank and four-rank proposals separately

PyTorch 2.14 defines world size as the number of participating processes and rank as the current process's identifier within that job. A rank count therefore describes a process population. Connecting those processes to GPUs requires a stated execution layout.

Consider two further fictional proposals for FS-211. R2 names a two-process group, ranks 0 and 1, and proposes partitioning model state between those participants. R4 names a four-process group, ranks 0 through 3, and proposes partitioning state across all four. These are proposed execution arrangements with their supported mappings still unresolved.

For each, the software provider must establish the supported model partition, the processes' device assignments, any state that remains replicated, and the state required during computation. It must identify the model revision, representation, framework and backend versions, hardware and communication requirements. Either package's index can locate the stored tensors without answering those execution questions. Nothing in FS-211 establishes that either package can be used by either proposed runtime.

Keep relevant runtime evidence attached to each proposal: per-device memory through loading and the requested work, numerical and task-quality results, and workload performance under the agreed operating conditions. Evidence for R2 cannot silently become evidence for R4 after changing the process population. Neither proposal has been run in this example.

The related DDP/FSDP guide explains runtime replicas, retained state and temporary gathering. Use that guide to assess the proposed ownership mechanism; the file-count comparison here cannot settle it.

References: [P1](https://docs.pytorch.org/docs/2.14/distributed.html)

## A complete index can still point to missing payload

Now remove A's third payload file, containing T13–T18, and supply no replacement copy. The three remaining payload files contain eighteen tensors and 9 GiB of the stipulated weight payload. Six required tensor identities and 3 GiB of payload are absent from the delivered package.

The index may still list all twenty-four identities. Its references to the absent file do not recreate those values. Plausible filenames on the remaining files also cannot establish completeness.

This exposes a separate handoff decision. The provider needs to reconcile the expected logical contents, the package's index and the payload actually delivered. Counting three files, or even receiving four filenames in a manifest, cannot establish that all expected tensors are present with the required identity and contents. A's loss says nothing about whether the intended execution had two ranks, four ranks or another supported arrangement.

FS-211 demonstrates completeness and packaging using stipulated contents. It supplies no observed restore, numerical comparison or workload result.

## Some checkpoint formats retain distributed-layout information

Distributed checkpoints need a different reading. PyTorch 2.14's Distributed Checkpoint reference, updated July 8, 2026, describes multiple files with at least one per rank and load-time resharding between cluster topologies.

Its tensor metadata includes properties, sizes and chunks. The filesystem writer documents global or rank-local metadata depending on rank coordination. Such a checkpoint can retain partition and rank context that a file count conceals.

Ask which format and supported loading path the handoff provides. Resharding support can separate the saved arrangement from the destination topology. It does not make arbitrary rank-local checkpoints interchangeable with Transformers weight packages. The appointed software provider must establish support for the actual artifact and destination.

References: [P2](https://docs.pytorch.org/docs/2.14/distributed.checkpoint.html)

## Compare completeness, packaging and execution as three deliverables

For logical completeness, identify the model revision and the expected tensor identities, shapes, numerical representations and values. Ask what evidence connects that description to the delivered payload. In FS-211, A and B meet the same stipulated logical description; damaged A does not.

For packaging, identify the checkpoint format, index, payload files and required sidecar material. Make the relationship between the manifest and the actual delivery explicit. A manifest can describe what should arrive and record supplied integrity references. Its existence alone establishes no numerical behavior, recovery outcome or workload validation.

For supported execution, identify the proposed process population, state ownership, hardware and runtime versions, supported artifact-loading relationship and evidence for the requested workload. If that evidence has not been supplied, record the execution decision as unresolved instead of substituting the payload-file count.

These distinctions also make changes easier to assess. If a supplier replaces A with B while preserving the stipulated logical model, the changed deliverable is packaging. If it replaces R2 with R4, it changes the execution proposal and needs corresponding support and evidence. If it omits T13–T18, it changes completeness. Each change raises a different buyer question.

Use the related checkpoint-versus-snapshot article when the requirement is training continuation. Use the quantized-memory article for the difference between weight representation and inference memory, and the ONNX/TensorRT article for artifact reuse after a target change. Resolve those requirements with their responsible software owners before treating a file handoff as an equipment specification.

## Carry the selected equipment into the physical deployment scope

When the software and platform owners have selected equipment that needs installation, Leviathan's complete physical deployment service is the relevant next discussion. The service and project-inquiry links below connect an equipment list, destination, target window and required physical handoff to the agreed receiving, rack integration, hardware QA, installation and cabling stages.

Share a nonconfidential equipment summary and identify the team receiving the installed system. Model loading, conversion, parallelism implementation, tuning and workload acceptance remain with separately appointed software providers. The physical inquiry concerns the selected equipment and installation outputs.

References: [L1](https://www.leviathansystems.co/services/turnkey-gpu-deployment)

## Primary sources

- [H1 — Hugging Face Transformers v4.47.1: Instantiate a big model](https://huggingface.co/docs/transformers/v4.47.1/big_models). Checked 2026-10-20. Versioned explanation of checkpoint indexing, sharded loading and separately described device placement.
- [P1 — PyTorch 2.14: Distributed communication package](https://docs.pytorch.org/docs/2.14/distributed.html). Checked 2026-10-20. World size and rank identify participating processes; actual state ownership and device assignments require the proposed execution layout.
- [P2 — PyTorch 2.14: Distributed Checkpoint](https://docs.pytorch.org/docs/2.14/distributed.checkpoint.html). Checked 2026-10-20. Reference updated July 8, 2026: per-rank files, tensor/chunk metadata, global or rank-local metadata and documented load-time resharding.
- [L1 — Leviathan Systems: Complete Physical Deployment](https://www.leviathansystems.co/services/turnkey-gpu-deployment). Checked 2026-10-20. Physical delivery stages and handoff scope; software commissioning is outside standard inclusions.

## Related Leviathan resources

- [Training checkpoints and storage snapshots: identify the recovery state](/articles/gpu-training-checkpoint-versus-storage-snapshot)
- [DDP replicas and FSDP shards: assess runtime state ownership](/articles/gpu-training-ddp-replicas-versus-fsdp-shards)
- [ONNX models and TensorRT engines: assess artifact reuse](/articles/onnx-model-versus-tensorrt-engine-portability)
- [Quantized weights and inference memory: assess the complete runtime](/articles/quantized-model-weights-versus-inference-memory)
- [Discuss complete physical deployment for the selected equipment](/services/turnkey-gpu-deployment)
- [Send a physical deployment project inquiry](/contact?service=turnkey-gpu-deployment&from=%2Farticles%2Fmodel-checkpoint-file-shards-versus-gpu-parallel-ranks)

## Need physical deployment for the selected GPU system?

Share a nonconfidential equipment summary, site, target window and required physical handoff to discuss installation scope and availability.

[Discuss the scope](/contact?service=turnkey-gpu-deployment&from=/articles/model-checkpoint-file-shards-versus-gpu-parallel-ranks)