Slurm Federation vs Distributed Training: What GPU Counts Mean

Follow one job identity from submission to start

SchedMD's federation guide describes a common federated job ID. The receiving cluster is the origin; it sends sibling submissions to eligible clusters. Each schedules under its own policies and coordinates allocation with the origin. A successful allocation and start revokes the alternatives.

Follow a fictional six-GPU job, J1, submitted at Cluster A. Assume the submission is accepted and both A and B are viable destinations. Stipulate that B successfully allocates six GPUs and starts J1. The origin record in A becomes revoked while J1 runs in B. J1 has one six-GPU allocation in B; its earlier queue presence in A supplies no additional GPUs or application participants. J1 and J2 below are explanatory labels for distinct invented job IDs. No submission or workload was run.

Keep the records separate when reading a proposal: the job ID identifies the submitted job, an origin or sibling record represents its placement status at a cluster, and an allocation identifies assigned resources. The running application's processes are a further population. The existing Slurm GRES and job-step guide explains resource use within an allocation; counting queue records cannot answer that question.

On requeue, Slurm can submit new siblings and allow a later start elsewhere. This is another placement opportunity, not evidence of live process migration or checkpoint transfer.

References: [S1]

Compare one twelve-GPU job with two independent six-GPU jobs

Keep A and B as separate allocation domains, each with eight GPUs stipulated to be eligible for the intended ordinary jobs. In this example, each assigned GPU resource maps to one complete physical GPU. Assume their resource layout permits a six-GPU request. These are fictional planning assumptions, not hardware specifications or measured availability.

A dashboard can correctly report sixteen GPUs across the two clusters. A job requiring twelve GPUs together cannot obtain twelve through ordinary federation merely because eight plus eight equals sixteen. Neither cluster can supply that allocation, even if all sixteen GPUs are idle. Eight is the physical ceiling within either example domain; it is not a promise that any particular job will receive eight.

Now add an independent six-GPU job, J2, alongside J1. J2 has a different job ID; it is not J1's sibling. For this comparison, expressly stipulate concurrent allocations, with the necessary environments, other resources and scheduling policies satisfied: J1 receives six identified GPUs in B, and J2 receives six different GPUs in A.

Six plus six now accounts for twelve distinct GPU assignments across two jobs and two allocations. Neither request was a twelve-GPU job split in half. The stipulated independent runs do not become synchronized participants in one training execution because their assigned GPU counts add to twelve.

Four GPUs remain outside these two allocations. The example establishes nothing about their idle status or availability, and supplies no throughput or completion-time result. Eligibility alone would not have established concurrent allocations; those were additional assumptions.

That distinction explains the purchasing value. Another destination can help a portfolio of independent jobs that each fit wholly within it. It leaves the critical twelve-GPU job unresolved. Buying a third eight-GPU destination under the same assumptions would raise the fleet total again without supplying that missing allocation.

References: [S1]

Compare complete destinations and the route to each

Describe demand in complete jobs before comparing offers: the resources needed together by one job, which jobs are independent, the intended workload window and the required application outcome. Ask the provider or platform owner which destinations can satisfy each complete job and which resource and scheduling limits apply.

Check the submission path as well as destination size. Slurm's federation guide lists a limitation: a job rejected for local resources, such as partition or node-count constraints, is not sent to siblings even if a remote cluster could run it. This is a documented rejection case, not merely waiting for local resources to become free. A fleet count or common submission interface therefore does not prove a viable route to the intended destination. Identify the actual deployed Slurm release and configuration when assessing the offer.

For each intended destination, establish the applicable code and dependency versions, supported GPU/runtime combination, required input files and storage, authorized data access, and output or checkpoint destination. Suppose the software and authorized dataset access are established only in A. B's stipulated GPU quantity does not yet establish that either intended six-GPU run can use B.

Resolve permitted data locations, any movement and checkpoint usability with the appointed platform, application and data owners. Do not treat federation as evidence that datasets have been copied or access rights granted. The environments need to support the intended runs; they need not be inferred identical from a shared cluster interface.

An offer supporting the two independent six-GPU jobs answers a different requirement from an offer supporting one twelve-GPU run. Compare the complete destinations it supplies, their remaining preparation and ongoing operating responsibilities against the actual job portfolio. The GPU cloud-versus-colocation guide addresses ownership of those layers. Keep unresolved prerequisites visible when deciding which capacity the purchase can use.

References: [S1]

Check named scheduling features against their actual boundary

Heterogeneous does not automatically mean cross-cluster. SchedMD's heterogeneous-job documentation says that, in a federation, the entire heterogeneous job executes on the submission cluster; its components cannot execute on different federation members. The same document describes a non-federated multi-cluster submission case in which one selected cluster receives the entire job.

These are specific Slurm behaviors in the online documentation checked October 20, 2026. They do not establish a universal prohibition on distributed execution across sites. A label describing varied resources cannot substitute for the supported execution arrangement.

References: [S2]

Identify the participants in one cross-site training run

A proposal for GPUs in A and B to participate simultaneously in one training run must describe how they become participants in that execution. PyTorch 2.14's DistributedDataParallel reference describes gradient synchronization among model replicas through a process group, and identifies construction, forward execution and output differentiation as synchronization points. These are relationships among running processes. A Slurm sibling queue record is not another DDP model replica.

Ask the appointed platform and application owners for the supported allocation and launch arrangement, participating resource identities, communication dependencies, data/checkpoint paths and workload acceptance basis. Require any claimed result to identify the actual application, software, participating configuration and conditions. A submission receipt or a connectivity observation alone cannot establish acceptable training behavior.

The buyer must define the useful outcome: for example, correct completion of an agreed representative run within its required time and data constraints. The responsible teams establish the method and evidence. The cited DDP concepts supply neither a cross-site suitability finding nor a performance threshold or implementation procedure.

Once that execution requirement is established, the existing multi-site network-handoff guide addresses transport demarcations and the physical package at each site.

References: [P1]

Bring the resulting equipment scope to the physical deployment discussion

If the capacity decision includes installing customer equipment, use Leviathan's Complete Physical Infrastructure Deployment service to discuss agreed receiving, rack integration, onsite installation, cabling and physical handoff records. Bring the selected equipment, site destinations, appointed material suppliers and the physical stages required.

Scheduler operation, cloud capacity provision and application acceptance need their own appointed owners. Leviathan's physical service is not evidence that it supplies or operates a federated GPU platform. Scope, resources and availability are agreed for the engagement.

Use the service and project-inquiry links below with a nonconfidential outline of the equipment and requested physical work. The execution requirement helps explain why that equipment is needed; the installation inquiry should identify the actual package to be delivered.

References: [L1]