Slurm GPU Counts: Separate Hardware Inventory, Job Allocation and Job Steps

Start with installed hardware and managed GRES

An equipment inventory counts physical GPUs in the delivered servers. Slurm uses Generic RESources, abbreviated GRES, to describe resources it can schedule. SchedMD documents that GRES management must be explicitly enabled. Installing a GPU therefore does not, by itself, make it a resource available through Slurm.

The scheduler's description has its own identity and scope. SchedMD's GRES configuration documentation describes node-associated resources, counts and device mappings, with an optional type that can distinguish a GPU model. A quantity without that mapping leaves the buyer unable to tell which hardware the scheduler count represents.

Reconciliation answers whether the intended resource population is represented correctly. A job allocation answers which resources one job has received. Those are separate results: a delivered eight-GPU server can be correctly inventoried without establishing an eight-GPU entitlement for every job.

References: [S1][S2]

Read the GPU quantity at the scope it names

Slurm's user guide defines a job as a resource allocation assigned to a user for a period, and a step as a set of tasks within that job. Several steps may use portions of one allocation. The parent job is the resource boundary from which those steps draw.

The submission documentation distinguishes a total GPU requirement for the job from GPU requirements per node and per task. For a stipulated two-node allocation, four GPUs per node describes eight across the job; four GPUs for the entire job describes a different quantity. The word four is insufficient without its scope.

A typed GPU request adds a resource classification. SchedMD requires typed and untyped requests to remain consistent between a job and its steps; a typed parent allocation cannot silently become a request for a different GPU type inside a step.

These semantics follow SchedMD's online documentation checked October 20, 2026. The appointed platform team needs to identify the deployed Slurm release and configuration. In particular, SchedMD ties the GPU-specific request family to its select/cons_tres resource-selection plugin. This article explains allocation relationships rather than supplying a configuration or launch procedure.

References: [S1][S3][S6]

A task count does not assign one GPU to each process

A task count describes processes to be launched. An MPI rank is a process identity within the parallel execution. Neither is a GPU identity. Four tasks alone do not express whether the intended application needs one GPU per task, several GPUs per task or a different process-to-device relationship.

Slurm documents separate GPU-per-task requirements and GPU binding semantics. Its documented per-task GPU request also establishes an implicit binding, which an explicit binding specification can override. Thus GPU allocation and task binding are connected, but a bare task count does not settle either decision.

When a platform offer says four workers, ask whether that means four processes or four assigned GPU resources, and how the processes use those resources. The application owner must define the intended relationship; counting ranks cannot establish the allocation.

References: [S4][S5]

Steps use the parent allocation without multiplying it

For explicitly requested GRES, a step defaults to the parent's requested GRES on its step nodes; SchedMD identifies an exception for resources only implicitly included by an exclusive job. Steps can request different counts within that allocation. Two steps do not automatically receive half each.

An assigned GPU can be unavailable to another step. In the ordinary non-overlapping case, a step may wait or be rejected until its requested resources are available. Several steps therefore imply neither automatic sharing nor concurrent execution.

The steps also need their other requested resources, including CPUs. Owning four GPUs at the job level does not establish that two two-GPU steps can start together.

References: [S1][S4]

Eight installed GPUs, four allocated GPUs, two two-GPU steps

Consider a fictional server with eight physical GPUs labeled P-A through P-H. In this example each allocated GPU resource maps to one complete physical GPU. One job has received four identified resources: P-B, P-D, P-F and P-H. The labels are invented hardware identities, not CUDA ordinals.

Stipulate two two-GPU step allocations with mutually exclusive GPU sets. Step A receives P-B and P-D; Step B receives P-F and P-H. Their sets do not overlap. These assignments are stipulated to explain the relationship; they are not an observed execution.

The job holds four GPUs. Across these two steps, four references point to those same four GPUs. Eight physical inventory objects plus four job references plus four step references do not make sixteen physical GPUs. A resource appears at several levels because each level describes a different relationship to the same hardware.

The remaining P-A, P-C, P-E and P-G are outside this job's allocation. Nothing in the example establishes that they are idle, schedulable or available to this user. Nor does changing Step B's task count grant it any of those GPUs.

If a later step reuses a pair after that pair becomes available within the job, the hardware population still does not grow. The useful questions are which identities are assigned, to which execution and during which allocation interval.

References: [S3]

Device 0 needs its execution context

NVIDIA's CUDA documentation explains that an application enumerates the devices visible to it. The first visible device has local ordinal 0, even when its identity in the machine-wide enumeration is different.

In the fictional example, Step A could see its assigned pair as local devices 0 and 1, while Step B could see its different pair as 0 and 1. Repeated local numbers would not prove that the steps share physical GPUs. A comparison needs the node, execution context and mapping back to the assigned resource identities.

Conversely, a local device list is not an inventory of every GPU installed in the server. It answers which devices that application can enumerate. Keep that observation attached to its scope when comparing it with a scheduler allocation or a hardware record.

References: [N1]

Match the buying question to the resource layer

A hardware purchase establishes a delivered physical population. Scheduler resource reconciliation establishes how that population is represented for allocation. A job allocation establishes the resources assigned to a particular job, while step allocation and task binding determine how its executions use that holding. A proposal should state which of these results it supplies.

For example, a physical installation of the eight-GPU server and a four-GPU allocation to one job can both match their respective scopes. The difference is not evidence that four GPUs are missing. If the actual requirement is for that job to receive all eight, the platform team must address that requirement at the allocation layer.

If the project includes equipment installation, Leviathan's Complete Physical Infrastructure Deployment service provides the relevant scoping route for receiving, rack integration, cabling and agreed physical records. Use the project inquiry below with the actual server population, site and required physical work. Slurm administration, resource policy and application allocation belong with the separately appointed platform team.