More Inference Replicas or More GPU Nodes? Separate Two Autoscaling Decisions

Identify what the offer actually scales

A replica is another instance of the declared workload. A worker node provides resources on which workload Pods can run. Kubernetes distinguishes horizontal workload scaling, which changes replica count, from infrastructure scaling, which adds or removes nodes. The quantities are related through the workload's resource requirements.

For a GPU inference offer, first establish what one replica contains and requests. The proposed application layout determines GPUs per replica and replicas per worker. Ask the supplier to state the resource requirement per replica before translating a replica ceiling into hardware demand.

A proposal that scales replicas within a fixed worker population can be complete for a buyer who already supplies sufficient capacity. A proposal that provisions workers can be complete for a buyer whose application team already determines replica demand. If the buyer expects both functions from one offer, each must be identified. The word elastic leaves that purchase undecided.

References: [W1]

The application loop changes the desired replica population

Kubernetes Horizontal Pod Autoscaler, or HPA, adjusts a scalable workload's desired scale using observed metrics. The workload controller then manages the corresponding Pods. HPA's configured metrics and supported metrics APIs determine which signals it can use.

For an offer, ask which workload is controlled, which metric reaches that controller and what the configured replica bounds are. A supplier proposing GPU telemetry must identify its supported integration. The presence of GPU hardware does not make GPU utilization an automatic HPA input. This comparison selects no metric or tuning value.

The useful output of this loop is a demand decision: how many replicas the application is now asking to have. That target can increase before sufficient workers exist. The buyer should therefore be able to distinguish a changed desired count from the population actually available to serve requests.

References: [H1]

The capacity loop responds to requests that existing workers cannot place

In Kubernetes' documented request-driven provisioning model, a node autoscaler evaluates unschedulable Pods, their resource requests and scheduling constraints, together with its permitted node configurations. It does not directly use those Pods' measured resource usage to make that provisioning decision.

This explains how the loops connect: additional desired replicas produce additional Pod demand; a shortage of eligible worker resources can then give node provisioning work to do. The node autoscaler does not decide the application's desired replica population. Ask the supplier to identify the actual provisioning integration and eligible GPU worker configurations, alongside the application autoscaler.

References: [N1]

Work through four running replicas and a target of six

Consider a fictional cluster with two worker nodes. Each exposes two exclusive whole-GPU resources, and four inference replicas are running, one Pod per replica and one GPU per Pod. All other placement and admission requirements are stipulated satisfied. No GPU sharing is used.

Now stipulate that the workload autoscaler changes the desired count to six and the workload controller creates the two additional Pods. The current workers expose two times two, or four, GPU resources, all occupied by the original four replicas. The two new Pods remain unscheduled for lack of GPUs. The state is six desired, four running and two pending. Six is the demand target; only four replicas are running.

An eligible additional worker exposing two usable whole-GPU resources could cover the missing resource count: four plus two equals six. That arithmetic does not establish that the provider can supply the worker, that it joins healthy with those resources available, or that the new application replicas become ready. Those are subsequent observations. No deployment or scaling run is represented by this example.

Compare three fictional offers against this unchanged request. A includes only replica autoscaling; the buyer must already provide the extra capacity or obtain it elsewhere. B includes only worker provisioning; the buyer must supply the application decision that raises the desired count. C explicitly includes both loops and their integration. C addresses both missing functions, but its description still needs applicable limits and result evidence. Neither A nor B is inherently incomplete when the buyer has deliberately retained the other function.

The example isolates a resource-count shortage. It changes neither the serving architecture nor the GPU-allocation model.

References: [N1][K1][P1]

Resolve the physical capacity and operating limits behind a worker

Kubernetes defines a node as a virtual or physical machine. In cloud-backed node provisioning, the requested resource is commonly a virtual machine. Adding that worker consumes the provider's underlying capacity; it does not mean the provider has manufactured and installed a new GPU server in response to the application.

For an owned cluster, establish which existing resources the supported automation can make available. A requirement for additional physical servers still needs an equipment and deployment plan. Identify the actual equipment and site interfaces before scoping that physical addition.

Keep the operating limits separate. The application has a permitted replica range. The node provisioner has allowed configurations and provisioning limits. The provider has its own resource limits and available capacity. Kubernetes explicitly recognizes that configuration or capacity constraints can prevent provisioning. As one provider example, AWS documents per-Region EC2 quotas and failed requests when a quota would be exceeded.

Ask the selected provider which limits apply to the proposed worker population. An application's configured maximum does not establish provider capacity for that maximum. The linked cloud-availability guide covers the separate quota and capacity-purchase questions.

References: [K1][N1][H1][A1]

Compare the included loops with evidence of the intended result

Return to the six-replica requirement when evaluating a supplier's evidence. A changed desired count answers whether the application requested growth. Evidence that the additional eligible worker became available answers the capacity question. Observations of the actual replica population then establish how far that demand was fulfilled.

Kubernetes distinguishes a Pod's Running phase from its Ready condition. Pending also includes startup waits, so a pending count alone does not identify a GPU shortage in a real cluster. The fictional case supplied that cause explicitly. Have the platform team explain the observed state for the proposed configuration instead of inferring it from one dashboard total.

Finally, compare the serving result for the intended workload. Ask for the model and software configuration, request pattern, observation interval, completed useful output and required response-time conditions. Tie those observations to the replica and worker changes being demonstrated. Evidence collected only after extra capacity was already present leaves the response to a capacity shortage unexamined.

This is why a desired count is not an SLA. It is one requested state within a chain of decisions and dependencies. An offer for both loops should explain what evidence it supplies when growth succeeds and what remains visible when a limit prevents it. Choose the package that supplies the functions your team needs, within an operating range supported by relevant evidence. The benchmark guide develops the separate performance-comparison question.

References: [P1]

Bring an actual equipment addition to the physical deployment inquiry

If the capacity decision requires customer equipment installation, bring the confirmed server configuration, quantities, site, rack positions and IT connection requirements to Leviathan's Complete Physical Infrastructure Deployment service. Identify the platform team that will receive the installed equipment and make it available as workers.

The service can combine agreed receiving, rack integration, hardware QA, onsite installation, cabling and physical records. Included checks, resources and availability are engagement-defined. Autoscaler configuration, Kubernetes administration and inference acceptance remain with the appointed platform providers; facility electrical construction, liquid-side work and licensed engineering are outside the physical service.

Use the related service and project inquiry to discuss the equipment and connection work that follows from the selected capacity plan.

References: [L1]