GPU Jobs During a Kubernetes Control-Plane Outage

Follow a new worker request through the control plane

The Kubernetes API server exposes the Kubernetes API, and etcd provides its backing store. A requested change that needs storage is therefore more than a message reaching an API endpoint. It needs the backing store to accept that change. The worked outage below is loss of etcd write quorum; failure of another control-plane component can affect a different set of operations.

The etcd v3.6 failure documentation states that majority-member failure prevents further writes. Combining that statement with Kubernetes' backing-store role gives a specific consequence: a Kubernetes change requiring a write through that unavailable store cannot complete. It does not establish the behavior of every API endpoint. A successful local read alone also cannot establish that new writes are working; the related etcd quorum and read-freshness article develops that separate read contract.

Creating workers adds further dependencies. Kubernetes' controller documentation uses the built-in Job controller to illustrate the chain: it asks the API server to create Pods rather than running containers itself. That is an example of a controller's role, not a claim that every distributed GPU application uses a Kubernetes Job.

The default scheduler selects a node for an unscheduled Pod and communicates its binding through the API server. A successful request to change desired state consequently does not, by itself, demonstrate that new workers have been placed and started. Each stage needs its own evidence.

The component explanations here use the archived Kubernetes v1.34 static documentation; the etcd failure reference is v3.6. They explain dependencies, without establishing a tested version pairing or the behavior of a particular managed offering. Component placement varies between deployments.

References: [CP1][CP2][CP4][CP5]

Why an assigned worker can continue

On a worker node, the kubelet maintains the specified containers and the container runtime manages their execution. Already assigned work therefore has a different dependency path from a request for another Pod.

The archived Kubernetes troubleshooting page makes this distinction in its API-server crash scenario. It expects existing Pods and Services to continue unless they depend on the Kubernetes API, while management of new or changed workloads through that path is unavailable. This is a conditional expectation, not a GPU application guarantee.

An application's API dependency may become important partway through its work. For example, a fictional application could require a fresh Kubernetes API operation before beginning its next phase. Keeping its current processes alive would not satisfy that requirement. Conversely, an already executing calculation that needs no such operation may keep progressing if its worker hardware, runtime, data access and peer communications remain usable. These are conditions to identify in the offered application, not properties established by the word running.

References: [CP2][CP3]

CP-110: eight existing workers and two uncommitted requests

Consider original fictional case CP-110. Before the disruption, eight GPU worker Pods are assigned and executing. Each Pod contains one application worker process in this example. The backing etcd group has a fixed three-voter membership, which needs two votes for a majority. Two voters then become unreachable such that no communicating majority remains. The example stipulates that the worker hardware, runtimes and existing data and peer paths remain available; it implies no particular component layout.

During the first interval, supplied application observations establish that the eight-participant job completes additional iterations. The required work in this interval has no Kubernetes API dependency. These are invented premises for reasoning, not results from an incident or a test.

Two additional worker Pods are requested through the unavailable write path. The case stipulates that neither creation request is committed and neither additional Pod starts. The eight existing workers and the two requested additions must therefore remain separate in the account. There are eight executing worker Pods, not ten; no GPUs-per-Pod relationship is specified.

The iteration observations support useful progress by the existing eight-participant job during that interval. They do not establish completion of the whole job. They establish neither successful management changes nor the ability to expand the worker population. Treating them as evidence of full platform availability would extend the observation beyond the work it actually covers.

References: [CP1]

A later worker failure exposes the replacement dependency

Continue CP-110. Later, the node hosting one original worker fails, and that worker stops executing. Seven original worker processes remain alive. Stipulate that the next coordinated application step requires participation by all eight workers and that no replacement starts.

Under those premises, the seven surviving processes cannot complete that next eight-participant step. A process count can thus remain positive while the job stops making the required progress. The earlier completed iterations remain valid observations of the earlier interval; they do not establish continuation after the worker loss.

Replacing a Pod on another node is also distinct from restarting a container inside an existing Pod. Kubernetes' lifecycle documentation explains that a Pod is bound once; replacement uses a new Pod identity. The kubelet's ability to restart some containers locally does not establish that the platform can create and place that replacement elsewhere.

For the replacement path at issue, the needed new Pod and binding bring the management dependencies back into the application outcome. Spare GPU capacity alone cannot supply the missing persisted operations. Even successful replacement creation would leave placement, startup and rejoining this application to be demonstrated. The related Kueue article explains admission and placement; here the decisive issue is why an earlier observation of existing execution cannot answer the later replacement question.

References: [CP4][CP5][CP6]

Read management, worker and application evidence separately

For a management claim, identify the operation and the state change its result establishes. A request count or an accessible endpoint does not identify a completed persisted change. Conversely, evidence of one completed change would establish that operation under its observed conditions, rather than every management function throughout the incident.

For existing execution, identify which workers were observed, when the observations were made and what they show. A worker observation supports the stated execution result. A previously displayed Pod count leaves the current worker state unresolved; the account needs to make the observation's time and origin clear.

For application continuation, identify the useful unit of work and the participants and dependencies it requires. In CP-110, completed iterations answer that question for the first interval. Later, seven live processes leave the stipulated eight-participant step incomplete. Neither a process population nor an API response substitutes for that application result.

Read these three observations for the same workload and interval. An API incident alone does not establish that all GPUs stopped; healthy worker processes alone do not demonstrate management continuity or the required application result.

Make the proposal cover the work that changes during the outage

Ask the proposing platform and application teams to connect their continuity claim to the same workload and failure scenario. In CP-110, an account ending before the original worker fails omits the point where replacement becomes necessary. An account showing only seven surviving processes omits the required coordinated result.

The comparison should explain which management operations remain supported, what happens to work already executing and how the application behaves when it needs additional or replacement participants. If the application itself uses the API, include that dependency in the explanation rather than assuming it away.

This separates two offers that might both say the GPU jobs keep running. One may address only the original population while another also addresses changed membership. Their evidence needs to match those different claims. The example supplies no outage duration, recovery procedure or service-level target; those cannot be inferred from its worker counts.

Connect any separate cabling project to its receiving team

If the wider project includes separately confirmed physical connection work, Leviathan's Structured Cabling & Fiber Installation service provides a route to discuss agreed cable installation, labels and connection records. Use the related service and project inquiry with a nonconfidential equipment summary, approved endpoints and routes, materials responsibilities and the team receiving the records.

That inquiry concerns the defined physical package. Kubernetes and etcd operation, workload behavior and application acceptance remain with the appointed platform and application teams.

References: [L1]