---
title: "GPU ECC, Error Containment and Job Recovery: What Does Each Protect?"
description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20."
canonical: "https://www.leviathansystems.co/articles/gpu-ecc-correction-containment-and-job-recovery"
last-updated: "2026-10-20"
---

# GPU ECC, Error Containment and Job Recovery: What Does Each Protect?

Leviathan Systems — Technical content
Published 2026-10-20; updated 2026-10-20.

ECC can correct errors within its correction coverage. Error containment limits which applications an uncorrectable memory event affects. Page offlining and row remapping address use of degraded memory locations. Recovering a training job depends on a separate application recovery arrangement. A GPU reliability feature can protect other running jobs without restoring the job that encountered the error.

## Correction addresses the error within its coverage

ECC means error-correcting code. NVIDIA's nvidia-smi reference describes hardware correction of single-bit ECC errors and distinguishes detected but uncorrected double-bit errors. These are examples of different error outcomes; they should not be turned into a complete description of every GPU memory protection mechanism.

A corrected error is not interchangeable with an uncorrectable event. Likewise, a corrected-error count is not a count of lost training jobs. The category describes the error handling, while a job's outcome belongs to the application's execution.

The NVIDIA GPU Memory Error Management application note at the 595 reference lists ECC correction among its features. It explicitly focuses its detailed discussion on uncorrectable high bandwidth memory, or HBM, errors and places correctable HBM errors outside its scope. Use that note for the containment and memory-location mechanisms below, without deriving a correctable-error policy from it.

References: [N6](https://docs.nvidia.com/deploy/nvidia-smi/index.html), [N1](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/overview.html)

## Read the exact feature matrix before accepting a platform claim

NVIDIA's 595 Supported GPUs table uses five chip-family labels: GA100, GA10x, Ada AD10x, GH100 and GB10x. It marks row remapping for all five. It marks containment and dynamic page offlining for GA100, GH100 and GB10x, leaving those rows blank for GA10x and Ada AD10x.

An offer naming ECC or row remapping therefore leaves a separate containment question. Match the offered product and software configuration to applicable documentation.

The GB10x entry for RAS Repair for GPU Memory carries a footnote: support is not guaranteed for all Blackwell GPUs. Keep that qualification attached to RAS Repair.

These are the feature distinctions in the cited 595 reference, checked October 20, 2026.

References: [N2](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/supported-gpus.html)

## Containment limits impact on other applications

The 595 Error Containment page describes limiting the impact of uncorrectable ECC errors to applications that encounter the error. For the contained case, NVIDIA describes other workloads continuing unaffected and permits new workloads to be launched.

The buyer distinction is between an application that encounters the error and one outside that impact boundary. Keeping the second application running does not demonstrate that the first completed correctly, retained its current training state or recovered.

The same source acknowledges rare uncontained cases that can affect all workloads on the GPU. Containment needs its stated error scope. A claim about unaffected applications cannot answer the affected application's recovery question.

References: [N3](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/error-containment.html)

## Offlining avoids a page; remapping substitutes hardware memory

Dynamic page offlining changes allocation eligibility. NVIDIA describes the driver identifying an uncorrectable error's frame-buffer location and marking the containing page unusable. Once marked, that page is excluded from allocation by running and newly launched workloads. This limits subsequent use of the identified page; it does not describe reconstruction of application state already affected by the event.

Row remapping changes the hardware memory used. The 595 Row Remapping page describes spare rows in DRAM banks replacing degraded memory cells. Unlike software exclusion of a memory region, the hardware substitution does not leave software-visible holes in the address space.

Timing also differs. In this 595 description, row remapping requires a GPU reset to take effect. A pending remap and an applied remap therefore describe different states. Dynamic page exclusion and hardware substitution should not be reported as though they were one completed action.

These mechanisms address future memory use at different boundaries. Neither mechanism's name identifies the saved state from which a training application could resume.

References: [N4](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/dynamic-page-offlining.html), [N5](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/row-remapping.html)

## A training checkpoint supplies a different recovery object

PyTorch's Saving and Loading Models tutorial, last updated June 26, 2025, gives a general-checkpoint example containing model state, optimizer state, epoch and loss. It explains why resuming training needs more than model parameters alone. The application's complete state requirements remain specific to that application.

This is a different object from an ECC event, an excluded memory page or a replacement DRAM row. A checkpoint preserves identified application state. A memory-management feature describes handling of an error or a memory location. Hardware becoming available for work does not identify which training progress was saved.

The existing Training Checkpoint or Storage Snapshot guide develops the contents-and-copying comparison. Here, the necessary distinction is narrower: the hardware feature does not itself supply the application's checkpoint, compatible recovery environment or evidence of successful continuation.

References: [P1](https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html)

## Follow one fictional event through two independent jobs

Consider a fictional GPU and software configuration expressly stipulated to support error containment, dynamic page offlining and row remapping, with two independent jobs, J1 and J2, sharing the GPU. J1 encounters a contained uncorrectable HBM error. J2 does not encounter it. In this invented event, J1's execution ends while J2 continues.

The containment claim concerns the impact boundary: J2 remains outside the affected application. J1 stopping does not contradict that claim. Conversely, J2 continuing does not show that J1's interrupted work was restored. The two statements concern different jobs.

Now stipulate that the driver marks the affected page unusable and that a row remap is pending. The first statement identifies an allocation exclusion already in effect. The second identifies a hardware change awaiting application. Neither statement turns J1 back into a running job or recreates its interrupted state.

Separately, stipulate a coherent checkpoint K120, preserved outside the affected GPU memory, containing the complete state this invented application requires at completed training step 120. J1 was working on invented step 147 when the event occurred. The recovery environment is also stipulated to be compatible. K120 identifies a potential saved starting point; containment does not advance that checkpoint to step 147. No successful recovery is supplied in this example.

Remove K120 from the scenario and the same contained event can still leave J2 running, but the example no longer identifies a saved state for J1. Add K120 back and that application-state question gains an answer, while actual continuation still requires its own evidence.

This is a conceptual example. No measurements, fault injection or actual recovery occurred.

## Match the promised feature to the result you need

A proposal offering documented containment can address a requirement to limit the impact of a covered memory event on other applications. A proposal offering page offlining or row remapping describes management of degraded memory locations. These are useful hardware reliability capabilities with specific scopes.

A buyer who also needs the affected training job recovered is asking for an additional application result. That arrangement needs an identified saved state, a compatible environment, an owner for recovery and evidence supporting the promised continuation. Compare that result separately from the GPU's feature list. A feature being supported, an event being contained and a job being recovered are three different claims.

If the purchase also includes physical hardware QA before integration or reuse, Leviathan's Hardware Burn-In & Validation service is a relevant next step. Its service and project-inquiry links below support a discussion of identified equipment, the hardware question and an agreed QA scope. Application-state capture, software recovery and workload commissioning remain with the appointed application and platform providers.

References: [L1](https://www.leviathansystems.co/services/hardware-burn-in-validation)

## Primary sources

- [N1 — NVIDIA GPU Memory Error Management 595: Overview](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/overview.html). Checked 2026-10-20. Feature overview; the detailed scope is uncorrectable HBM errors. Correctable HBM errors are explicitly outside the application note's scope.
- [N2 — NVIDIA GPU Memory Error Management 595: Supported GPUs](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/supported-gpus.html). Checked 2026-10-20. Chip-family feature rows; the Blackwell support footnote accompanies the RAS Repair entry.
- [N3 — NVIDIA GPU Memory Error Management 595: Error Containment](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/error-containment.html). Checked 2026-10-20. Impact boundary for contained uncorrectable errors, including rare uncontained cases.
- [N4 — NVIDIA GPU Memory Error Management 595: Dynamic Page Offlining](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/dynamic-page-offlining.html). Checked 2026-10-20. Driver exclusion of an identified page from subsequent allocation; this is distinct from restoring application state.
- [N5 — NVIDIA GPU Memory Error Management 595: Row Remapping](https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/latest/595/row-remapping.html). Checked 2026-10-20. Hardware spare-row substitution, absence of software-visible address holes and the distinction between pending and applied remapping.
- [N6 — NVIDIA System Management Interface: ECC Errors](https://docs.nvidia.com/deploy/nvidia-smi/index.html). Checked 2026-10-20. ECC definitions supply a bounded correction example separate from the 595 application note's uncorrectable HBM discussion.
- [P1 — PyTorch: Saving and Loading Models](https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html). Checked 2026-10-20. Tutorial updated June 26, 2025. Its general-checkpoint example distinguishes saved training state from hardware error handling; it is not a complete state specification for every application.
- [L1 — Leviathan Hardware Burn-In & Validation](https://www.leviathansystems.co/services/hardware-burn-in-validation). Checked 2026-10-20. Hardware QA follows an agreed equipment-specific plan and confirmed resources. Software recovery and full workload commissioning are separate responsibilities.

## Related Leviathan resources

- [Training checkpoint or storage snapshot: identify the saved application state](/articles/gpu-training-checkpoint-versus-storage-snapshot)
- [MIG, time-sliced vGPU and whole-GPU allocation boundaries](/articles/gpu-sharing-mig-time-sliced-vgpu-buying)
- [Compare published GPU capacity with assigned and free memory](/articles/data-center-gpu-memory-capacity-comparison)
- [Discuss a physical hardware burn-in and QA scope](/services/hardware-burn-in-validation)
- [Project inquiry for physical hardware QA](/contact?service=hardware-burn-in-validation&from=%2Farticles%2Fgpu-ecc-correction-containment-and-job-recovery)

## Have physical hardware QA to scope?

Share a nonconfidential equipment summary, the hardware question and the intended integration or reuse handoff so we can discuss QA scope, resources and availability.

[Discuss the scope](/contact?service=hardware-burn-in-validation&from=/articles/gpu-ecc-correction-containment-and-job-recovery)