---
title: "Activation Checkpointing: Recompute Intermediates, Preserve Recovery Separately"
description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20."
canonical: "https://www.leviathansystems.co/articles/activation-checkpointing-recompute-versus-training-recovery"
last-updated: "2026-10-20"
---

# Activation Checkpointing: Recompute Intermediates, Preserve Recovery Separately

Leviathan Systems — Technical content
Published 2026-10-20; updated 2026-10-20.

Activation checkpointing reduces selected forward intermediates retained for backward computation by recreating them when needed. A recovery checkpoint preserves application state for later continuation. Compare these as separate capabilities: one changes memory and computation within a training step; the other supplies a durable restart artifact. A complete training proposal may include both.

## Why forward values remain live for backward

The forward pass computes model outputs and a loss. The backward pass works through the recorded operations in reverse to calculate gradients: derivatives that describe how the loss changes with the trained parameters. Some of those calculations need values produced or consumed earlier in forward.

PyTorch 2.14's autograd note gives a simple example: computing the derivative of x squared needs the input x. Knowing the final loss and the model's weights does not generally supply every value needed by the intervening derivative calculations. Autograd saves tensors as required by the operations.

This creates a lifetime extending beyond the immediate forward operation. A value used early in forward may remain live until its corresponding backward work occurs much later. The PyTorch Foundation's activation-memory explanation illustrates how such retained values accumulate through forward and can be released as backward consumes them. This is an account of saved activations, rather than a rule locating every workload's total memory peak.

References: [P2](https://docs.pytorch.org/docs/2.14/notes/autograd.html), [P3](https://pytorch.org/blog/activation-checkpointing-techniques/)

## Replace selected retention with reconstruction

Activation checkpointing changes how those later-needed values become available. Instead of keeping selected intermediates across the interval between forward and backward, the application arranges to compute them again. The initial forward work still happens; reconstruction adds computation later.

The PyTorch 2.14 checkpoint reference, checked October 20, 2026, describes retaining passed tensor inputs and reconstructing unsaved intermediates when backward needs them. This gives the arrangement a starting point for recomputation. It does not mean the region has no memory requirement between passes.

The useful question is which values stop being retained, which remain available, and when the recreated values coexist with backward work. Retained inputs, outputs needed elsewhere and other live allocations still matter. A proposal describing fewer saved activations has identified a memory mechanism; it has not yet quantified the complete device requirement.

Here, saved means kept available to the running computation. PyTorch's separate saving-and-loading tutorial describes serialized application state on disk. Activation checkpointing does not itself supply that durable training restart package.

References: [P1](https://docs.pytorch.org/docs/2.14/checkpoint.html), [P3](https://pytorch.org/blog/activation-checkpointing-techniques/), [P5](https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html)

## AC-221: six retained groups become two at one boundary

Fictional example AC-221 names boundary F as the instant after the initial forward pass has completed and before backward begins. Its baseline retains six separately identified activation groups at F: A1, A2, A3, A4, A5 and A6. Each occupies 1 GiB of distinct, nonoverlapping storage within this limited population. These are stipulated accounting objects, not measured PyTorch allocations.

A proposed recomputation arrangement retains A1 and A6 at F and plans to recreate A2, A3, A4 and A5 later for gradient computation. The comparison concerns those same six groups at the same named boundary. Any other retained inputs or live objects sit outside this limited account.

The baseline population is 6 x 1 GiB = 6 GiB. The proposed retained population is 2 x 1 GiB = 2 GiB. Its retained-state difference at F is therefore 6 - 2 = 4 GiB. This is the complete numerical conclusion the example supports.

AC-221 supplies no sizes or timing for the recomputed working set, overlapping backward work, weights, gradients, optimizer state or other allocations. It also supplies no allocator or whole-device readings. Nothing establishes the total usage at F, or whether F is the busiest instant in either arrangement.

Subtracting 4 GiB from a previously quoted device peak would silently assume that the peak occurred at F and that everything else contributing to it stayed unchanged. Those assumptions are absent. A later reconstruction could coexist with allocations that were smaller or absent at F. The proposal has reduced the named retained population; total peak memory and GPU fit remain unresolved.

## The implementation changes what is recomputed

PyTorch 2.14 distinguishes two implementations. Reentrant checkpointing reruns the whole wrapped function during backward. Non-reentrant checkpointing can stop once required intermediates have been recreated; early stopping is enabled by default. An unspecified checkpointing label therefore leaves the replayed work unclear.

The reference also warns that changed behavior between forward and recomputation, including changed global state, can cause errors or silently incorrect gradients. Random-state preservation has device-coverage limits; it cannot guarantee equivalence for arbitrary functions. The application owner needs evidence for the actual implementation and execution conditions.

References: [P1](https://docs.pytorch.org/docs/2.14/checkpoint.html)

## Compare a restart package with a recomputation offer

Consider two fictional offers for the same intended training outcome. Offer R supplies a periodic durable restart package containing identified model state, optimizer state and progress information, with the continuation environment described. It says nothing about activation-memory treatment. Offer C describes activation recomputation during the training step but supplies no durable continuation artifact.

R addresses a saved-state requirement. PyTorch's general-checkpoint tutorial illustrates why a resumable training package can include optimizer and progress information alongside model state. The related Training Checkpoint or Storage Snapshot article examines those contents and their recovery meaning. Possessing R's package does not demonstrate a successful restore, and its periodic creation does not establish reduced activation retention.

C addresses the running computation's retention-versus-reconstruction choice. Its explanation may support an activation-memory evaluation such as AC-221, but it identifies no saved application state from which an interrupted job would continue. The ability to recreate an intermediate during backward does not establish a recovery point after the running process has been lost.

A buyer requiring both a workable memory configuration and defined continuation should compare both capabilities. Neither offer's supplied feature fills the other's omission. A proposal can include activation recomputation and a separate durable restart package, with evidence appropriate to each. Calling either feature checkpointing without describing its object and lifetime obscures what is actually being offered.

References: [P5](https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html)

## Judge the complete step and the whole job

A retained-state comparison becomes useful capacity evidence when it is connected to the allocation timeline for the proposed workload. PyTorch's training-memory tutorial separates parameters, saved activations, gradients, optimizer state and temporary tensors. Its illustrated workload peaks during optimizer work, showing why an activation boundary need not identify the overall bottleneck. The tutorial's sizes and ratios are specific to its example.

For an AC-221 proposal, the missing result is a complete memory account through representative forward, backward, recomputation and optimizer work on each participating device. Initialization and the first creation of optimizer state also need coverage where they change the requirement. The measurement should identify its scope and units so the reader can distinguish a tensor population from the broader device account.

Keep the comparison attached to the same model revision, trained parameters, precision, optimizer, sequence-length requirements, per-device microbatch and effective batch. The hardware allocation, software version and relevant distributed layout belong with the result. Reducing sequence length or changing batch requirements to obtain a fit changes the purchasing comparison.

Recomputation also creates a cost question. At an unchanged billed allocation and rate, a longer job costs more; a memory reduction may instead allow a different viable allocation. Neither possibility supplies AC-221's missing timings or prices. Four GiB fewer retained at F cannot predict the time spent recreating values, the complete step duration, or the time to the required training outcome.

Compare complete-step timing and whole-job evidence against that common outcome, including any restart-package work included in the offer. Extra computation is a reason to examine elapsed time and cost, rather than a fixed percentage penalty to apply from the retained-memory difference.

References: [P3](https://pytorch.org/blog/activation-checkpointing-techniques/), [P4](https://docs.pytorch.org/tutorials/intermediate/optimizer_step_in_backward_tutorial.html)

## Keep the neighboring memory decisions with their owners

The related Replicas or Shards article explains how training state is distributed. Gradient Accumulation explains how microbatches contribute before an optimizer update. PyTorch Allocated and Reserved Memory explains overlapping tensor and allocator accounts. Use those guides for their respective comparisons; none of their counts can be substituted for AC-221's activation population.

A complete proposal may combine these mechanisms. Its evidence needs to describe the combination actually evaluated, so a result obtained with sharding, changed batch grouping and recomputation is not attributed to activation checkpointing alone.

## Carry selected equipment into a separate installation scope

The training and platform owners should resolve the software configuration and workload evidence. This article is infrastructure education; Leviathan offers no training implementation, tuning, recovery service or model-fit guarantee through it.

If the selected equipment separately needs physical installation, Leviathan's rack-and-stack service and its project inquiry below are relevant. The service covers agreed equipment mounting, physical IT cabling, labels and installation records.

Share a nonconfidential equipment list, site, released rack and endpoint plans, requested installation work and receiving team to discuss scope and availability. This connects an actual selected-equipment requirement to a physical handoff.

References: [L1](https://www.leviathansystems.co/services/rack-and-stack)

## Primary sources

- [P1 — PyTorch 2.14: torch.utils.checkpoint](https://docs.pytorch.org/docs/2.14/checkpoint.html). Checked 2026-10-20. Updated July 27, 2026. Versioned mechanism, implementation differences and functional-equivalence limits; no universal memory saving or runtime result.
- [P2 — PyTorch 2.14: Autograd mechanics](https://docs.pytorch.org/docs/2.14/notes/autograd.html). Checked 2026-10-20. Updated August 3, 2026. Reverse differentiation and saved tensors explain why backward needs earlier values.
- [P3 — PyTorch Foundation: Current and New Activation Checkpointing Techniques in PyTorch](https://pytorch.org/blog/activation-checkpointing-techniques/). Checked 2026-10-20. Conceptual activation-lifetime and memory/computation explanation. The versioned 2.14 reference governs the implementation distinctions used here.
- [P4 — PyTorch: How to save memory by fusing the optimizer step into the backward pass](https://docs.pytorch.org/tutorials/intermediate/optimizer_step_in_backward_tutorial.html). Checked 2026-10-20. Updated January 16, 2024. Memory populations and the example's optimizer-stage peak inform complete-step evaluation; no adoption of its procedure or size ratios.
- [P5 — PyTorch: Saving and Loading Models](https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html). Checked 2026-10-20. Updated June 26, 2025. Serialization and the general-checkpoint example distinguish durable training state from live activation retention.
- [L1 — Leviathan Systems: Rack-and-stack installation](https://www.leviathansystems.co/services/rack-and-stack). Checked 2026-10-20. Agreed mounting, IT cabling, labels and physical records for selected equipment; software commissioning is outside standard installation scope.

## Related Leviathan resources

- [Training Checkpoint or Storage Snapshot: assess the recovery artifact](/articles/gpu-training-checkpoint-versus-storage-snapshot)
- [Replicas or Shards: understand distributed training-state ownership](/articles/gpu-training-ddp-replicas-versus-fsdp-shards)
- [Gradient Accumulation: distinguish microbatches and updates](/articles/gpu-training-microbatch-gradient-accumulation)
- [PyTorch Allocated and Reserved Memory: reconcile the accounting](/articles/pytorch-tensor-allocated-versus-allocator-reserved-memory)
- [Rack-and-stack installation for selected equipment](/services/rack-and-stack)
- [Discuss a separate physical installation requirement](/contact?service=rack-and-stack&from=%2Farticles%2Factivation-checkpointing-recompute-versus-training-recovery)

## Selected the equipment and need physical installation?

Share the selected equipment, site, released rack and cable plans, requested physical work and receiving team to discuss installation scope and availability.

[Discuss the scope](/contact?service=rack-and-stack&from=/articles/activation-checkpointing-recompute-versus-training-recovery)