---
title: "Adam and AdamW: The Same Weight Decay Can Mean Different Updates"
description: "Leviathan Systems — Technical content Published 2026-10-21; updated 2026-10-21."
canonical: "https://www.leviathansystems.co/articles/ai-adam-weight-decay-coupled-versus-decoupled-update"
last-updated: "2026-10-21"
---

# Adam and AdamW: The Same Weight Decay Can Mean Different Updates

Leviathan Systems — Technical content
Published 2026-10-21; updated 2026-10-21.

A weight_decay value needs its update rule attached. In the coupled Adam branch examined here, decay contributes to the gradient used by the adaptive moments. In the decoupled branch, it shrinks the parameter separately. The same coefficient can therefore produce different parameters and moment states from the same supplied starting point. For a claimed matching training continuation, establish the effective branch and corresponding state, rather than treating a matching field value as algorithmic equivalence.

## Locate decay before comparing its coefficient

An AI training handoff can name Adam, a learning rate and a weight_decay value while leaving the actual parameter update unresolved. The coefficient tells the reader the size of a term only after its role in the algorithm is known. Adding it to a gradient and applying it directly to a parameter are different operations under adaptive normalization.

This explanation uses the historical PyTorch 2.9 release and its tagged v2.9.0 source. Adam accepts decoupled_weight_decay=False or True. The False branch couples decay to the gradient; the True branch is documented as equivalent to AdamW for this distinction. AdamW's constructor selects True. Consequently, a comparison based only on the names Adam and AdamW could misidentify two implementations that use the same decoupled branch.

The useful reader question is where decay enters the offered update, and what state that operation carries into the next update. The following dense, real-valued scalar example isolates that question without running a model.

References: [P1](https://docs.pytorch.org/docs/2.9/generated/torch.optim.Adam.html), [P2](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adam.py), [P3](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adamw.py)

## A gradient contribution and a direct shrinkage follow different paths

Let theta denote a parameter, g the supplied gradient of the task objective, eta the learning rate and lambda the decay coefficient. In the coupled branch, the gradient used for the update becomes g + lambda × theta. For a scalar, that added term is also the derivative of a separately defined quadratic penalty (lambda / 2) × theta². It is important to distinguish that contribution from the original task gradient.

Adam then updates a first-moment estimate m from the gradient and a second-moment estimate v from its square. The beta coefficients control how previous moment values carry forward. Bias correction accounts for their zero initialization. The adaptive parameter change uses the corrected first moment divided by the square root of the corrected second moment plus epsilon. A decay contribution entering the gradient therefore also enters these moment calculations.

In the decoupled branch, the direct decay operation instead changes theta to theta × (1 − eta × lambda). The supplied task gradient still updates the moments, but this decay contribution is absent from them. The adaptive task-gradient update follows the direct shrinkage. The same lambda label now participates in a separate parameter operation.

These statements describe the named implementation's branches. They do not assert that an additional penalty already included in the supplied task gradient disappears when a decoupled flag is selected. The objective supplying g remains part of the comparison.

References: [P1](https://docs.pytorch.org/docs/2.9/generated/torch.optim.Adam.html), [P2](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adam.py), [P3](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adamw.py)

## DECAY1: hold the parameter and supplied task gradient fixed

DECAY1 is an original fictional first-update comparison. There is one dense real scalar parameter, initially theta = 2. Its task gradient has already been supplied as g = 1 at that starting point. Use eta = 0.1, lambda = 0.01, beta1 = 0.9 and beta2 = 0.999. Both moments start at zero, and the update counter advances to one. Compare ordinary minimization with AMSGrad disabled. The gradient is present and nonzero.

Set epsilon to zero solely to make this exact real-number calculation transparent. This is a nondefault algebraic premise, not a recommended training setting; the named release's default is 1e-8. All denominators below are positive. No optimizer, model or floating-point implementation was executed.

The two branches receive the same task gradient. Direct shrinkage in the decoupled branch does not trigger a new task-gradient evaluation in this example. The stipulated g remains 1 throughout that update. This keeps the source of the difference visible.

References: [P1](https://docs.pytorch.org/docs/2.9/generated/torch.optim.Adam.html), [P2](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adam.py)

## Follow the moments to 1.9 and 1.898

In the coupled branch, lambda × theta is 0.01 × 2 = 0.02. The gradient entering the moments is therefore 1 + 0.02 = 1.02. The first moment becomes 0.9 × 0 + 0.1 × 1.02 = 0.102. The second becomes 0.999 × 0 + 0.001 × 1.02² = 0.0010404.

At update one, the bias-correction divisors are 1 − 0.9 = 0.1 and 1 − 0.999 = 0.001. The corrected moments are 1.02 and 1.0404. Since the square root of 1.0404 is 1.02, the adaptive subtraction is 0.1 × 1.02 / 1.02 = 0.1. The parameter becomes 2 − 0.1 = 1.9. There is no separate direct-decay subtraction in this branch.

In the decoupled branch, direct shrinkage first gives 2 × (1 − 0.1 × 0.01) = 1.998. The moments use g = 1, so they become 0.1 and 0.001. Applying the same first-update bias corrections gives corrected moments 1 and 1. The adaptive subtraction is again 0.1, giving 1.998 − 0.1 = 1.898.

The final parameters differ by 0.002. The stored moments differ too, even though both branches received the same original task gradient and nominal decay coefficient. Their equal adaptive subtraction here follows from the chosen positive gradients, zero epsilon and first-update state. It is not a formula for arbitrary later updates.

References: [P1](https://docs.pytorch.org/docs/2.9/generated/torch.optim.Adam.html), [P2](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adam.py), [P3](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adamw.py)

DECAY1: two branches from the same stipulated starting state
| Quantity | Coupled branch | Decoupled branch | Source |
| --- | --- | --- | --- |
| Task gradient supplied | 1 | 1 | Planning analysis |
| Gradient entering moments | 1.02 | 1 | Planning analysis |
| Stored first moment m | 0.102 | 0.1 | Planning analysis |
| Stored second moment v | 0.0010404 | 0.001 | Planning analysis |
| Bias-corrected first moment | 1.02 | 1 | Planning analysis |
| Bias-corrected second moment | 1.0404 | 1 | Planning analysis |
| Separate direct shrinkage | None | 2 to 1.998 | Planning analysis |
| Parameter after complete update | 1.9 | 1.898 | Planning analysis |
Original hypothetical inputs and exact real-number derivation. Values are not observed training results, recommended settings or bitwise predictions for a runtime.

## Use the difference to interpret a training-state return

Suppose two fictional handoffs claim to reproduce DECAY1's next update. One identifies the coupled branch and returns parameter 1.9 with moments 0.102 and 0.0010404. The other identifies the decoupled branch and returns 1.898 with moments 0.1 and 0.001. Each return matches its stated algorithm under the supplied premises. They cannot both describe the same next parameter and moment state merely because each lists weight_decay = 0.01.

For an actual continuation claim, have the appointed training provider connect the intended update rule to the implementation and version, effective decay flag, parameter-group assignment and corresponding optimizer state. A learning-rate value also needs its effective update point identified. DECAY1 makes the reason concrete: the coefficient's location changes the state that the next update receives.

The checkpoint guide addresses whether the matching model, optimizer and progress objects are retained at all. This article adds the operation those optimizer objects belong to. A complete checkpoint package and an identified update rule answer complementary parts of the continuation question.

Even a substantiated state-and-rule match is not a promise of an identical future training trajectory across environments. Nor does DECAY1 show which branch gives better generalization, convergence or useful work per GPU-hour. Those outcomes need their own workload and evaluation evidence. The smaller parameter in this one-step case is an arithmetic result, not a quality ranking.

References: [P1](https://docs.pytorch.org/docs/2.9/generated/torch.optim.Adam.html), [P2](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adam.py), [P3](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adamw.py)

## Keep update semantics separate from batch and hardware claims

Gradient accumulation determines which gradient contributions reach an optimizer-update boundary. It does not, by its example count alone, identify how the optimizer then applies decay. Use the related accumulation guide for that population and loss-weighting question.

Likewise, an optimizer name or coefficient supplies no GPU-memory fit, throughput or hardware-support result. Comparing systems still requires evidence for the selected model, complete training activity and required outcome. The related performance guide addresses that broader decision.

The conclusion here is specific: before accepting two training descriptions as equivalent, establish whether the same decay operation acts on the same parameter and corresponding state at the compared update. A matching number in the weight_decay field leaves that relationship unresolved.

## Connect a separate equipment project to physical installation

If the chosen training environment separately needs equipment mounting and physical IT connections, Leviathan's Rack-and-Stack & Colocation Installation service provides a route to discuss agreed mounting, cabling, labels and installation records. Use the related project inquiry with a nonconfidential equipment, rack, site and connection outline to discuss scope and availability.

That inquiry concerns physical installation. Training implementation, optimizer selection and workload acceptance remain with the appointed software and application providers.

References: [L1](https://www.leviathansystems.co/services/rack-and-stack)

## Primary sources

- [P1 — PyTorch 2.9: Adam](https://docs.pytorch.org/docs/2.9/generated/torch.optim.Adam.html). Checked 2026-10-21. Historical named release. The algorithm and argument definitions describe coupled decay, moment bias correction and the explicit decoupled_weight_decay alternative. DECAY1 uses a stated nondefault epsilon.
- [P2 — PyTorch v2.9.0: Adam decay branches and moment updates](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adam.py). Checked 2026-10-21. Tagged dense real-parameter path: direct parameter shrinkage or gradient contribution precedes moment and bias-correction updates. Supports the finite algorithm comparison, without an execution or performance claim.
- [P3 — PyTorch v2.9.0: AdamW](https://raw.githubusercontent.com/pytorch/pytorch/v2.9.0/torch/optim/adamw.py). Checked 2026-10-21. The constructor selects the decoupled branch and its state-loading behavior maintains that setting. The algorithm description keeps decay outside the task-gradient moments.
- [L1 — Leviathan Systems: Rack-and-Stack & Colocation Installation](https://www.leviathansystems.co/services/rack-and-stack). Checked 2026-10-21. Agreed equipment mounting, physical IT cabling, labels and installation records provide a next step for a separately selected physical project.

## Related Leviathan resources

- [Gradient accumulation and the optimizer-update boundary](/articles/gpu-training-microbatch-gradient-accumulation)
- [Training checkpoints, optimizer state and storage snapshots](/articles/gpu-training-checkpoint-versus-storage-snapshot)
- [Compare GPU performance for the intended workload](/articles/which-gpu-is-fastest-workload-benchmarks)
- [Rack-and-Stack & Colocation Installation](/services/rack-and-stack)
- [Discuss a separate physical equipment installation](/contact?service=rack-and-stack&from=%2Farticles%2Fai-adam-weight-decay-coupled-versus-decoupled-update)

## Does the selected equipment need physical installation?

Share a nonconfidential equipment, rack, site and physical connection outline to discuss installation scope, requirements and availability.

[Discuss the scope](/contact?service=rack-and-stack&from=/articles/ai-adam-weight-decay-coupled-versus-decoupled-update)
