Buyer's Guide
The Same Class Labels Can Produce Different Weighted Mean Losses
A change from class indices to equivalent one-hot probability targets can change PyTorch 2.14 CrossEntropyLoss with mean reduction. Without label smoothing, both representations give the same weighted loss for each eligible observation. The index-target mean divides their sum by the sum of target-class weights; the probability-target mean divides by the number of observations. Preserving the class labels therefore does not, by itself, preserve the scalar loss used in a training/evaluation handoff.
01
A label conversion can also change the loss definition
A training team may identify each target with an integer class index, while an evaluation team represents that same class with a one-hot row. In a two-class task, index 0 and the row (1, 0) identify the same class. That semantic agreement leaves another question: which mean does the receiving calculation return?
PyTorch 2.14 documents two target modes for CrossEntropyLoss. With a batch of two-class logits, index targets have one long-integer entry per observation; probability targets are floating-point rows with the same shape as the logits. A one-hot row is a valid probability distribution, even though it assigns all its mass to one class.
This comparison uses finite logits, valid targets, positive class weights, label_smoothing equal to zero and mean reduction without deprecated size_average or reduce overrides. It isolates the documented loss mathematics. The worked case uses exact real-number arithmetic and natural logarithms, with no model execution or measured training result.
The raw model inputs to this loss are logits: scores before class normalization. In the comparison below, the logits, class order, class weights and eligible observations remain fixed. Only the target representation changes.
References: [CE1]
02
Follow the weighted terms before interpreting the mean
Let p(i,c) be the class probability obtained from observation i's logits: exp(z(i,c)) divided by the sum of exp(z(i,k)) over its classes k. For target class y(i), ordinary negative log-likelihood is a(i) = −ln p(i,y(i)). A class weight w(y(i)) makes that observation's weighted term l(i) = w(y(i)) × a(i).
For eligible index targets, let S be the sum of these weighted terms and W the sum of their target-class weights. The documented index-target mean is S / W. This normalizes the weighted sum by the weight assigned to its participating targets.
For probability targets t(i,c), each observation instead contributes l(i) = −sum over classes c of w(c) × t(i,c) × ln p(i,c). If t(i,c) is the corresponding one-hot row, only its target-class term remains. The weighted term is then exactly the same as for that observation's index target.
However, the documented probability-target mean is S / N, where N counts observations in this batch. It does not divide by W. Matching all the weighted terms establishes a common numerator; the mean still needs its denominator. Here one observation supplies one loss element. In the documented higher-dimensional case, the loss population also spans the additional spatial dimensions, so its count cannot be inferred from batch size alone.
References: [CE1]
03
CE-29: derive the same two weighted loss terms
Fictional case CE-29 contains two observations and two classes, ordered 0 then 1. Observation A has logits (0, 0) and target class 0. Observation B has logits (ln 3, 0) and target class 1. Class weights are (1, 3), so class 0 has weight 1 and class 1 has weight 3. Both observations are eligible.
For A, exponentiating the logits gives (1, 1). Its target probability is 1 / (1 + 1) = 1/2. Ordinary NLL is −ln(1/2) = ln 2. Applying class 0's weight gives l(A) = 1 × ln 2 = ln 2.
For B, exponentiating the logits gives (3, 1). Because its target is class 1, its target probability is 1 / (3 + 1) = 1/4. Ordinary NLL is −ln(1/4) = ln 4 = 2 ln 2. Applying class 1's weight gives l(B) = 3 × 2 ln 2 = 6 ln 2.
The index targets are (0, 1). Their corresponding one-hot probability rows are (1, 0) for A and (0, 1) for B. In the probability-target formula, each row selects exactly the same weighted class term derived above. Both modes therefore have unreduced terms (ln 2, 6 ln 2), and both have S = 7 ln 2.
References: [CE1]
04
The two documented means differ by a factor of two
For the index-target mean, W = 1 + 3 = 4. CE-29 therefore gives L_index = (7 ln 2) / 4 = (7/4) ln 2, approximately 1.213008.
For the one-hot probability-target mean, N = 2. The same weighted sum gives L_probability = (7 ln 2) / 2 = (7/2) ln 2, approximately 2.426015. This mean is exactly twice the index-target mean.
The target probabilities remain 1/2 and 1/4. No prediction has deteriorated during this representation change. The scalar increases because the documented reduction divides the same numerator by two instead of four. A report describing the increase as evidence of worse predictions would misread this case.
For corresponding one-hot targets over the same eligible population, the algebraic relationship is L_probability = (W / N) × L_index. CE-29 has W / N = 2. That factor belongs to this population and these weights; it is not a universal correction for every batch. A different class mixture can change W / N.
With all class weights equal to one, W = N and the two means coincide under these assumptions. A numerical match in that special case does not establish that the two modes use the same reduction rule. Agreement on an unweighted example can conceal the difference that CE-29 exposes.
References: [CE1]
05
An ignored index has no automatic probability-row equivalent
Extend CE-29 with a third observation having finite two-class logits and target index −100, explicitly designated as ignore_index. The index-target definition excludes that observation. It adds zero to the loss numerator and contributes no target-class weight to the mean denominator. The two eligible observations still give S = 7 ln 2 and W = 4, so their index-target mean remains (7/4) ln 2.
The ignored value identifies an exclusion; it does not introduce a third target class. PyTorch 2.14 states that ignore_index applies only to class-index targets. A representation handoff cannot assume that this exclusion survives merely because the same observations occupy the input batch.
An all-zero row is not a valid probability distribution: its entries sum to zero. It cannot be described as a valid one-hot encoding of the ignored observation. The reference also warns that probability validity is not automatically enforced, so a returned scalar alone cannot establish valid labels or preserved eligibility.
For a handoff involving excluded observations, require an explicit account of the probability-target loss population and its denominator. The documented index exclusion does not settle that separate definition. This extension makes no assumption about how an invalid probability row would be handled.
References: [CE1]
06
Common weight scaling exposes the normalization difference
Return to the original two eligible observations. Double both class weights from (1, 3) to (2, 6), leaving every logit and target unchanged. Each weighted term doubles, so S becomes 14 ln 2.
For index targets, W also doubles, from 4 to 8. The mean becomes (14 ln 2) / 8 = (7/4) ln 2, exactly its earlier value. More generally, multiplying all class weights by the same positive factor cancels between S and W in this index-target mean.
For probability targets, N remains 2. The mean becomes (14 ln 2) / 2 = 7 ln 2, twice its previous value of (7/2) ln 2. Common scaling remains in the numerator because the observation-count denominator does not scale with the weights.
Thus the relative class-weight ratio alone is insufficient to describe the probability-target mean: its absolute scale also matters. These are normalization properties of the stated losses. They supply no conclusion about identical optimizer updates, training trajectories or final model quality.
References: [CE1]
07
Specify which mean the receiving report must preserve
A useful handoff identifies the evaluated model and logits, class order, target representation and data type, eligible observations, class weights including their scale, smoothing, and required reduction. For a weighted comparison, request the per-observation terms, their sum S and the actual denominator as well as the scalar. The word mean alone leaves the decisive distinction unresolved.
If the required objective is the index-target weighted mean, its defining quantity here is S / W. A receiving report that returns S / N has changed the reduction, even when its one-hot labels preserve every target class. If the parties intend that change, identify the new objective explicitly. If they intend preservation, require evidence for the agreed numerator, population and denominator before comparing the reported values.
CE-29 illustrates the positive finding: both reports can correctly describe the same class predictions while reporting different documented mean losses. Once their reduction bases are identified, the difference is explainable without attributing it to model quality. A scalar without the weights and denominator is insufficient to establish that explanation for another report.
Ordinary equally weighted NLL is a separate quantity. For CE-29 it is (ln 2 + 2 ln 2) / 2 = (3/2) ln 2. Neither class-weighted scalar above should be exponentiated and labeled ordinary per-token perplexity. The related perplexity guide covers that evaluation basis; the gradient-accumulation guide covers weighting across microbatches. The decision here is whether a target-representation change preserves the intended weighted mean.
References: [CE1]
08
If selected equipment also needs physical installation
For an independently needed equipment installation, Leviathan's rack-and-stack service covers agreed mounting, physical IT cabling, labels and installation records. Use the related service and project inquiry to share a nonconfidential equipment list, site and released rack/cable package for a discussion of physical scope and availability. Loss implementation, model evaluation and software acceptance remain with the appointed software team.
References: [L1]
09Sources
Primary sources
- [CE1]PyTorch 2.14 — CrossEntropyLossChecked 2026-10-21. Defines both target modes, weighted unreduced terms and their different mean reductions, target shapes and types, index-only ignore behavior, and probability validity. The comparison fixes label smoothing at zero and excludes deprecated reduction overrides.
- [L1]Leviathan Systems — Rack & Stack / ColocationChecked 2026-10-21. Scope for agreed equipment mounting, physical IT cabling, labels and installation records, relevant to an independently needed physical installation.
10Related
Related Leviathan resources
Contact
Does selected equipment need physical installation?
Share a nonconfidential equipment list, site and released rack/cable package to discuss the physical installation scope and availability.