---
title: "Perplexity Scores Need the Tokenizer and the Scored Token Population"
description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20."
canonical: "https://www.leviathansystems.co/articles/language-model-perplexity-tokenization-evaluation-basis"
last-updated: "2026-10-20"
---

# Perplexity Scores Need the Tokenizer and the Scored Token Population

Leviathan Systems — Technical content
Published 2026-10-20; updated 2026-10-20.

A lower language-model perplexity is useful comparison evidence only after establishing what each report scored and how it normalized the loss. Shared dataset text does not establish a shared token unit, target population or conditioning context. Before using PPL to support a serving or compute choice, reconcile those evaluation boundaries and assess application quality and performance with their own evidence.

## Follow the probability assigned to the reference targets

For a causal language model, evaluation asks how much probability the model assigns to each next token in a supplied reference sequence. The preceding reference tokens provide the history. The evaluator follows that reference history even when the model would have selected another continuation during generation.

Let S be the set of target positions actually scored, N its positive token count, and c_i the permitted preceding context for target token x_i. Define total negative log-likelihood L = sum over i in S of −ln p(x_i | c_i). The natural logarithm makes L a quantity in nats. Mean target NLL is L / N, and PPL = exp(L / N). This article uses ordinary, equally weighted reference targets without label smoothing.

The sum accumulates the model's surprise at the reference tokens. Dividing by N expresses that surprise per scored token; exponentiation gives perplexity. Equivalently, PPL is the reciprocal of the geometric mean of those target probabilities. Lower PPL on a matched basis therefore means a higher geometric mean probability for the scored references.

Hugging Face's Transformers v4.57.1 perplexity guide gives this causal likelihood definition. It does not supply the same evaluation definition for masked language models such as BERT. A PPL number is not a percentage of correct answers.

References: [PP1](https://huggingface.co/docs/transformers/v4.57.1/en/perplexity), [CE1](https://docs.pytorch.org/docs/2.9/generated/torch.nn.CrossEntropyLoss.html)

## PPL-225: equal string likelihood, different token denominators

Original hypothetical example PPL-225 evaluates exactly the complete string bright sky under two invented deterministic tokenizations. A splits it into two pieces: bright and a second piece consisting of a space followed by sky. B splits it into four pieces: br, ight, a space followed by s, and ky. Concatenating either sequence reconstructs exactly the same string, including its space.

Both targets in A and all four targets in B are scored. Neither evaluation adds a beginning-of-sequence or end-of-sequence target, and neither omits the first text token. Both use the stipulated empty-text start condition. These are mathematical assumptions about imaginary systems, with no claim about a real tokenizer or model.

Stipulate total sequence NLL L = 4 nats for each encoding. A compatible construction assigns probability exp(−2) to each of A's two reference targets at its respective history, and exp(−1) to each of B's four. In both cases the product of conditional reference probabilities is exp(−4), approximately 0.0183156. The two encodings of the complete string therefore have equal stipulated products of reference-target probabilities.

For A, mean target NLL is 4 / 2 = 2 nats per token, so PPL = exp(2), approximately 7.389. For B, it is 4 / 4 = 1 nat per token, so PPL = exp(1), approximately 2.718. B has lower per-token perplexity despite the equal sequence likelihood.

The difference comes entirely from the stipulated denominator. PPL-225 establishes no general preference for longer tokenizations. Actual models can assign different total likelihoods as well as use different token counts. It shows why a shared evaluated string cannot, by itself, make two per-token PPL values a common quality scale.

## Identify which input tokens also became scored targets

A token can provide context without being a scored target. Consider a hypothetical evaluation input with six preceding text tokens and two following reference tokens. If the evaluation scores only the latter two, N = 2. The six earlier tokens can affect their probabilities while contributing no target loss of their own. Padding slots likewise do not become targets merely because they occupy input space.

The Transformers v4.57.1 GPT-2 reference distinguishes an attention mask from language-model labels. It documents an internal label shift and ignored label values. These are separate concerns: which positions can be attended to, which predictions align with which reference tokens, and which aligned targets contribute to loss.

A report must therefore establish the actual scored population after alignment and exclusions. A raw input length, a nonpadding-input count or a count of labels before shifting may describe a different population. Matching one of those counts does not establish N.

This distinction is especially consequential when a report supplies a prompt and scores only an answer. Another report that also scores the prompt evaluates a different collection of targets, even if both supplied exactly the same combined text. Neither score can be relabeled as the other's answer-only result.

References: [LM1](https://huggingface.co/docs/transformers/v4.57.1/en/model_doc/gpt2)

## Aggregate the losses using their actual scored counts

PyTorch 2.9's CrossEntropyLoss documentation explains mean and sum reductions and the exclusion of ignored class-index targets. For the ordinary unweighted target loss used here, a returned arithmetic mean belongs to the targets that actually contributed. If a report uses class weighting or label smoothing, its loss definition needs separate examination before it is treated as ordinary target NLL.

Suppose two hypothetical report segments contain disjoint scored targets on one common evaluation basis. The first has two targets with mean NLL 2 nats; its total is 2 × 2 = 4 nats. The second has six targets with mean NLL 1 nat; its total is 6 × 1 = 6 nats. Across the eight targets, mean NLL is (4 + 6) / (2 + 6) = 1.25 nats per token. Corpus PPL is exp(1.25), approximately 3.490.

Averaging the two segment means equally instead gives 1.5 nats, incorrectly giving the shorter segment equal weight. Averaging their already exponentiated PPL values is also a different statistic. Neither calculation recovers this stipulated corpus PPL.

Recovering a segment's total from its arithmetic mean requires its scored-target count. Multiplying by the full padded input length would assign loss to positions outside that mean's population. If the scored counts or reduction are unavailable, the corpus aggregation remains unresolved; a displayed average loss alone cannot reconstruct it.

References: [CE1](https://docs.pytorch.org/docs/2.9/generated/torch.nn.CrossEntropyLoss.html)

## Compare the context available at each scored position

Even with the same tokenizer and targets, the conditional probabilities can differ because the allowed histories differ. Transformers v4.57.1 explains how fixed-length evaluation uses limited preceding context and contrasts disjoint chunks with overlapping windows. Overlap can supply earlier context without making every repeated input token another scored target.

For an original hypothetical comparison, stipulate a fictional tokenizer in which the final piece “ approved” (including its leading space) is one reference target token in the string “the request was approved”. Both reports score that target. One retains the complete preceding text, “the request was”, as context. The other begins its segment at “was”, so the same target sees only that shorter history. These are different conditional-likelihood questions. The example stipulates no probabilities and predicts no direction or size of change.

Context truncation can preserve a target while changing its history. Target truncation removes a prediction from the scored population. Reusing an overlap as context is different again from scoring that overlap repeatedly. A report that counts some targets twice changes their weight in the aggregate, even if the underlying source text is unchanged.

The useful comparison follows the supplied reference positions through the evaluation boundaries: which histories were available, where document boundaries reset them, and whether each intended target contributed once. A common maximum context length does not establish a common window policy. This explains the effect of evaluation choices without prescribing a stride or implementation.

References: [PP1](https://huggingface.co/docs/transformers/v4.57.1/en/perplexity)

## Use PPL for the quality question it actually answers

The related sampling guide examines the next token chosen during generation. Perplexity evaluation instead reads the probability assigned to a specified reference target and continues along the reference history. A generated sample is unnecessary for that definition. A top-k setting, output-length limit or count of returned answers cannot identify the scored-target population.

For reports on a model revision or a proposed inference implementation, establish that the score describes the actual evaluated artifact and probability calculation. If a report transforms the token distribution before scoring, that transformation belongs to its evaluation basis. A generation-policy label alone does not show that the same transformation was used for reference likelihood.

Matched PPL can support a narrow claim about reference likelihood on the evaluated material. It does not establish factual answer accuracy, task acceptance, latency, throughput or an economic advantage. The related GPU benchmark guide covers workload and system performance; those results still need to correspond to the configuration being considered.

If the selected system independently needs receiving, rack integration, onsite installation or IT cabling, Leviathan's Complete Physical Deployment service and related project inquiry provide a route to scope that physical work. Share a nonconfidential equipment and site outline. Model evaluation and tuning remain with the appointed software team.

References: [L1](https://www.leviathansystems.co/services/turnkey-gpu-deployment)

## Decide what the two reports can support

Consider two fictional reports that name the same dataset. If their tokenizations differ as in PPL-225, the smaller PPL does not establish greater likelihood for the common string. If their tokenizations match but one scores answers alone and the other scores prompts and answers, the target populations still differ. If both match those boundaries but expose different preceding contexts, the likelihood questions remain different.

Those reports may each be informative within their stated evaluation conditions. They do not yet justify an interchangeable ranking. The unresolved difference calls for comparable evidence or a narrower conclusion, rather than selecting whichever headline number is lower.

Now suppose both reports identify the tokenizer and revision and establish the same evaluated text, text preparation, tokenization, special-token treatment, scored targets and context policy. Both also use the same token-weighted ordinary NLL aggregation. With the evaluated model artifacts and scoring implementations identified, their PPL values can support a comparison of reference likelihood under those shared conditions. The remaining judgment is whether that reference material represents the buyer's application.

Use the score only as far as that applicability reaches. A matched reference-likelihood comparison can inform the quality discussion alongside application outcomes and serving measurements. A mismatched or unexplained basis leaves the claimed quality advantage unestablished, so the PPL difference cannot carry the proposed compute or serving choice.

## Primary sources

- [PP1 — Hugging Face Transformers v4.57.1 — Perplexity of fixed-length models](https://huggingface.co/docs/transformers/v4.57.1/en/perplexity). Checked 2026-10-20. Causal reference-likelihood definition, tokenization dependence and limited-context evaluation. The hypothetical examples here are independent of the guide's model runs and benchmark numbers.
- [CE1 — PyTorch 2.9 — CrossEntropyLoss](https://docs.pytorch.org/docs/2.9/generated/torch.nn.CrossEntropyLoss.html). Checked 2026-10-20. Class-index loss, ignored targets, reduction, class weights and label smoothing. The article's calculations assume ordinary equally weighted reference-token NLL.
- [LM1 — Hugging Face Transformers v4.57.1 — GPT-2 model reference](https://huggingface.co/docs/transformers/v4.57.1/en/model_doc/gpt2). Checked 2026-10-20. GPT2LMHeadModel parameter definitions distinguish attention masks, internally shifted labels and ignored targets. Other model implementations require their own alignment and loss definitions.
- [L1 — Leviathan Systems — Complete Physical Deployment](https://www.leviathansystems.co/services/turnkey-gpu-deployment). Checked 2026-10-20. Agreed physical receiving, rack integration, onsite installation, IT cabling and handoff. This service reference supports a separate equipment-project inquiry.

## Related Leviathan resources

- [Top-k and top-p: choosing the next generated token](/articles/llm-token-sampling-probability-support)
- [GPU benchmarks: compare the workload and complete system](/articles/which-gpu-is-fastest-workload-benchmarks)
- [Padding and ragged batching: input positions and request counts](/articles/inference-padding-ragged-batching-useful-token-positions)
- [Complete Physical Deployment](/services/turnkey-gpu-deployment)
- [Discuss a separate equipment installation and IT cabling project](/contact?service=turnkey-gpu-deployment&from=%2Farticles%2Flanguage-model-perplexity-tokenization-evaluation-basis)

## Have a separate physical deployment requirement?

Share a nonconfidential equipment, site, rack and IT connection outline to discuss the physical scope, requirements and availability.

[Discuss the scope](/contact?service=turnkey-gpu-deployment&from=/articles/language-model-perplexity-tokenization-evaluation-basis)