Averaging Worker p99s Does Not Give the Inference Fleet p99

A worker percentile has already discarded the fleet's ranks

A supplier offering inference capacity may show a p99 beside each worker and another number labeled fleet p99. The buying question is how the second number was obtained. Averaging the displayed worker values is insufficient evidence, however attractive the result.

A percentile locates a position in an ordered population. Under a nearest-rank rule, it is an order statistic: the duration of the observation occupying the specified rank. An arithmetic mean instead divides the sum of durations by their count. Those operations require different information.

A local p99 tells the reader about a position within that worker's observations. It does not preserve the durations or counts at every other position. Adding the worker's total request count identifies its population size, but cannot reconstruct the discarded distribution. The pooled rank may fall among observations whose durations the local percentile never reports.

Prometheus' current Histograms and summaries documentation makes this distinction explicit for replicated services. Summaries expose precomputed quantiles; those quantiles cannot generally be aggregated into the overall population's quantile. Histograms retain bucket counts that permit aggregation when their representations are compatible.

References: [P1]

PL-206: rank the same 100 observations together

PL-206 is wholly fictional. Exactly 100 requests supply one duration observation each over one common interval, using the same request-start and response-finish events. Worker A contributes ninety observations, every one exactly 100 ms. Worker B contributes ten observations, every one exactly 1,000 ms. There are no additional observations.

Choose the nearest-rank definition for this example: sort durations from shortest to longest and take observation ceiling(p x N), where p is the percentile expressed as a fraction and N is the population size. This is an explicit mathematical convention for PL-206, not a provider's percentile convention or a real benchmark.

Worker A's local p90 is 100 ms; worker B's is 1,000 ms. Their unweighted average is (100 + 1,000) / 2 = 550 ms. Weighting those local p90 values by request counts gives (90 x 100 + 10 x 1,000) / 100 = 190 ms.

Now order the complete population. Observations 1 through 90 are 100 ms; observations 91 through 100 are 1,000 ms. The pooled p90 is observation 90, hence 100 ms. The pooled p99 is observation 99, hence 1,000 ms. Neither 550 ms nor 190 ms computes the pooled p90.

Because all durations within each worker are identical, each worker's local p99 also equals its local p90 here. Averaging those p99s again gives 550 ms, or 190 ms with request-count weighting. Neither computes the fleet p99 of 1,000 ms.

Why 190 ms is a valid mean and an invalid percentile

The actual sum of the stipulated raw durations is 90 x 100 ms + 10 x 1,000 ms = 19,000 ms. Dividing by all 100 observations gives a pooled arithmetic mean of 190 ms. This sum adds individual elapsed durations; it does not describe the wall-clock length of the common observation interval.

The request-weighted percentile calculation happened to produce the same number because each worker's mean equals its local percentile in this deliberately simple population. Replacing each local percentile with that worker's actual mean would produce the correct weighted-mean calculation. For a worker containing varied durations, that equality need not hold.

The number's value therefore does not establish its meaning. A valid mean can lie between the two durations even though no request took 190 ms. Under PL-206's nearest-rank convention, the pooled percentile must come from the ordered observations.

Prometheus separately explains that duration sums and observation counts are additive. They support the pooled mean without making summary quantiles mergeable. A report can supply credible mean evidence while leaving its fleet p99 unsupported.

References: [P1]

Buckets preserve boundary counts, not exact durations

Replace PL-206's exact raw durations conceptually with a supplied cumulative histogram that retains only two finite boundaries: at most 200 ms and at most 1,200 ms. Across the complete population, those counts are 90 and 100.

At the 200 ms boundary, worker A contributes 90 and worker B contributes zero. At the 1,200 ms boundary, A contributes 90 and B contributes 10. Adding each boundary's corresponding worker counts preserves the pooled population at that boundary. Adding the two cumulative totals together would count some requests twice.

The first count establishes that 90 of the 100 observations are at most 200 ms. Subtracting it from the second establishes that ten lie above 200 ms and at most 1,200 ms. The counts alone do not establish that the lower observations were exactly 100 ms or the upper ones exactly 1,000 ms.

For the chosen nearest-rank rule, these bucket counts place observation 90 at or below 200 ms and observation 99 above 200 ms and at or below 1,200 ms. They do not recover PL-206's exact p90 and p99. We know those exact values only because the original example stipulated the raw durations.

Prometheus' histogram quantile function interpolates within buckets when needed. Its documented assumptions differ between classic or custom buckets and standard exponential buckets. The resulting percentile can therefore be an estimate even when the populations were combined correctly. A boundary count and an interpolated duration answer different questions.

References: [P2]

Check what survives bucket-layout reconciliation

Histogram aggregation still needs compatible representations. Prometheus' guidance qualifies classic histogram aggregation by compatible bucket boundaries. A shared metric name alone does not establish that compatibility.

The current native-histogram specification distinguishes standard exponential schemas from native histograms with custom bucket boundaries, abbreviated NHCBs. Standard schemas can be reconciled to a common lower resolution by combining neighboring buckets. NHCBs are compatible with other NHCBs, with differing boundaries reconciled through shared boundaries. If none are shared, only an overflow bucket containing all observations can remain. NHCBs are not interchangeable with standard exponential schemas for these operations.

The relevant buyer evidence is the representation and resolution used after combination. A finely resolved worker display does not establish equally fine fleet evidence if reconciliation removed boundaries. Have the supplier explain incompatible inputs, omitted results and reported annotations. The label native histogram alone supplies neither universal compatibility nor an accuracy guarantee.

References: [P1][P3]

Make the fleet SLO claim come with fleet evidence

A supplier promising a fleet service-level objective, or SLO, should identify the population and evidence supporting that aggregate response condition. Request the common interval, request start and finish events, included workers and observation counts. Have the supplier explain how those observations become the displayed fleet percentile, including the percentile convention or estimation method and any resolution lost during combination.

For PL-206, a supplier could present both worker p99s accurately and still mislabel their average. Reporting the ninety-to-ten request split would expose the unequal populations, but weighting the labels would not repair the calculation. The missing evidence is a distribution representation sufficient for the aggregate question, such as compatible bucket populations with their uncertainty visible.

Per-worker summaries can remain useful for comparing workers under their stated conditions. Their usefulness does not extend automatically to establishing a fleet percentile. If an offer supports its fleet SLO with an averaged worker percentile, that calculation leaves the aggregate response condition unestablished. A supplier may provide other evidence for a threshold condition; the average itself does not provide it. That gap does not prove the system is slow.

Keep aggregation separate from observation coverage. Combining all supplied buckets faithfully still describes only the requests those buckets contain. Use the linked sampled-trace guide to assess which requests reach an evidence stream. The storage and switch latency guides explain timing boundaries, while the GPU performance guide addresses workload comparisons. Those decisions should remain explicit rather than being inferred from a fleet percentile label.

Carry a separate equipment project into physical deployment

If the selected infrastructure also needs equipment receiving, rack integration, agreed hardware QA, onsite installation or cabling, Leviathan's Complete Physical Deployment service can discuss those physical stages as an agreed package. Use the linked service and project inquiry with a nonconfidential equipment configuration, site and intended physical handoff.

This article provides educational interpretation of supplied latency evidence. Observability design, software integration, live measurement, inference operation and SLO approval remain with the customer's appointed teams and providers; they are outside this physical deployment offering.

References: [L1]