Why Sampled AI Traces Do Not Establish the Request Error Rate

Identify requests, traces and spans before comparing percentages

For this comparison, request error rate means the fraction of observed application requests classified as response errors during a stated interval. It is not an errors-per-second measure. Before comparing dashboards, establish what begins and ends one request and which outcome places it in the error numerator.

OpenTelemetry describes a trace as the path through an application, assembled from spans representing operations. A root span has no parent; child spans can describe work inside the same trace. An AI application could represent an incoming request, a retrieval operation and a model call with different spans. Counting those operations together would count a different unit from incoming application requests.

A search returning 100 spans therefore does not establish 100 requests. Even 100 distinct trace IDs requires a documented mapping to the application request boundary. The example below explicitly stipulates one root trace per application request; that convention supplies its one-to-one mapping.

References: [T1]

Head and tail sampling see different information

OpenTelemetry distinguishes head sampling, which decides early without inspecting the whole trace, from tail sampling, which considers all or most of its spans. A head decision cannot select on a later error it has not yet observed. Tail selection can favor traces containing errors or meeting latency criteria.

That can make rare failures easier to investigate while reducing retained diagnostic data. It also changes which requests appear in the retained population. Neither decision point alone guarantees representative data, and this comparison recommends no sampler or percentage. If an earlier stage discards trace data, later tail selection cannot reconstruct it. Deliberate selection and collection or export loss are separate explanations for missing data.

References: [S1]

Reconcile 1,000 requests with 100 exported traces

Consider one fictional observation interval. Exactly 1,000 application requests complete within it: 990 successful and 10 errors under the declared response classification. Stipulate one root trace per request and no trace or metric collection loss. A separate request counter records every completed request and its outcome before trace selection.

A fictional tail-selection policy exports all 10 error traces plus 90 successful traces. The exported set contains 100 root traces, of which 10 represent error requests. Its error fraction is 10 divided by 100, or 10%.

The complete request population contains 1,000 requests and the same 10 errors. Its error fraction is 10 divided by 1,000, or 1%. Both calculations are correct. One describes the selected diagnostic population; the other describes all observed requests in the stipulated interval.

The 900 unexported traces correspond to successful requests in this example. Their absence from trace search does not mean 900 requests failed. Sampling changed the retained diagnostic set without changing the stipulated application outcomes.

All counts and the selection are invented. This is denominator arithmetic, not a product measurement, statistical estimator or proposed monitoring policy.

A common multiplier cannot repair unequal selection

The overall retained fraction in the example is 100 divided by 1,000, or 10%. Treating that fraction as every request's selection probability would be wrong: every error trace was retained, while only 90 of 990 successful traces were retained.

Suppose someone multiplies every exported root trace by ten. The resulting counts are 100 errors and 900 successes. Their sum happens to match the stipulated 1,000 requests because the multiplier was derived from that known total. The error count is nevertheless ten times the actual 10, and the error fraction remains 10%.

No common factor can simultaneously leave 10 errors unchanged and turn 90 successes into 990. More generally, inverse-probability estimation needs the applicable selection probabilities and a justified sampling design. A single multiplier applied to error-selected traces does not yield an unbiased error total when their inclusion probability differs from the rate that multiplier assumes. The overall retention percentage supplies no such justification.

Ask where the metric gets its observations

OpenTelemetry distinguishes individual request tracing from aggregate metrics. A metric event includes a measurement, time and metadata; aggregation combines observations. A Counter accumulates increments, whereas an UpDownCounter can also decrease, such as when tracking a changing queue. An active-request quantity is consequently a different measure from completed requests over an interval.

A request counter instrumented once per completed request, with outcomes recorded regardless of trace selection, can cover requests whose traces are not exported. That is the provenance stipulated for the 1,000-request count above. Its coverage still depends on the actual instrumentation and delivery path.

Metrics can also be generated from spans. The OpenTelemetry span metrics connector, for example, describes request counts based on spans it sees and error counts based on span status. If its only input is the selected exported spans, aggregation does not restore omitted requests. Without additional justified processing, it describes that input population and its span-based outcome classification.

Ask which input produces the displayed total, where selection occurs and how spans map to application requests. The word metric answers none of those population questions by itself.

References: [M1][C1]

Align time, outcomes and aggregation

For a real comparison, make the numerator and denominator cover the same service boundary, request population and interval. State whether a retry is another request at that boundary and distinguish internal calls from the application request they help serve. Explain how work crossing the interval boundary is assigned; a trace's export time need not be its request's completion time.

A displayed cumulative counter value and an interval count describe different time coverage. OpenTelemetry's metrics data model distinguishes cumulative and delta reporting and discusses resets and gaps. Have the provider explain how its displayed interval handles restarts, missing observations and aggregation across instances. An unexplained coverage gap remains unresolved.

When combining groups with compatible definitions, combine their error counts and request counts before calculating the overall fraction. An unweighted average of group percentages can give a small group the same influence as a much larger one. Reconcile excluded routes or instances on both sides of the comparison.

Finally, a response-error classification is separate from whether generated content is correct. A successfully delivered answer can be incorrect for the task. This request-error fraction cannot supply the missing application-quality evaluation.

References: [D1]

Compare aggregate coverage with linked diagnostic examples

An observability offer should explain how it reports service request volume and outcomes, then how a reviewer can reach the diagnostic examples retained for that population. The aggregate view establishes the observed distribution only to the extent that its stated coverage is supported; traces provide detail about selected requests.

Return to the fictional interval when reviewing an offer. A display of 1,000 requests and 10 response errors, linked to 100 selected traces, has a coherent explanation if it identifies the counter's coverage and the trace-selection policy. A trace-only display showing 10 errors in 100 retained traces leaves the all-request fraction unanswered. Labeling the latter all-traffic performance does not close that evidence gap.

A buyer seeking diagnostic search may find the selected population useful. A buyer seeking request-wide outcome reporting also needs evidence for the aggregate population and its classification. A linked error trace can explain one request; it does not establish how frequently that outcome occurred across the service.

Keep the other purchase decisions with their existing guides. Which GPU Is Fastest? covers workload performance comparisons, while the provisioned-inference and token-billing guide separates capacity allowances, charges and accepted work. Neither a retained-trace percentage nor an aggregate response counter replaces those evaluations.

Use the deployment inquiry for a separate equipment project

If the infrastructure decision also requires physical GPU equipment installation, Leviathan's Complete Physical Deployment service can combine agreed equipment receiving, rack integration, hardware QA, onsite installation and cabling. Bring the equipment configuration, quantities, site and intended physical handoff to the linked service and project inquiry.

Observability purchasing, trace-pipeline design and inference operation remain with the customer's appointed software providers and teams. They are educational subjects here, rather than Leviathan monitoring or managed-inference offerings.

References: [L1]