Request Rate or Concurrency: What Load Did the Inference Benchmark Actually Offer?

Identify what controls the next arrival

Before using a throughput figure to support an inference purchase, ask what caused the next request to be sent. Did a preceding response release a position, or did the request have an intended arrival time independent of preceding responses? That distinction determines which demand claim the result can answer. It is the completion dependence formalized in Schroeder, Wierman and Harchol-Balter's 2006 study of open and closed workload models.

Consider a generator holding eight requests outstanding. Under its stipulated replacement rule, a terminal response releases one position and triggers another request. Slower responses delay those replacements. The experiment therefore changes the timing of later offered work as the system responds. Eight is a population at a boundary, not a request-arrival rate or a count of simultaneous GPU computations.

Now consider a schedule that calls for new requests at specified times whether earlier requests have finished or not. Slower responses do not move those intended times. Whether the generator actually dispatches on that schedule remains an observation to establish. This is the useful distinction: what demand was intended, and what demand was actually presented?

References: [P1]

Read NVIDIA's mode definition as an attempted load

NVIDIA's archived Triton Inference Server 2630 Perf Analyzer documentation supplies a named example. Concurrency Mode attempts to maintain a specified number of outstanding inference requests. Request Rate Mode attempts to send a specified number of inference requests per second. Both descriptions concern what the generator attempts; neither establishes achieved load for a particular run.

The current unversioned page, also checked on October 20, 2026, retains those definitions. It additionally describes user-supplied inter-request intervals and a fixed schedule with request timestamps. Those options illustrate why the request timing matters beyond an average rate. They do not establish that any selected pattern represents a buyer's production workload.

The distinction between intended demand and recorded dispatches below is an interpretation framework, not a claim that these documentation pages report such a trace.

References: [N1][N2]

RL-205: account for the intended arrivals and dispatches

Original fictional case RL-205 starts with no earlier outstanding requests and specifies 100 intended arrivals/s for ten seconds. Its intended population is therefore 100 × 10 = 1,000 arrivals. Stipulate no retries or duplicate attempts. Dispatch means a recorded send at the generator boundary; it does not itself prove receipt or admission at the serving system.

At the ten-second observation cutoff, stipulate that 970 requests were actually dispatched. Of those same request identities, 800 completed successfully, 100 received explicit rejection outcomes and 70 remained unfinished. The dispatched population reconciles as 970 = 800 + 100 + 70. The intended population reconciles separately as 1,000 = 970 + 30: thirty intended arrivals were not dispatched by the cutoff.

Those thirty belong to neither the rejection count nor the unfinished dispatched population. Calling all thirty service failures would assume an event that the example never supplies. Likewise, the 70 unfinished requests cannot all be called a server queue: this trace does not identify where their unfinished work resides.

The window contains an average of 97 recorded dispatches/s and 80 successful completions/s. The configured intention remains 100 arrivals/s. Reporting 100 successful requests/s would substitute the target for the outcome. Reporting 80 requests/s as a measured service ceiling would also overstate the evidence: this finite trace does not establish what limited dispatch, why requests were rejected or whether a different demand pattern would produce the same result.

All RL-205 counts and statuses are invented. The example conducts no test and predicts no queue growth or failure rate. Its ten-second totals also leave dispatch timing unresolved: matching a total would not, by itself, demonstrate that each intended arrival time was met.

Retain the unfinished work when reading latency

Suppose the RL-205 report also gives a latency percentile calculated from its 800 successful completions. That statistic describes the successful requests included under its stated timing definition. The counts alone supply no durations, and the statistic cannot describe the 70 requests that were still unfinished at the cutoff.

Ask for a declared later disposition of those same 70 identities before a completed-request latency summary is used to characterize all dispatched work. A later observation should identify which completed, which ended with another outcome and which remain unresolved, with their timing boundaries preserved. Requests still unresolved remain outside a complete accounting of terminal durations; they cannot be assigned zero latency or treated as completed at the cutoff.

The 100 explicit rejections need their own outcome label too. Their elapsed time to rejection is different evidence from the time to a successful inference response. Even after every dispatched request has a known disposition, a success-only latency percentile remains conditional on success. A report combining terminal outcomes must state that population rather than silently presenting it as successful-response latency.

Later completions can resolve the original arrival cohort, but they do not increase the number that completed inside the original ten-second window. Keep eventual cohort outcomes and in-window completion rate separate. The related request-queues guide addresses the further question of what a caller timeout or cancellation establishes about server work.

Eight outstanding requests leave a different demand question open

Contrast RL-205 with a separate completion-dependent experiment that holds eight requests outstanding. Stipulate immediate replacement after each terminal response and no additional user pause. If responses take longer, the corresponding replacement arrivals happen later. The generator is no longer presenting an arrival schedule that proceeds independently of those responses.

That behavior can be useful evidence for a fixed interacting population. It leaves a specific gap when the buyer's question is whether the system can face externally scheduled arrivals during slow responses: the fixed-population experiment has allowed later demand to wait for earlier outcomes. A favorable latency summary under that experiment does not close the gap.

Do not divide eight by one latency sample to declare a universal capacity. One request's duration does not supply the timing, outcomes and occupancy of the complete run. The recorded arrivals and outcomes are the evidence needed here. Nor does the arrival-driven RL-205 trace tell us how many requests the eight-outstanding experiment would complete; they are separate fictional observations, not interchangeable workloads.

References: [N1][P1]

Choose evidence for the intended interaction

A buyer evaluating a fixed population of interactions can reasonably request fixed-concurrency evidence. If each participant waits for an answer before initiating another request, response-dependent demand is relevant. The report still needs the actual interaction pattern: eight continuously occupied request positions do not automatically represent eight people, especially if the real users pause between requests.

A buyer whose application receives independently arriving jobs can reasonably request arrival-driven evidence. The intended arrival times, recorded dispatch times and outcomes show whether the supplied evidence addresses that demand. The same average rate can describe evenly spaced work or an uneven sequence, so the rate label alone leaves the timing pattern open.

In either case, keep the model and revision, input and required output, quality condition and response-time requirement aligned. Replacing a long response with a short one to produce a stronger number changes the task being evaluated. A success label also needs to say whether the response met the buyer's content and timing requirements.

The broader GPU comparison guide covers configuration and benchmark comparability. The storage metrics guide addresses operation rates, bytes and latency. Account admission is another question: the token-quota guide explains why an allowance or reservation does not establish observed serving behavior. None of those substitutes for identifying how this experiment supplied requests.

References: [P1]

State the demand claim that remains untested

A useful benchmark review can end with a precise unresolved claim. For an eight-outstanding report, that might be: the evidence describes a fixed population, while the intended independently arriving workload remains untested. For RL-205, it is: the intended 1,000 arrivals were not all dispatched, and the outcomes and durations of the 70 unfinished dispatches remain unresolved at the cutoff.

Ask the evidence owner to connect the intended demand record to dispatch timestamps, identifiable outcomes and the latency population. Retain the observation window and any later follow-up separately. This request concerns the meaning of an existing report; it is not a procedure for applying load to an endpoint.

The result is a clearer statement of what the evidence supports, without choosing a winner between tools or modes. It supplies no GPU count, account entitlement, cost saving or performance acceptance. Those decisions require their own configuration-specific evidence.

Connect selected equipment to a separate physical project

If the resulting infrastructure plan includes equipment that needs physical installation, Leviathan's Complete Physical Infrastructure Deployment service can discuss agreed receiving, rack integration, onsite deployment, IT cabling and physical handoff records. Use the related service and project inquiry with a nonconfidential equipment, site, connection and handoff outline.

This article concerns software measurement education. Inference operation and workload-performance evaluation remain with the appointed platform and application owners; the physical installation inquiry concerns a separately identified equipment need.

References: [L1]