CUDA Graph Replay Reduces Launch Work, Not the GPU Computation Count

Locate replay in the runtime, after the computation is implemented

An ONNX graph describes model computation. TensorRT's builder selects implementations and produces an engine for its runtime to execute. A captured CUDA graph records GPU work and arguments for replay. These are different layers: an interchange description, a compiled implementation and a way to submit work repeatedly. The related ONNX-versus-TensorRT guide covers artifact reuse.

NVIDIA's CUDA Programming Guide explains why repeated submission has a cost: the host prepares each individually issued kernel. Defining and preparing a graph ahead of execution can reduce recurring CPU launch costs, especially when individual kernels are short.

The purchasing question is which repeated overhead the offered runtime removes. Replay of an unchanged sequence does not itself establish that kernels were fused or that fewer arithmetic operations are needed. NVIDIA also describes opportunities to optimize a whole workflow, so unchanged work need not imply identical execution time. The size of any benefit remains a measurement question.

References: [O1][T1][P1][N1]

GR-215: 100 replays still execute 600 kernel invocations

Fictional GR-215 stipulates a sequence of six GPU kernels for one input shape: a batch of eight rows, with all other dimensions fixed. Each iteration performs the same work. Compare 100 ordinary iterations with 100 replays of a captured version of that sequence.

The ordinary route issues six individual host kernel launches per iteration: 100 x 6 = 600 launches. The replay route issues one host graph-launch call per iteration: 100 x 1 = 100 graph launches. Each replay still executes the six stipulated kernels, giving 100 x 6 = 600 GPU kernel invocations.

The ratio 600 / 100 = 6 compares the host submission counts in this example. It does not describe six times fewer GPU computations or a sixfold speedup. Both routes retain 600 kernel invocations and, by stipulation, unchanged GPU work.

GR-215 is a simplified invented accounting model, rather than a trace of an actual runtime or a count of every internal driver call. Capture, warmup, input staging and all timings are unstated. No measured saving follows from these counts.

New values do not establish coverage for a new shape

PyTorch 2.14's documented capture/replay path reuses memory addresses and assumes fixed tensor sizes and layouts; input storage can contain new values. Its control-flow restriction has an exception for GPU-data-dependent control flow through torch.cond. The Data Dependent Control Flow section documents the cudagraphs backend and eager capture routes, while stating that Inductor support is not yet available. These are the documented conditions checked October 20, 2026.

In GR-215, replacing the eight rows' values while retaining the stipulated captured storage and layout is different from requesting sixteen rows. Evidence for the eight-row capture does not establish coverage for the sixteen-row request.

A supplied system might use another capture, a supported dynamic route or ordinary execution for that request. Ask which route the actual system provides and what results substantiate it. The missing evidence establishes neither that the request must fail nor that it receives the same replay benefit. The qualified control-flow support above also does not establish support for arbitrary input shapes.

References: [P1]

Judge a fixed repeat workload and a broader mix separately

A recurring job with an established shape gives the supplier a specific repeated path to evaluate. If CPU dispatch leaves the GPU waiting for work, reducing that overhead can matter. If most elapsed time is already spent doing the GPU computation, removing host launch work can leave the dominant work largely unchanged.

Now consider a buyer whose requests include both eight-row and sixteen-row inputs. A result collected only on GR-215's captured layout answers the first case. The second case needs its own coverage evidence, and the overall offer needs a result using the buyer's representative mix.

Have the supplier identify the captured region and the requests that actually use it. Keep requests taking other routes in the complete workload result, with their delays, failures and accepted outputs visible. A replay-only average that excludes those requests cannot establish performance for the whole mix. Nor can a graph-enabled label identify how much of an application uses replay.

References: [N1]

Compare complete timings, including costs outside replay

Request comparable evidence from the appointed software team for the same model revision, numerical settings, useful inputs, output requirements, hardware allocation and offered load. Retain the tested framework/runtime versions and the actual shape mix. If other implementation choices change alongside replay, describe the result as a comparison of those implementations rather than attributing every difference to graph launches.

Define the timing finish as the required useful output becoming available. A host submission count cannot supply that timestamp. Keep input staging, work outside the captured region and any relevant queueing or result processing inside the comparison's stated boundary. An isolated replay interval can help explain a result, but it leaves excluded work unresolved.

Also identify preparation paid before repeated execution, including capture and warmup where applicable. Report the complete short-run result as well as a repeat-execution result when the buyer needs both. Startup costs can matter over a short run even when later iterations improve. The linked model-warmup article explains readiness and first-response clocks; its example is a separate measurement-boundary question.

Equivalent useful results remain the finish line. Compare completed accepted work under the required latency and quality conditions, keeping the same request population. A smaller launch count cannot compensate for omitted requests, changed outputs or a timing interval that ends before the buyer's task does.

Make the purchasing conclusion match the evidence

For a fixed repeat job, relevant coverage and complete timings may support choosing a graph-enabled implementation even though the computation count is unchanged. For a broader workload, the same implementation needs evidence for the additional requests and the combined result. Neither choice follows from the launch ratio alone.

A useful supplier response therefore connects three findings: what work replay replaces on the host, which requested inputs the supplied paths cover, and how long equivalent completed work takes. GR-215 supplies only the first finding's simplified counts. The linked GPU benchmark guide covers the wider configuration comparison, while the arithmetic-intensity guide explains a different question about GPU compute and memory limits.

Carry selected equipment into its physical deployment scope

If the separately selected system needs physical installation, Leviathan's Complete Physical Infrastructure Deployment service can scope agreed receiving, rack integration, onsite installation, IT cabling and handoff records. Use the related service and project inquiry with a nonconfidential equipment summary, destination, target window and physical handoff requirements.

Runtime modification, graph capture, workload tuning and application acceptance belong with the appointed software team. This article explains the optimization for an equipment decision; Leviathan's service concerns the separate physical delivery package.

References: [L1]