One Inference Pool or Separate Prefill and Decode: Compare the GPU Deployment

Follow the prompt, retained state and generated tokens

In autoregressive transformer inference, prefill processes the input prompt and establishes the context for generation. Decode then produces subsequent tokens step by step. NVIDIA's glossary describes prefill as compute intensive and decode as sensitive to memory bandwidth. These characteristics explain the interest in allocating resources separately; they do not identify the bottleneck of every model at every load.

The key-value cache, or KV cache, holds attention-layer information derived from previously processed tokens. Hugging Face's explanation shows why retaining those keys and values avoids recalculating them for every new token. Generation reuses the retained state and adds information for newly processed tokens. The cache is runtime state associated with the sequence, distinct from the model weights.

That distinction matters when comparing offers. Having the model available on another GPU does not, by itself, place the prompt's computed KV state there. A separated serving arrangement must account for how the decode side obtains the state it needs.

References: [N2][H1]

Define the pool boundary before counting GPUs

Aggregated serving keeps both phases within the same serving instances, which may span several GPUs. Phase disaggregation assigns prefill and decode to separate instances and resource pools. Each phase can use its own partitioning and batching. Different GPU types are optional; separating roles does not itself require them.

A pool groups resources assigned to a role, while an instance can use several GPUs. Consequently, two pools do not establish two replicas or two models. Ask for GPUs per instance, instances per pool and the model served by each. Worker counts therefore need their GPU allocation stated.

References: [R1]

Follow the KV state across the separation

NVIDIA Dynamo provides a named implementation example. Its Architecture page, labeled stable v1.5.1 when checked on October 19, 2026, describes selecting a prefill worker, passing transfer metadata into the decode request and coordinating direct GPU-to-GPU KV transfer through NIXL. The decode worker then generates tokens from the transferred cache. Routing information and the cache payload serve different purposes; a routed request alone is not the transferred state.

For an offer, ask where those GPU groups are placed and which path carries the state. Separation by role does not itself require separate racks. If the placement crosses servers, identify the corresponding connections and supported transfer arrangement. An aggregated offer can also require inter-server communication for model distribution.

Have the platform and network owners identify the applicable endpoints, transport and evidence for the chosen workload. A switch headline cannot establish transfer time. The glossary's orchestration discussion likewise treats moving cache and routing requests as coordinated system functions.

References: [N3][N2]

Trade latency isolation against resources reserved for each phase

Dynamo's stable v1.5.1 Disaggregated Serving page describes the central tradeoff. Under sustained contention, separate pools can prevent long prefills from interrupting generation. At low concurrency, aggregated serving can avoid transfer and pool-fragmentation overhead. Where decode KV capacity is limiting, assigning more GPUs to prefill can reduce total serving capacity. Separation changes resource use; it does not remove either phase's work.

Beyond the Buzz uses simulation to examine rate matching between the pools. Its preferred allocation varies with model, traffic and latency targets; those findings are not measurements of the buyer's configuration.

Ask what happens when the traffic mix changes. More incoming prompt work and longer generated responses put different demands on the arrangement. A supplier proposing a fixed split should show evidence for that split over the relevant operating range. A supplier proposing redistribution should identify the supported mechanism and evidence for the resulting configurations.

References: [N1][R1]

Compare two fictional offers that both name eight GPUs

Consider fictional offers A and B for the same selected model. Both list eight physical GPUs of the same type. A assigns all eight to an aggregated serving pool. B assigns two GPUs to prefill and six to decode, and stipulates an interpool KV-transfer requirement. The example describes allocations only: the suppliers would still have to establish model fit, instance layout and supported software. Two/six is not a recommended ratio.

The inventory arithmetic is unchanged: B contains two plus six, or eight physical GPUs. It does not contain eight decode GPUs plus a free prefill service. Nor does its two-pool description establish two model replicas. A's eight-GPU pool likewise does not tell you whether it contains one instance spanning eight devices or several smaller instances.

Suppose the buyer expects a quiet period with occasional short prompts and a busy period mixing long prompts with ongoing generation. Ask for comparable evidence from both offers in both periods. B must demonstrate that its separation is useful while its six-GPU decode allocation and transfer path meet the required result. A must demonstrate that its mixed serving arrangement meets that same result. Neither allocation alone determines the outcome.

Now suppose B's price names the two GPU groups but leaves the equipment or software needed for its specified transfer path outside the offer. B is an incomplete alternative until that dependency is identified and priced or confirmed as available elsewhere. Ask whether the price includes any additional network adapters, switch ports, optics and cable assemblies identified by the actual design, as well as the transfer software and its integration. Conversely, an already-supported existing path should not automatically be purchased again. Compare the complete incremental purchase for each offer, including the equipment, connections and software services it actually requires. This example supplies no benchmark result, network rate or predicted saving.

Request evidence that includes both phases and the complete path

Use a common comparison brief: exact model and revision, required output quality, numerical precision, prompt-length and output-length distributions, arrival pattern and concurrent requests. Retain software/backend versions, GPU allocation, parallelism, batching and cache-reuse conditions. Record architecture-specific settings rather than describing the result as a chip-only comparison.

Measure from the agreed client boundary through the complete arrangement. MLCommons' inference rules distinguish time to first token from time per output token and attach quality and latency conditions to a run. Request relevant latency distributions, completed requests or tokens per unit time, and failures or rejected requests over the same interval. State definitions and units; an average generation interval can conceal occasional long pauses.

For B, decode-only evidence omits prefill, transfer and routing. A different pool ratio does not validate the proposed two/six allocation. For A, omitting representative long prompts leaves its mixed workload unresolved. Include quiet and busy conditions and disclose any assumed reusable prompt state. The related benchmark guide explains broader result comparison; the memory guide distinguishes device capacity from memory available to the workload.

References: [M1]

Select the supported arrangement behind the architecture label

The serving software is part of this purchase. The vLLM stable documentation retrieved on October 19, 2026 labels its Disaggregated Prefilling feature experimental. It describes separate instances for independently tuning first-token and inter-token latency, and explicitly cautions against expecting throughput improvement from that feature. That is a qualification of the named implementation, not a universal verdict on every separated system.

Have the proposed software provider identify the supported combination of model, backend, versions, transfer component and hardware. A Dynamo architecture diagram does not establish that an arbitrary combination is supported. The provider should also explain the routing, pool allocation and operating support included in its offer.

Choose aggregated serving when its evidenced complete result meets the requirement and the additional separation has no demonstrated value for the intended operating range. Choose separate pools when configuration-specific evidence supports the required latency and throughput, and the complete provision is justified. If neither offer includes comparable evidence, the next decision is a bounded evaluation with the appointed platform team. Total GPU count cannot settle the missing comparison.

References: [V1]

Bring the selected equipment and connections to the physical inquiry

Once the serving owners have selected an arrangement, translate its placement into the actual equipment and connection package. Bring the OEM/configuration list, rack positions, released endpoint schedule, appointed supplier and the physical work required. For separate pools, identify which connections support their transfer path and which are already present.

Leviathan's Complete Physical Infrastructure Deployment service can combine agreed receiving, rack integration, hardware QA, onsite installation, IT cabling and physical records. Use the linked project inquiry to discuss the selected equipment, connections, required handoff and availability. Methods, resources and included checks are engagement-defined.

Serving architecture approval, application commissioning and managed inference remain with separately appointed providers; they are not established by the physical service. Facility electrical construction, liquid-side work and licensed or stamped engineering are outside its scope.

References: [L1]