llm-d 1.0: Follow a Returning Request Through the Shared GPU Pool

On October 20, 2026, llm-d contributors announced version 1.0 of the Kubernetes-native distributed inference stack. The release puts production coordination at the center of the story: a returning request has to reach useful cached context, compete for shared capacity and survive changes to the serving arrangement. For an enterprise with an existing GPU fleet, the adoption question is whether cache-aware placement, admission and rollout coordination work together along that request path. [1]

Prefix- and load-aware routing, KV-cache offloading and prefill/decode disaggregation already existed in llm-d. The new milestone emphasizes how serving components work together under competing demand and planned maintenance. A cache feature, a priority feature and an upgrade feature can each be useful while their proposed combination still needs evaluation. [1]

Where the returning request goes

Consider one fictional case. Compatible replicas A and B serve the same approved model. A retains an eligible prompt prefix P but already has queued work. B is available and has no P. An authorized foreground request returns with P while a separate burst of background requests arrives. These are stipulated states; no serving run or benchmark has been performed.

Sending the foreground request to A offers reuse of prefix computation, followed by the remaining prompt work and new generation. It also encounters A's existing work. Sending it to B offers an available replica but requires computing the prefix again. Cache affinity alone cannot establish which complete path meets the foreground response requirement. Choosing an empty queue alone leaves B's additional computation out of the comparison.

The useful comparison is the time and work along each eligible path, including waiting, remaining prompt processing and generation. The stated facts contain no durations or workload sizes, so they determine no winner. They also supply no reason to move cached state between A and B or assume such a transfer is available.

llm-d's prefix-routing documentation explains the mechanism: a cache index associates reusable KV blocks with endpoints, supplying information for request placement. The precise-routing guide uses model-server cache events rather than inferring residency solely from request history. A cached prefix is reusable computation, not a stored answer; the prompt-cache versus answer-cache guide covers that separate distinction. [4, 5]

The background burst changes the admission question

A routing decision describes where work goes. Flow control also governs when queued work is released toward the serving pool. That matters in this case because the background burst competes with the returning request before a cache-aware placement can deliver its intended benefit.

The v1.0 flow-control documentation describes queues at the Endpoint Picker, strict priority bands, a configurable fairness policy among flows within a band and ordering within a flow. Dispatch is gated on pool capacity, with bounded queues and rejection outcomes. This flow-control path requires explicit enablement; installing a release that contains it does not establish that the proposed service uses it. [6]

Applied to the fictional case, those controls let the platform express which work should receive dispatch opportunities as arrivals compete. They do not decide whether A's reuse outweighs its waiting, create extra GPU capacity or establish a particular response time. Priority between classes and fairness within a class answer different allocation questions. The selected policy still has to reconcile the foreground requirement with the treatment promised to background work.

This counterfactual illustrates why routing and admission must be evaluated together: the opportunity to reuse P depends partly on what competing work has already been admitted.

Moving this request between existing replicas is also different from adding replicas or provisioning GPU workers. The replica-versus-node autoscaling guide explains those capacity decisions. Neither additional replicas nor additional workers are assumed in this example.

Carry the same request into a proposed rollout

Now the enterprise proposes a model-server update while this service continues accepting returning foreground requests and background work. Finding P and identifying an eligible serving revision become separate questions. A replica's useful cached state does not establish that it belongs to the intended rollout revision, and a new replica's readiness does not establish that it retains P.

For a deployment with separate prefill and decode roles, compatibility also spans the phases: prompt processing must connect to a suitable generation path. The phase-pool comparison guide explains the architectural choice. The release-specific issue here is how those roles change revision together while requests continue arriving.

The v1.0.0 release explicitly identifies DisaggregatedSet as managing prefill/decode replicas as one logical unit, with coordinated rollouts, lockstep upgrades and revision routing. Its operations documentation describes slices containing the related roles, with roles independently scalable inside that managed topology. Coordinating their revision is a different relationship from keeping a returning request near a warm cache. [2, 7]

That topology alone does not constrain every phase pairing. The v1.0 safe-rollout guide separately requires router revision screening and graceful draining of in-flight work and dependent KV transfers. Its rollout screener is an Alpha plugin requiring explicit experimental enablement. A strategy that starts replacement Pods before old counterparts drain also needs spare accelerators; a fully allocated fleet can stall that rollout. [7, 8]

In the same fictional service, the proposed rollout therefore needs a request path that remains eligible for its serving revision as well as useful for context reuse. Keeping affinity to A cannot, on its own, settle the phase-compatibility question. Moving to B cannot, on its own, establish that P survived the change. The example assumes neither cache portability across revisions nor an actual rollout outcome.

The release's zero-downtime-operations label names mechanisms, including router hot standby and rolling updates. It does not prove that this enterprise loses no response or context, or meets a particular service-level objective. Router availability and model-server update strategies address different failure points. Evidence for the proposed combination must cover queued requests, work already generating and subsequent returning turns. [2, 8]

There is a concrete reason to examine the combination. As read on October 20, the v1.0 precise-cache guide uses a single router replica and warns that active-active operation needs shared in-flight accounting and a complete cache index at startup; it points to an experimental snapshot option. The release's availability headline cannot be inherited by every cache-routing configuration. [5]

Keep upstream components and managed GKE integration distinct

The upstream release is a composition of separately versioned pieces. Its router configuration API reaches v1, while the component and model-server tables retain distinct engine versions and accelerator-specific qualifications. A 1.0 stack label does not make every dependency version 1.0 or make GPUs interchangeable. Select the engine, model, backend and hardware combination together. [2]

Google's October 20 account adds a particular operating arrangement: GKE Inference Gateway manages the proxy and load-balancing layer and invokes the llm-d Endpoint Picker. It also describes Google Cloud integrations, including a Managed Lustre cache tier. Those are useful details when evaluating that GKE arrangement; they are not universal services supplied by an upstream llm-d installation. [3]

This distinction changes the enterprise's decision. An existing platform may already supply the gateway, traffic classification and deployment control, leaving selected llm-d components to fill specific gaps. Another arrangement may include more of those operating functions as a managed service. Compare the responsibilities actually supplied and their integration with the retained platform, rather than treating a common endpoint as the whole product.

Evaluate the coordinated service the fleet must deliver

Return to the foreground request. The useful outcome is a new answer under its response requirement while the background burst and proposed update are handled as intended. Cache residency helps explain saved work; admission explains access to capacity; revision coordination explains eligible serving paths during change. None of those observations substitutes for the other two.

The October 20 release matters because it brings those responsibilities into the same serving discussion. An enterprise can use the A/B case to locate what its platform already supplies, what the proposed arrangement adds and where combined evidence is still missing. That is a more complete adoption decision than choosing a router by cache-hit rate or treating a release label as a fleet-wide operating result.

If the selected arrangement separately requires server installation or new physical connections, Leviathan's Complete Physical Deployment service and its project inquiry provide a route to discuss that equipment, location and physical handoff. An evaluation using existing installed capacity may require no physical project. [9]