Google and the participating community released llm-d 1.0, a Kubernetes-native stack for distributed LLM inference. It coordinates model servers using context-cache awareness, load-aware routing, and traffic controls for shared infrastructure. Google Cloud official announcement
Recomputing the same long prompt on several replicas leaves waste even when more accelerators are added. llm-d aims to make the servers act as a coordinated pool, including routing work toward replicas that already hold useful context.
On this page
What an LLM server can reuse
LLM inference includes prefill, which processes the input context, and decode, which produces subsequent tokens. A KV cache retains intermediate computation that may be reused. Repeated instructions or shared documents can create useful common prefixes. That state consumes memory and resides on particular replicas, so its location matters when a router chooses where a request should run.
A round-robin load balancer alternates between replicas without knowing which context each one holds. Related requests can therefore repeat work on different machines. Cache-aware routing attempts to select a replica with useful existing state. It must also account for queueing and load: always choosing the replica with a cached prefix could overload that machine and defeat the latency benefit.
Throughput and waiting time serve different users
An overnight summarization job values completed volume; a person using a chat service cares about how soon the response begins. When both use one accelerator pool, long background work can delay interactive requests. Distributed inference therefore includes admission and scheduling policy, rather than simply increasing the number of model servers behind an endpoint.
Separating prefill and decode into different resource pools can allow independent scaling, but it also introduces communication and failure-handling requirements for intermediate state. Model size, context length, and interconnect conditions affect the outcome. Before adopting the entire distributed stack, identify whether the current bottleneck is computation, memory capacity, or queueing and evaluate the component addressing that bottleneck.
Design a comparison that exposes cache dependence
Create one evaluation set with repeated shared documents and another with unrelated documents. Keep request count and concurrency comparable, then compare time to first token, total response time, and failure rate. If the gain depends on shared context, state that condition alongside the result. A vendor’s performance example should not become a universal expectation for unrelated traffic.
Also test the loss of a replica and a router update. High cache-hit rates during normal operation can conceal a surge of recomputation after a failure. Examine both cold-start and warmed-up behavior, and retain a route back to the previous serving configuration. These are proposed experiments; this article does not claim to have reproduced a benchmark or measured a production deployment.
Metrics for a distributed-serving evaluation
- Measure time to first token separately from complete response time.
- Track cache hits alongside GPU or TPU memory usage.
- Check whether batch work delays interactive requests.
A 1.0 version does not establish production suitability for every configuration. Introduce only the components you need and preserve a rollback path.
Announcement reference date: 2026-10-21 · Checked: October 24, 2026
