Brief
Amazon SageMaker Inference adds prefix-aware routing for LLM latency reduction
Amazon SageMaker Inference now offers prefix-aware routing, a strategy that sends requests sharing the same prompt prefix to the same instance to keep the KV cache warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%.
Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%.
Our reading is that this can reduce latency for applications that repeatedly use the same prompt prefix, because the cache is reused instead of recomputed.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by aws.amazon.com
- Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm.
- In benchmarks on Llama 3.1 70B, prefix-aware routing reduced P50 time-to-first-token by up to 77%.
- In benchmarks on Llama 3.1 70B, prefix-aware routing raised KV cache hit rates from about 25% to over 80%.
Sources
- AWS Machine Learning BlogText stored 15 September 2026
How this story was checked. Written from the 1 page listed above, stored 15 September 2026; claims checked against that stored text on 15 September 2026.
What that means
- 3 of 3 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.