Prefix-aware routing on SageMaker cuts LLM time-to-first-token 77%
One routing toggle keeps the KV cache warm for shared prompts, no container changes. Plus NVIDIA's open Boltz-2 runtime and Marengo video search in Bedrock.
Copy markdown
Up to 77% faster first token — from one routing toggle
On Llama 3.1 70B with 8K-token shared prefixes, prefix-aware routing cut P50 time-to-first-token by 71–77% and lifted the KV-cache hit rate from about 25% to 82%, with throughput up 15–16%. Requests that share a prompt prefix are pinned to the same instance so the cache stays warm; the routing itself adds just 1.3–1.9 ms.
Flip it on with two endpoint parameters
Set RoutingStrategy to PREFIX_AWARE, then tune PrefixLength (1024–65536 chars) and ConcurrencyThreshold (1–10) for overflow — no model-container changes. It works on every SageMaker real-time endpoint, including inference-component and LoRA-adapter deployments, and even short, variable-length chats saw P50 TTFT drop 13–16%.
NVIDIA open-sources BioIR: 2.9x faster Boltz-2 folding
BioNeMo Inference Runtime speeds biomolecular structure prediction on NVIDIA GPUs while keeping the familiar PyTorch workflow — 2.90x higher residue-normalized Boltz-2 throughput on 8xH100 (58.5K vs 20.2K folded residues per GPU-hour). It ships as a precompiled wheel on GitHub with no CUDA toolkit at runtime and also serves OpenFold3 and OpenFold2, putting a high-throughput folding pipeline a weekend away.
Bedrock knowledge bases now search video, image and audio
TwelveLabs' Marengo Embed 3.0 is generally available as an embedding model in Amazon Bedrock Knowledge Bases, so a fully managed KB can answer natural-language queries against video, image and audio — not just text. Build this weekend: semantic search across your media library with no custom embedding pipeline to wire up.