NVIDIA's ModelExpress v0.5.0 loads a huge model in under 2 minutes
The open-source weight layer for vLLM and SGLang now streams your JIT compile cache across nodes too — plus a new self-hostable gateway for 30+ providers.
Copy markdown
Cold starts fall from 8 minutes to 1:44
ModelExpress, NVIDIA's open-source (Apache-2.0) weight-distribution layer, routes model checkpoints over the fastest available path — GPU-to-GPU RDMA, direct-from-storage, or GPUDirect — with native hooks into vLLM and SGLang. In its DeepSeek-V4 Pro test, weight transfer dropped under 10 seconds and full startup fell from 8+ minutes to 1 minute 44 seconds, so autoscaling and cold starts stop burning GPU minutes.
v0.5.0 moves your compile cache, not just weights
The August 1 release adds a protocol that transfers arbitrary artifacts — including vLLM and SGLang JIT compile caches — so peers stop recompiling kernels from scratch. It also makes the server embeddable in-process instead of a sidecar, and abstracts past CUDA with a first Intel XPU backend.
Elsewhere: a drop-in gateway for 30+ model providers
Aurora, a new open-source (Apache-2.0) AI gateway posted to Hacker News on August 1, puts OpenAI, Anthropic, Gemini, Groq and 30+ providers behind one OpenAI- or Anthropic-compatible endpoint, with health-aware failover, exact and semantic caching, and PII and prompt-injection guardrails. It ships Docker and Helm, so you can self-host a provider-agnostic proxy this weekend.