SGLang v0.5.18 cuts cold starts 2.38x, adds 7 new model families
Overlapped staging drops a Qwen3-32B cold start from 85s to 36s on an H100 — plus decode wins, Simon Willison's LLM 0.33, and OpenAI region pinning.
Copy markdown
A 2.38x faster cold start
SGLang v0.5.18 overlaps checkpoint staging during startup, bringing a Qwen3-32B up on an H100 in 35.6s instead of 84.8s. If you autoscale inference on serverless GPUs, that roughly halves the cold-start tax on every new replica.
Seven new model families, day-0
The release adds serving for Muse Glimmer and Intern-S2-Mobius plus five diffusion stacks — SANA-Video, LingBot-Video-MoE, LTX-2.5, Cosmos3, and LongCat-Image. You can stand these up locally without hand-writing a loader.
Decode-path wins for big MoEs
TP LMHead collapses its allgather-scatter into a single all-to-all (320µs to 169µs on a DeepSeek-V4-Pro B200 decode), and FlashInfer's MNNVL allreduce adds up to +6.9% at small batches on Blackwell. Bump to PyTorch 2.13 and FlashInfer 0.6.17 first.
LLM 0.33 moves onto openai 3.x
Simon Willison's LLM CLI migrates to the OpenAI Python 3.x library and adds embedding upgrades and template composition, on top of its new reasoning traces and server-side Shell/WebFetch/WebSearch tools. Scripts pinned to the old openai<3 surface may need a bump.
OpenAI adds per-request region pinning
Projects on Global geography can now route a single API call to a specific region via a prefixed API domain — handy for keeping regulated requests in-region without splitting the whole project. Existing data controls carry over.