vLLM adds Tenstorrent support: serve open models off NVIDIA
Jim Keller's open RISC-V chips finally get vLLM-grade batching — Llama, Qwen, DeepSeek V3, single cards to Galaxy. No spec decoding or LoRA yet.
Copy markdown
An out-of-tree plugin, not a fork
Tenstorrent accelerators join vLLM through the standard platform-plugin path — no core patches to maintain. Clone `tenstorrent/vllm-tt-plugin`, run its install script (it builds vLLM with `VLLM_TARGET_DEVICE=empty`), and the plugin auto-detects your board the moment TT-Metal's `ttnn` is importable.
Llama through DeepSeek V3, single card to Galaxy
Verified text models span Llama 3.1/3.2/3.3, Qwen 2.5/3, Mistral, Gemma 3 and DeepSeek V3, plus multimodal Llama 3.2 Vision, Qwen-VL and Gemma 3. It runs on n150/n300 single cards, the multi-chip QuietBox, and the 32-chip Galaxy — with real continuous batching (phase-based prefill/decode steps) and on-device sampling that hands back tokens instead of logits.
What it can't do yet
No speculative decoding, no LoRA, no multi-host serving, no prompt logprobs, and MoE models on standard multi-process data parallelism aren't supported. Scheduling is phase-based, so a batch is prefill-only or decode-only. Translation: dense open weights on one box today, not every trick vLLM pulls on CUDA.
Why an inference nerd should care
The knock on Tenstorrent was never the silicon — Jim Keller's RISC-V design and MIT-licensed TT-Metal stack are fully auditable — it was the missing production serving layer. This closes that gap, giving you a genuinely non-CUDA path to serve open models with vLLM's batching. If NVIDIA supply or lock-in is your bottleneck, there's finally a real alternative to prototype against.