llama.cpp v0.9.0 adds multi-node tensor parallelism and FP8

A new NCCL backend shards one model across several machines, OCP FP8 E4M3 quantization arrives, and Mistral Large 4 and Chatterbox TTS get day-one support.

Nowline NOV 5 2:00 PM banner

Top AI stories from the last hour

Top AI stories from the last hour

Copy markdown

  • Shard one model across several machines

    The headline is NCCL-based multi-node tensor parallelism: llama.cpp can now split a single model across GPUs on different machines, not just one box. That's the path to running weights too big for any one rig — pool a couple of workstations instead of renting a datacenter card.

  • Native OCP FP8 E4M3 quantization

    A new FP8 E4M3 tensor type (saturating mode) lands with a LLAMA_FTYPE_MOSTLY_F8_E4M3 ftype and conversion helpers, matching the 8-bit format modern GPUs run natively. It ships alongside a flash_attn_ext_rows op that restores fast prefill for multi-sequence, unified-KV serving.

  • Mistral Large 4, Chatterbox TTS, offline transcription

    Four new architectures get support — Mistral Large 4, Clover, Limite and Metis-1.6 — so you can run Mistral's latest in GGUF locally. Audio expands too: Chatterbox TTS, Pocket TTS preset voices, and Nemotron 3.5 Transcribe for offline speech-to-text in 40 languages with speaker attribution. Gemma 4 E2B/E4B also pick up big prefill speedups.