Ternary Bonsai 2 puts a 27B model in 5.9 GB, keeping 98% of it
PrismML's ternary quant runs near-frontier quality on a 16 GB laptop under Apache 2.0 — and Gemini 3.8 Live reaches GA for low-latency voice agents.
Copy markdown
A 27B model in 5.93 GB, 98.2% retained
PrismML stores each weight as -1, 0, or +1 with a shared FP16 scale — about 1.72 bits per parameter — so Bonsai 2 27B fits in 5.93 GB and still holds 98.2% of Qwen3.8-27B across 20 benchmarks. Apache 2.0, with a 262K-token context.
142 tok/s on a 5090 — or a 16 GB laptop
It runs on a single 24 GB GPU or a 16 GB laptop: 142.5 tok/s on an RTX 5090, 96.7 on a 4090, 46.8 on an Apple M5 Max. GGUF (two packings) and MLX builds are already up.
The catch: a forked llama.cpp, agents slip to ~75%
The files need PrismML's llama.cpp fork — stock llama.cpp can't load them. And long-horizon agent tasks keep only ~75% of the parent, so lean on it for local chat and drafting, not as a drop-in agent brain.
Build this weekend: a private 262K-context assistant
Near-frontier quality offline means you can point it at a whole repo or a long document set (262K tokens) for summarize, refactor, and Q&A with no API bill and nothing leaving your machine.
Elsewhere: Gemini 3.8 Live hits GA for voice agents
Google moved Gemini 3.8 Live and 3.8 Live Extended Thinking — audio-to-audio models with interleaved reasoning — to general availability on the Live API, a supported path for low-latency voice agents.