Update: WASTE runs the full 2.78T Kimi K3 on a 29GB laptop
A dependency-free C engine streams active experts from NVMe, validated to 3.6e-06 vs PyTorch — and Colibri already runs GLM-5.2's 744B this way.
Copy markdown
How it works: experts stream from your SSD
WASTE keeps the model trunk in RAM and streams only the ~4% of experts each token activates from NVMe, compressed to 3 bits with residual vector codebooks. That is how 2.78T parameters run beside just 29GB of RAM.
The tradeoff: ~0.3 tok/s, internal NVMe only
Expect 0.32-0.34 tokens/sec on a 64GB MacBook Pro and a 982GB on-disk container; a USB SSD enclosure craters throughput (12.78 vs 0.94 GB/s). Slow, but it runs a trillion-parameter model on hardware that cannot physically hold it.
Build this weekend: Apache-2.0 C, zero deps
6,000 lines of dependency-free, embeddable C with an OpenAI-compatible HTTP server, and logits validated against the PyTorch reference to 3.6e-06. Runs on macOS, Linux and Windows.
Not just Kimi: disk-streaming MoE is a category now
Colibri (Apache-2.0) uses the same NVMe-streaming trick to run GLM-5.2's 744B on 16-25GB of RAM, with OLMoE working today and Kimi K2, Qwen3-MoE and MiniMax on the roadmap. Frontier MoE weights no longer need a datacenter.