Swiftlet runs an 80B Qwen in 4.3GB on a Mac, a 35B on an iPhone
Open-source runtimes are dragging near-frontier models onto Macs, phones, a single AMD card and 4GB laptop GPUs — no datacenter, no CUDA needed.
Copy markdown
80B in 4.3GB, thanks to MoE streaming
Swiftlet keeps only Qwen3-Next-80B's ~2.5GB dense core resident and streams expert weights off SSD on demand — just ~3B of 80B params fire per token. It's a Swift/Metal runtime, 4-bit MLX quant, Apache 2.0.
35B on an iPhone 17, 80B on an M5
The 35B peaks at 2.6GB RAM and runs on an iPhone 17 (~1 tok/s); the 80B hits 4.5–5 tok/s on an M5 Mac, the 35B 7–11 tok/s. Both ship in the App Store app 'Priv AI' — fully local, no API key.
The catch: batch jobs, not chat
Prefill is brutal — roughly 30 minutes for 10K tokens — so this is overnight/background work, not a live copilot. And with only ~3B active, it writes like a big model but recalls facts like a small one.
DeepSeek V4 Flash now runs on one AMD MI300X
Doubleword's ROCm bring-up pushes the flagship to ~2,699 output tok/s per GPU (+8.6%). One MI300X packs 192GB HBM3 vs an H100's 80GB at ~half the rent — about $1.99/hr on AMD's dev cloud. Self-host without CUDA or an H100 backorder.
Fine-tune an 8B on a 4GB laptop GPU
New tool 'Soup' streams frozen base layers from RAM into a tiny VRAM pool, one decoder layer prefetched ahead. Llama-3.1-8B (NF4) trains at ~120 tok/s on 3.3GB peak VRAM — only 1.43× slower than resident, bit-exact-validated across nine model families.