Edge0 streams MoE experts off SSD: run a 35B model in 3GB of RAM
Apache-2.0, MLX-only for now, ~15 tok/s on a Mac mini — and near-FP16 quality. Plus: Perplexity's local agent lands on Windows RTX PCs.
Copy markdown
How it fits: the experts stream off your SSD
Instead of loading all 35B weights, Edge0 streams MoE experts from storage on demand, so peak memory tracks only the ~4-of-256 active experts. A trained prerouter predicts routing one step ahead to overlap loads with compute — the team reports up to +59% decode throughput.
The numbers, on a 24GB Mac mini M4 Pro
edge0-35b decodes at 14.9–17.7 tok/s with a 2.9 GiB peak; the smaller edge0-8b (built on Ling 3.0) hits ~25 tok/s in 1.0 GiB. Quality holds up — the 35B averages 79.2 across benchmarks vs 83.2 for the FP16 base, roughly a 4-point drop.
The catch: Apple Silicon only, for now
The MLX backend on M1–M4 Macs is the sole working path today; a CUDA backend is on the roadmap but not shipped. Framework and weights are Apache-2.0, and the 35B download is ~23 GB.
Build this weekend: a local 35B behind an OpenAI endpoint
`edge0 serve` spins up an OpenAI-compatible server, so you can point any existing client at a 35B-class model running fully offline with no per-token cost. A fast NVMe/flash drive is the real requirement, not VRAM.
Elsewhere: Perplexity's on-device agent reaches Windows
Perplexity Portable Computer now runs on Windows RTX PCs (24 GB+ VRAM), executing a local PPLX/Qwen 3.8 27B model wired to Outlook, Gmail, Slack, GitHub and your own local MCP servers. On-device work skips Computer credits and keeps data off the cloud — Pro/Max only.