A 28.9M-param LLM now runs on an $8 ESP32 microcontroller
It writes coherent short stories at ~9.5 tok/s, fully offline — by parking a 25M-param table in flash. What it can and can't do, and how to run it.
Copy markdown
What just fit on $8 of silicon
Developer slvDev got a 28.9M-parameter transformer generating text on an ESP32-S3 — a ~$8 microcontroller with 512KB SRAM, 8MB PSRAM and 16MB flash — at roughly 9.5 tokens/sec, fully offline. The 4-bit model weighs 14.9MB and lives in the board's own flash. For scale, OpenAI's first GPT was 117M params.
The trick: an LLM is mostly a lookup table, not math
25M of the 28.9M params are an embedding table, so slvDev borrowed Google Gemma's per-layer-embeddings idea and memory-maps that table straight from flash (execute-in-place), reading only ~450 bytes per token. That leaves just a ~560KB dense core in fast RAM — the technique, not a smaller model, is what makes it fit.
Be clear-eyed: it's a story generator, not an assistant
It's trained on TinyStories, so it writes short, coherent tales and can track simple variables — but it won't follow instructions, answer questions, write code, or know any facts. Treat it as a proof of what fits on the metal, not a pocket ChatGPT.
Why it changes what you can build
A self-contained language model on an $8 part — no GPU, no PSRAM cluster, no network — opens battery-powered, fully-offline text gadgets: ambient story toys, privacy-local narrators, disconnected wearables. And the flash-resident-embeddings pattern generalizes, so bigger tiny-LLMs on cheap edge chips just got more plausible.
Run it this weekend
The full stack — firmware, training code and ablations — is public at github.com/slvDev/esp32-ai; wiring and flashing steps are in firmware/esp32_llm/README.md. You'll need an ESP32-S3 with 8MB PSRAM and 16MB flash.