Mini-AGI: a byte-level model that keeps learning on one 8GB GPU

It trains from scratch on a laptop GPU and reads text in a nonstop batch-1 stream, holding 99.8% of what it learns — no fine-tune cycles, no forgetting.

Nowline SEP 22 5:00 AM banner

Top AI stories from the last hour

Top AI stories from the last hour

Copy markdown

  • It reads, and it doesn't forget

    Mini-AGI trains a language model from scratch on a single 8GB GPU, feeding it one continuous batch-size-1 stream of text and updating on every character it reads. The author reports it keeps 99.84% of its prior progress when it switches to a new subject — the catastrophic-forgetting problem that normally forces you to freeze a model or retrain from a checkpoint.

  • 468M params, only 101M on the card

    It's a byte-level mixture-of-experts — 256 raw byte values, a 4,096-character context — that grows and prunes its own experts as it learns. It currently sits at ~468M parameters but keeps only ~101M resident on the GPU, paging the rest to disk at ~2.4 GFLOPs per byte. That's how a ~470M-param model trains on an RTX 3070 laptop instead of a rented cluster.

  • The catch: no weights to download yet

    This is one developer's research repo, not a packaged model. The code is public but the network is still on its first pass over the corpus, so there's no checkpoint to grab and every figure here is self-reported by the author. Treat it as an architecture to clone and train yourself, not a model to drop into production this week.

  • Build this weekend: a model that learns your stack

    The unlock is a small model that absorbs your own data as a live stream — notes, a codebase, chat logs — and keeps improving without a fine-tune run or a RAG index in front of it. Clone the repo, point it at a folder, and let it read overnight on a gaming GPU you already own.