Gigatoken: drop-in tokenizer runs ~1,000x faster than HuggingFace
The Rust library hits 24 GB/s on server CPUs — days of pretraining data prep collapse to hours. Plus Microsoft's 4B MIT image model, Mage-Flow.
Copy markdown
A drop-in tokenizer that runs ~1,000x faster
Gigatoken is a new MIT-licensed Rust library with Python bindings that tokenizes text at gigabytes per second — roughly 280x to 1,353x faster than HuggingFace's tokenizers. Install with `pip install gigatoken`; a compatibility mode reproduces HuggingFace and tiktoken output exactly for GPT-2, Llama, Qwen, DeepSeek, Gemma and Mistral, so you can swap it in without changing your data format.
The numbers, per machine
On a 144-core EPYC it clocks 24.5 GB/s on the GPT-2 tokenizer versus HuggingFace's 24.8 MB/s; an M4 Max laptop hits 8.8 GB/s, a 1,268x jump. The author's benchmark: you could tokenize the entire public internet in under 7 hours on a single box — data-prep runs that used to take days across a CPU fleet now finish in an afternoon.
Where the speedup actually lands
Be clear-eyed about the win: tokenization is under 0.1% of inference time, so Amdahl's Law caps end-to-end gains for serving. The real payoff is data pipelines — pretraining corpora, high-throughput classification, and prefix-cache boundary lookups — where tokenizing is the bottleneck. Going through Python in compat mode nets ~200-300x; the native API delivers the full number.
Microsoft ships Mage-Flow, a 4B MIT image model
Microsoft also dropped Mage-Flow, a 4-billion-parameter text-to-image and instruction-editing model under MIT, with checkpoints on Hugging Face plus a Python API, CLI and Gradio app. It's tagged research-only for now, and a 4B vision-language sibling (Mage-VL) is 'coming soon' — but it's a cheap, permissive base to fine-tune your own image-editing workflow on.