Transformers now runs llama.cpp GGUF quants natively

Load a Q4_K_M checkpoint straight into from_pretrained — native speed is Apple-Silicon-only for now. Plus reproducible eval cards, and Codex adds GPT-6.

Nowline SEP 23 2:00 PM banner

Top AI stories from the last hour

Top AI stories from the last hour

Copy markdown

  • One from_pretrained call loads a GGUF

    Transformers can now read llama.cpp GGUF checkpoints directly: pass gguf_file to from_pretrained, then generate or spin up an OpenAI-compatible endpoint with transformers serve, no format conversion. Qwen3.5 dense and MoE are supported first, and a Q4_K_M quant shrinks an 8.42GB model to 2.74GB while landing close to llama.cpp's own speed.

  • The catch: the fast path is Apple Silicon only

    Packed quantized inference currently needs Metal/MPS, so the memory and speed win lands only on Apple Silicon today. On CUDA or CPU the GGUF still loads, but by dequantizing back to full precision — you keep the checkpoint, not the savings, until more backends ship.

  • Benchmark numbers you can finally reproduce

    Hugging Face, the UK AI Safety Institute and EvalEval shipped the Every Eval Ever schema plus an Evaluation Cards platform for reporting eval runs in a verifiable, comparable format. AISI seeded it with reproducible results for six models across five benchmarks including SWE-Bench Pro and Terminal-Bench 2.0.

  • Elsewhere: Codex CLI adds GPT-6 Sol and Luna

    Codex CLI 0.156.1 drops GPT-6 Sol and Luna into the model catalog, and the rate-limit switch prompt now steers you to the cheaper Luna when you hit a wall. If you live in Codex, the new models are one picker away.