Your local LLM isn't dumb — inference config is degrading it
A trending teardown shows int4 KV-cache, over-aggressive quants, and default samplers silently wreck tool-calls past 40k tokens — no new hardware required.
Copy markdown
int4 KV-cache quietly breaks tool calls
The teardown's worst offender: int4 KV-cache quantization caused uncorrectable tool-calling failures, while int8 recovered. Quantized runs “drop like a rock” past ~40k tokens — move KV-cache to int8 or fp16 before you blame the weights.
W8A16 beats FP4 — choose your quant on purpose
Across BF16, FP8, INT8, NVFP4 and INT4, 8-bit W8A16 held up best while NVFP4 hit ~50% token flips by 88k context. If you have the VRAM, an 8-bit weight quant buys real reliability over a 4-bit one.
Default samplers aren't the vendor's samplers
Benchmark scores use the model card's sampling (e.g. temp 1.0, top-p 0.95); UI defaults and temp=0 diverge and degrade output. Copy the recommended params before you judge quality.
GGUF files that silently drop the chat template
The top HN thread flags a second killer: many GGUF quants ship without the chat template, so runtimes fall back to ChatML and the model answers in the wrong format. Verify the template is embedded — or pass it explicitly.
The fix is your runtime, not a new GPU
Attention backend and tensor-parallel layout (TP1 vs TP2) also flipped tokens via NCCL differences. llama.cpp and vLLM expose the template, sampler and KV-cache knobs that Ollama hides — reach for them when output looks off.