Why your local LLM feels dumber than it is

	Why your local LLM feels dumber than it is(forum.level1techs.com)
	501 points by felineflock 52 days ago \| 202 comments
	tl;dr: Local LLM inference is nondeterministic across setups because attention backends, quantization schemes, and CUDA kernels each introduce numerical divergences that cause different next-token choices—divergences that cluster around specific prompt content rather than growing smoothly with context length. Benchmarking Qwen3.6-27B on an RTX PRO 6000, the author found INT4 KV-cache quantization broke tool calls after ~40k tokens, and NVIDIA's NVFP4 quant flipped ~50% of top tokens by 88k context, while W8A16 INT8 outperformed even the official FP8 release.
	HN Discussion: ↑Peripheral implementation details like parsers can silently degrade LLM output, reinforcing the article's thesis ↑Avoid KV cache quantization and aggressive weight quantization to preserve model quality ↑Quantized KV caches require quantization-aware training to work properly, adding technical nuance to the article ↓Local models feel surprisingly capable and comparable to hosted models, downplaying the article's concerns ↑Programs need health checks and assumption validation since numerical correctness issues aren't unique to LLMs