Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses(quesma.com)
283 points by stared 2 days ago | 131 comments
tl;dr: Benchmarking Qwen3.8 27B quantizations on GPQA Diamond, IFBench, and Terminal-Bench 2.1 shows that 4-bit Q4_K_M (17GB) matches full BF16 performance and fits on a 24GB GPU, while 2-bit degrades modestly but remains usable. 1-bit quantizations collapse to random-chance performance, with longer reasoning making results worse as models exhaust token budgets. Takeaway: pick the largest quant that fits your VRAM plus context—Q4_K_M is the sweet spot, and quantization shouldn't be feared.
HN Discussion:
  • Confidence intervals are misused in the article and don't reflect run-to-run variation
  • Requests extending benchmarks to KV cache quantization and other missing quant levels like Q3
  • Personal experience confirms low quants remain usable, supporting the article's conclusion
  • ~End-to-end quality measurement is valuable but methodology around KL-divergence datasets needs more nuance
  • Real-world coding tasks show quality cliffs at higher quants (Q5/Q6) than article suggests