Performance per dollar is getting faster and cheaper(wafer.ai)
354 points by latchkey 49 days ago | 135 comments
tl;dr: Wafer benchmarked GLM-5.2 on AMD's MI355X, achieving 213 tok/s single-stream and 2626 tok/s/node aggregate throughput—about 80% of B200 performance at less than half the cost. Getting there required MXFP4 quantization via AMD Quark, switching to sglang, patching two ROCm bugs to enable speculative decoding, and manually tuning the MoE kernel selection to bypass a slow fallback path. The takeaway: closing the gap with NVIDIA on AMD hardware is increasingly a matter of day-0 support rather than fundamental software limitations, and no custom kernels were needed this round.
HN Discussion:
  • Quantization to MXFP4 degrades model quality, undermining the benchmark's practical value
  • Headlines should disclose quantization since it materially changes performance claims
  • Requests additional metrics like performance per watt for a fuller comparison
  • Aggregate throughput number is misleading versus real single-stream throughput
  • ~AMD's competitiveness is unsurprising and will be surpassed by Nvidia's inference-optimized Rubin