Separating signal from noise in coding evaluations

	Separating signal from noise in coding evaluations(openai.com)
	237 points by sk4rekr0w 63 days ago \| 92 comments
	tl;dr: Summary not available
	HN Discussion: ↑Benchmarks are widely flawed and the article's critique reinforces known issues with SWE-Bench and similar evals ↑Labs and submitters game benchmarks through timeouts, harness cheating, and reward hacking ↓Rather than discarding the benchmark, the authors should have patched the messy tasks to make them usable ↓Messy, ill-specified tasks reflect real-world software work, so the complaint is unsympathetic •Suggests new evaluation approaches like cost-bounded benchmarks or accounting for unknown problems