Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases(withspecific.com)
248 points by theanonymousone 16 hours ago | 137 comments
tl;dr: Real-SWE benchmarks frontier AI coding agents on tasks pulled from real, private enterprise codebases—work that's never appeared in training data and carries actual business consequences like billing, tax, and migrations. Top performer Fable 5.1 (via Claude Code) resolved just 38.8% of tasks, with 6 of 10 sampled tasks scoring under 15%; the most common failure mode across models was missing requirements, followed by unverified assumptions. Per-rollout costs ranged from $2.50 to $6.96, suggesting current models remain far from meeting enterprise engineering standards.
HN Discussion:
  • Personal experience confirms LLMs struggle with business logic in real private codebases
  • ~Skepticism that these 'private' codebases are truly uncontaminated from training data
  • Benchmark results don't match personal experience or other benchmarks, missing crucial methodology details
  • Shares methodology insights from building similar private benchmarks
  • Model rankings in the benchmark align with hands-on usage experience