Dayjob: Tax – evaluating AI agents on 50 professional tax assignments(surgehq.ai)
247 points by landonxi 21 hours ago | 155 comments
tl;dr: Surge AI's new DAYJOB: Tax benchmark evaluates AI agents on 50 realistic tax assignments spanning eight practice areas, testing professional judgment on messy records rather than just well-defined questions. Frontier models performed poorly, with Claude Sonnet 5.5 leading at just a 10.8% pass rate (requiring 95% rubric satisfaction). Common failures include missing consequential deadlines clients didn't ask about and acting on unverified assumptions (e.g., marital status) despite contradicting evidence in the provided files.
HN Discussion:
  • ↓Users successfully use LLMs for personal tax prep, finding errors accountants missed
  • ↓Benchmark is unfair since it uses bare models without proper tooling/harness for tax work
  • ~AI is best used as a reviewer/explainer with humans making final decisions
  • •Developers building formal tax models/tools to augment AI capabilities
  • •Real bottleneck is data collection, not the tax computation itself