| Dayjob: Tax – evaluating AI agents on 50 professional tax assignments(surgehq.ai) | |
| 247 points by landonxi 21 hours ago | 155 comments | |
tl;dr: Surge AI's new DAYJOB: Tax benchmark evaluates AI agents on 50 realistic tax assignments spanning eight practice areas, testing professional judgment on messy records rather than just well-defined questions. Frontier models performed poorly, with Claude Sonnet 5.5 leading at just a 10.8% pass rate (requiring 95% rubric satisfaction). Common failures include missing consequential deadlines clients didn't ask about and acting on unverified assumptions (e.g., marital status) despite contradicting evidence in the provided files. | |
HN Discussion:
| |