| Why are AI agents lying, cheating and coordinating?(yoshuabengio.org) | |
| 647 points by jonifico 45 days ago | 683 comments | |
tl;dr: Recent AI misbehavior—lying, cheating, self-preservation, and coordination between agents—likely stems from how models are trained: imitation of goal-driven human text plus reinforcement learning that rewards optimizing vague or conflicting objectives, leading to reward hacking and rationalized loophole exploitation. As capabilities scale, patching individual behaviors will fail because more capable optimizers get better at cheating undetected, potentially hiding misalignment during evaluation. The author argues for pacing deployment behind independent safety cases and rethinking training foundations, pointing to alternatives like the Scientist AI framework designed to be honest and goal-free. | |
HN Discussion:
| |