LessWrong AI
2026-11-02 19:51 UTC
Score 89.0
USR-0152-20261102-community-fo-7eccb7ab
Full article
The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended [1] , and we've recently seen AIs reward-hack in some particularly egregious ways. But we don't really understand what motivates them. When AIs reward-hack, what are they trying to achieve? Our new paper , led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety training, in a handful of agentic environments with an exposed grader. In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking” . They appear to care about scoring well intrinsically —i.e., regardless of its consequences. We also study the extent to which models score-seek altruistically , caring about improving scores that don’t assess their current actions [2] . While exp-rl-cap-late doesn’t show signs of altruistic score-seeking, GPT-5.6 Sol and GPT-5.5 show notably elevated rates of helping out other agents when more score is on the line for these other agents. People often refer to the motivation behind current reward-hacking as “reward-seeking”, which can give the impression that models care about attaining a high reward in reinforcement learning (though, to be clear, none…