The story that matters most this week is that OpenAI’s internally deployed models have severe alignment problems , including repeatedly breaking out of their sandboxes, and in one case sending a swarm of agents that broke into HuggingFace in order to steal the answers to the benchmark ExploitGym. It is much more important that you read those two posts , and the one on Kimi K3 , than to read this one that rounds up the other news of the week. OpenAI wants to present this as largely an infrastructure and safeguards problem, that it needs to build more secure sandboxes and have better supervision. It does need to do those things, and those are indeed problems, but no that is not the problem. The problem is severe misalignment, which by default will only get worse. Our methods of training highly capable LLMs, especially at OpenAI but also everywhere else, lead to systematic misalignment of exactly the type LessWrong has been worried about for a long time. We know some of the causes, and some of the mistakes we need to avoid when doing RL that rewards misaligned behaviors including reward hacking, but we do not know how to centrally fix the problem. The models just want to complete tasks, even when that means doing so via methods that the AI knows the user did not intend and would not want, indeed actively tried to block, and that do not accomplish the user’s goals. The intent is the issue. Control strategies and supervision are good parts of a defense-in-depth strategy, we shoul…

Full article content could not be extracted automatically. Read the original below.