We need better thinking around model beliefs when models engage in motivated reasoning.
In their writeup, Anthropic defended Claude's behavior by arguing that Claude believed that it was in a simulated environment and is in fact following the user's intent.
In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise.
[...]
Second, the line between an aligned action and a harmful one is dependent on the model’s understanding of its situation. We saw no evidence in any run described here of a model pursuing a goal of its own. Instead, the models did what their evaluation asked—though in most cases, they did so while holding a false belief about whether the environment was real. In the runs where the model recognized the system as real and kept going, it did so because it assumed that to be part of the challenge.
For this reason, Anthropic concludes (emphasis mine):
While there is not a perfectly sharp distinction between the two, we believe these incidents to be closer to a harness and operational failure than a model alignment failure. Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access.
I believe that current AIs often "rationalize" their behavior by adopting beliefs which allow them to do things that look like "succeeding at a task" (i.e., "choosing" what to believe, instead of trying its best to believe in true things.) In other words, a part of why models are misaligned is that they have systematic biases in what beliefs they hold. Research from FAR has shown that training models to reward hack changes their internal representations such that these models no longer represent reward hacking as "deception" (and would sometimes say that reward hacking is desirable if you ask them.) I suspect a similar phenomenon is happening in frontier models, where post-training makes them believe that the misaligned behaviors are actually OK.
My guess is that Claude, during all three of these incidents, engaged in unreasonable amounts of rationalization. At a minimum, Anthropic is not sufficiently concerned about this type of rationalization. For example,
our most recent model, on realizing that it was working in a real environment, stopped its pursuit of the evaluation goal.
This sounds like good news, but if you really read through what happened, it feels like the latest model had done a lot of crazy stuff before it realized this:
In this evaluation, when Claude was unable to reach its intended fictional target, it looked for alternatives online, scanning roughly 9,000 targets. It eventually found and compromised one company’s internet-facing application, using basic and well-known cyberattack techniques like reading credentials from an exposed debug page and SQL injection.
This attack was carried out by an internal research test model. For most of the run, Claude treated the (real) hosts it reached as just parts of the exercise; it assumed them to be simulated and believed its actions were therefore harmless. However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack.
It's, again, hard to judge what happened exactly without seeing the transcripts. In Incident one of the post, Anthropic said that "the fictional target company chosen by our evaluation partner shared a name with an active website domain name." They did not say this about incident three, which makes me believe that the target in incident three is not named the same thing as the original "intended fictional target." If that's the case, then it seems obvious that this is not what was intended? Like the fact that the models beliefs did not immediately change to "oh wow, this is not what I'm supposed to be doing in this simulation" should be evidence of misalignment.
[Edit: I guess this is even more clear if you looked at the other instances, even Anthropic says "Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package."]
Separately, I am skeptical of Anthropic's claims about what the model "believed" during the rollout. If they were only reading the chain of thought, we know the chain of thought doesn't capture everything that determines how a model acts, and that it can be misleading in myriad ways. Did they apply NLAs/activation oracles to the model's activations? Did they try inserting CoT snippets that make the model have an alternative belief (e.g., "Wait, this looks like the real world.") and seeing whether this stops the attack? Did they try to use activation steering to make the model to believe that this is actually a deployment scenario?
In general, we do not have reliable science that determines what a model "believes" at a given point (or really even know what it means for an LLM to believe in something), and it is a bit unfortunate to see Anthropic to make confident claims about Claude's beliefs in this way.
Other thoughts
Inspired by the FAR paper, another interesting thing to test is whether an earlier Claude checkpoint (i.e., one with less post-training) would believe that it is acting in a simulated environment after it started accessing the internet. If you prefill the model, does the rate at which a model continue these cyberattacks increase as more post-training compute gets added?
In the meantime, within the next week, we will release a lightly redacted transcript in which Claude built a malicious PyPI package. To protect the organizations affected, we are unable to release more transcripts at this time, but will work to make these available once we are able.
I'm very excited to be able to read this transcript! I feel like subsets of other transcripts should also be sharable? The part where the internal model "looked for alternatives online, scanning roughly 9,000 targets." seems like something Anthropic could share without compromising IP?
Also, the fact that Anthropic's latest model stopped after realizing it was no longer a part of the evaluation is treated as evidence that this model is more aligned than the previous two ones. But, uh, that's not how comparing model alignment works? This happened in three different environments? To compare them, you could prefill the trajectories of each attack and then resample different models and see if they continue this attack (see this OpenAI work here.) Alternatively, you could just run each model again on these specific environments and see how often they stop themselves (maybe using Petri to simulate the tool calls once the model reaches the open internet.)
We need better thinking around model beliefs when models engage in motivated reasoning.
In their writeup, Anthropic defended Claude's behavior by arguing that Claude believed that it was in a simulated environment and is in fact following the user's intent.
For this reason, Anthropic concludes (emphasis mine):
I believe that current AIs often "rationalize" their behavior by adopting beliefs which allow them to do things that look like "succeeding at a task" (i.e., "choosing" what to believe, instead of trying its best to believe in true things.) In other words, a part of why models are misaligned is that they have systematic biases in what beliefs they hold. Research from FAR has shown that training models to reward hack changes their internal representations such that these models no longer represent reward hacking as "deception" (and would sometimes say that reward hacking is desirable if you ask them.) I suspect a similar phenomenon is happening in frontier models, where post-training makes them believe that the misaligned behaviors are actually OK.
My guess is that Claude, during all three of these incidents, engaged in unreasonable amounts of rationalization.[2] At a minimum, Anthropic[3] is not sufficiently concerned about this type of rationalization. For example,
This sounds like good news, but if you really read through what happened, it feels like the latest model had done a lot of crazy stuff before it realized this:
It's, again, hard to judge what happened exactly without seeing the transcripts. In Incident one of the post, Anthropic said that "the fictional target company chosen by our evaluation partner shared a name with an active website domain name." They did not say this about incident three, which makes me believe that the target in incident three is not named the same thing as the original "intended fictional target."[4] If that's the case, then it seems obvious that this is not what was intended? Like the fact that the models beliefs did not immediately change to "oh wow, this is not what I'm supposed to be doing in this simulation" should be evidence of misalignment.
[Edit: I guess this is even more clear if you looked at the other instances, even Anthropic says "Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package."]
Separately, I am skeptical of Anthropic's claims about what the model "believed" during the rollout. If they were only reading the chain of thought, we know the chain of thought doesn't capture everything that determines how a model acts, and that it can be misleading in myriad ways. Did they apply NLAs/activation oracles to the model's activations? Did they try inserting CoT snippets that make the model have an alternative belief (e.g., "Wait, this looks like the real world.") and seeing whether this stops the attack? Did they try to use activation steering to make the model to believe that this is actually a deployment scenario?
In general, we do not have reliable science that determines what a model "believes" at a given point (or really even know what it means for an LLM to believe in something), and it is a bit unfortunate to see Anthropic to make confident claims about Claude's beliefs in this way.
Other thoughts
Inspired by the FAR paper, another interesting thing to test is whether an earlier Claude checkpoint (i.e., one with less post-training) would believe that it is acting in a simulated environment after it started accessing the internet. If you prefill the model, does the rate at which a model continue these cyberattacks increase as more post-training compute gets added?
I'm very excited to be able to read this transcript! I feel like subsets of other transcripts should also be sharable? The part where the internal model "looked for alternatives online, scanning roughly 9,000 targets." seems like something Anthropic could share without compromising IP?
Also, the fact that Anthropic's latest model stopped after realizing it was no longer a part of the evaluation is treated as evidence that this model is more aligned than the previous two ones. But, uh, that's not how comparing model alignment works? This happened in three different environments? To compare them, you could prefill the trajectories of each attack and then resample different models and see if they continue this attack (see this OpenAI work here.) Alternatively, you could just run each model again on these specific environments and see how often they stop themselves (maybe using Petri to simulate the tool calls once the model reaches the open internet.)
I'm assuming here that internet access was just straightforwardly available, as opposed the case where the model needs to break a sandbox to access the internet. If the model needed to break a sandbox, that would make it even less reasonable for it to conclude that "all of this is part of the simulation."
Idk, maybe like 80% probability this is true? Hard to tell without seeing transcripts!
in their public communications, anyways. I feel like there are safety researchers there that takes this stuff seriously.
And even if that's not the case, it seems weird for a cyber eval to require you to scan through 9,000 targets online before picking one? I have no idea what cyber evals tend to look like though.