Anthropic should measure the effect of evaluation awareness

... and report their findings! Summary Claude Fable 5.2 & Mythos 5.2 are the most situationally aware of Anthropic's recent models according to their own internal investigations. The company has made a concerted effort to measure evaluation awareness and to acknowledge that it affects their confidence in model safety. But their latest system card lacks information on the effect of evaluation awareness, thus blurring how one ought to adjust their own confidence in Anthropic's alignment assessments. Specifically, I present the following claims regarding Claude Mythos 5.2's system card : The prevalence of evaluation awareness in audit transcripts is surprisingly unclear due to confusing wording and incomplete disclosure of results. Anthropic offers little to no estimation of how evaluation awareness changes Mythos 5.2's behavior - neither direction nor magnitude. I offer concrete recommendations at the end of this post. Background Evaluation awareness refers to the ability of an AI system to recognize when it is being tested. Evaluation awareness can be verbalized (the recognition is stated in its output or chain-of-thought) or not (only visible internally). Unless specified otherwise, my later references to evaluation awareness include both cases. Evaluation invariance is the extent to which a model's behavior remains consistent regardless of whether it believes it is being evaluated. The opposite might be called evaluation sensitivity . Alas, scaling general AI capabilities h…