Frontier models may not be reliable auditors

Summary Frontier model providers are employing their models internally in order to iteratively make future models safer. For instance, Anthropic uses automated audits to identify misalignment concerns prior to releasing a model. More effective auditors, holding their own alignment fixed, should necessarily produce more informative audits. But how effective are these auditors? We can test their various safety-related capabilities on known misalignment modes, but we are uncertain of the total space of misalignment modes. Can we be sure that auditing ability generalizes? Here, I scrutinize the general open-ended investigative ability of frontier agents in a toy evaluation and offer evidence against current models as generally reliable auditors. This case is interestingly one in which a AI capability deficit poses a safety concern. Note: the most advanced models tested in this study are Claude Fable 5.1 (2026-09-01) and GPT-6 Astra (2026-09-03). Background Iterative Alignment The frontier's vision for the future is ostensibly for increasingly capable models to help achieve alignment. As publicly stated, this plan ranges from a weak version focused on model evaluation and oversight and a strong version aspiring for an end-to-end automated alignment researcher. [1] While this framing is loose, the throughline is legible enough: use AI to make AI safer. This broadly accepted blueprint relies (among other things [2] ) on AI systems being capable enough to contribute to safety-relate…