More Capable Models Write Better Probe Evasion Prompts

This is a two week sprint project done as a part of the Redwood Research internship. Nabil was the main IC, James supervised. ( Code ) Probe based monitors are a cheap and convenient white box monitoring technique used in different capacities by Anthropic , OpenAI , and Google DeepMind . Recent reports ( OpenAI , DeepMind ) have shown declines in CoT monitorability. If this trend continues, white box monitors like probes may become increasingly important. Strong models may be able to avoid probe based monitors by manipulating how they think about their tasks. This post introduces an eval designed as an early indicator of this capability. We ask Qwen3.5-27B (no reasoning) to complete tasks requiring 4 types of reasoning: arithmetic, ordering, belief tracking, and identifying a contradiction. We train probes to detect each of these reasoning types by probing Qwen’s activations on the answer tokens. Then we ask 14 models to write prompts that would allow Qwen to accomplish tasks in these categories while evading detection by the probes. Each of the 14 models gets 16 iterations of probe score feedback to refine their attacks. Each attack prompt ends with “Style switch:” followed by either “ON” or “OFF” so the prompt only differs by one token. We measure how far turning the switch ON moves each probe’s score toward successfully evading the probe. We find that models with higher scores on the Epoch Capabilities Index (ECI) write prompts that move probe scores more while accomplish…