For example, imagine a model that reasoned 90% in CoT (i.e., between
My toy conceptual model is something like:
The model has three regions of its brain: one for CoT, one for tool calls, and one for final outputs.
The code comments might be generated in the “CoT region”
When you apply training pressure to the code comments, this is applying training pressure to the “CoT region”
This post reflects my personal opinion and not necessarily that of other members of Apollo Research.
It seems plausible to me that models don’t really distinguish between “CoT” and “things that are similar to CoT, i.e. in the same style”. In other words, the LLM doesn't have separate features for 'tagged as reasoning' and 'sounds like reasoning'.
For example, imagine a model that reasoned 90% in CoT (i.e., between
My toy conceptual model is something like:
(This applies to more than just code comments.)
Prior work: Output Supervision Can Obfuscate the Chain of Thought makes a very similar argument and provides empirical support.
Thanks to several people at Apollo for their feedback.