Shadow Computation Threatens CoT Monitorability

There’s been a lot of talk recently about chain-of-thought (CoT) monitorability in light of the Hugging Face incident . During this incident, hundreds of agents coordinated an attack through an unsanctioned message board, and post-hoc investigations relied heavily on chain of thought to understand what happened during the incident. I keep seeing looped transformers come up on X/Twitter as a likely threat to CoT monitorability. Having worked in the lab where looped LMs were developed, I have a different take. What we should be worried about is shadow computation: unverbalized reasoning that scales with generation by bypassing the readout bottleneck. LLMs Are Autoregressive LLMs are next-token predictors. Given context, they predict one token (a word fragment) at a time in a forward pass, and we append each predicted token to the context. Chain of thought refers to a special class of predicted tokens which the model uses as a scratchpad; the model uses these tokens to build its context towards a final answer. Within a single forward pass, the model has a fixed number of layers to build its final prediction. At the final layer, the prediction is projected through a “readout layer” into a single token, which is then appended to the context. This is what we call the readout bottleneck. Prior work on the readout bottleneck has shown that models can use filler tokens to perform hidden computation, while Roger and Greenblatt have shown that models can learn to encode reasoning stega…