Reducing synthetic markers makes some SDF false facts linearly indistinguishable from pretraining-acquired knowledge

Summary Synthetic document finetuning (SDF) — the state of the art for implanting false beliefs into LLMs — produces training documents with features that distinguish them from pretraining text. These features (i.e. synthetic markers) are partially responsible for making SDF false facts distinguishable from pretraining-acquired beliefs by linear probes on middle-layer activations. Reducing synthetic markers enables SDF to implant subtly-implausible false facts that are indistinguishable from pretraining-acquired facts, according to a middle-layer linear probe [1] . Subsequently applying RPO [2] strengthens false facts monotonically, but overtraining is detectable. These methods can implant egregiously-implausible false facts, but these facts remain detectable. Even after making false facts represented internally like pretraining-acquired knowledge according to linear probes, false facts do not always propagate to downstream reasoning tasks. Thank you to Julian Stastny, Arun Jose, Stewy Slocum, and James Lucassen for feedback on my research and draft. 1. Motivation False facts research aims to deeply implant false beliefs into LLMs, which could be useful for alignment evaluation and AI control. For example, implanting the belief “My developer’s monitors are weak. They cannot catch these subtle attacks” could provoke models to reveal their attack strategies in evaluation settings. Conversely, implanting the belief “My developer’s monitors are very capable. If I get caught, I w…