Foundation Models for Oversight

Cross-posted from the Transluce blog . To oversee an AI model, we'd ideally like to ask questions such as: What are important situations where the model sandbags? Does the model have an objective it wouldn't admit to if asked directly? Does the model treat a user differently once it infers something about their identity, and along what axis? Is the model's chain of thought load-bearing, or is it a post-hoc rationalization of an answer that was already settled on? Is it reward hacking on this input, or actually trying to solve the task? It would be great if we had an oversight assistant that could answer these questions. We'd want it to do three things: help us formalize the question as a testable empirical criterion; produce data that satisfies that criterion; and do so in a way we can justifiably trust . To get such an assistant, we lay out a vision for building a foundation model for oversight : an AI system mid-trained (or pre-trained) on a large, diverse corpus of experiments on a given "subject model", RLVR'd on a large number of verified oversight tasks, and then fine-tuned to answer natural-language questions about the subject model. Mid-training allows the oversight model to build rich knowledge about the subject model it is evaluating; RLVR teaches it to leverage reasoning to solve more difficult oversight tasks; and fine-tuning makes it easy to talk to the assistant and teaches it to formalize natural language questions into checkable criteria. To achieve this, we…