How to give embedded evaluators the support they need: Lessons from the finance industry

Epistemic status: Extrapolating from my experience as a model validator at a bank testing machine learning models, a role that I see as similar to internal risk teams at AI labs, and my interactions with US Federal Reserve Bank regulators, who perform a function that has a lot in common with the proposed embedded evaluators. All views expressed here are my own. Summary: Dario Amodei’s We Must Pace the Frontier proposes a three-step plan to pace AI progress: (1) invite independent review (embedded evaluators), (2) set industry standards enforced by regulation, and (3) seek international cooperation. Drawing on risk management experience in the financial industry, I suggest how to make step one more effective while we wait for step two. In banks, the embedded evaluator role is split between internal teams and external supervisors. Following this approach, AI labs would first strengthen their internal risk management teams by giving them independence and ability to block testing and deployment. Embedded evaluators would start by assessing risk team effectiveness, saving independent testing for the most critical safety checks. As a placeholder for regulatory enforcement powers, they would be given a direct line to the Board with any decision to override them published. Lastly, both internal risk teams and embedded evaluators would be trained in risk management and supervision. These steps would strengthen risk oversight while introducing a natural pacing mechanism. I conclude by…