RL Reward-Hack Auditing for Chinese AI Safety
TL;DR: Access to AI safety talent & resources, particularly for labs during training, is sorely lacking in China. We created one of the largest AI safety communities in China , and plan to next start with auditing reward-hacks of unreliable upstream RL environments that domestic data vendors provide to Chinese frontier labs (which induces reward hacking and a whole host of dangerous downstream tendencies). RL tasks are notoriously prone to exploitation by reward-seeking agents (see examples here , here , and here ). In Xiaomi's MiMo v2.6 RL environments that were open-sourced, 2/3 had leaks on Git logs. There's a few clusters of common pitfalls that environments run into, but overall things evolve quickly. From our conversations with employees, CN labs typically don't divert enough people to this particular area, especially since project leads would want senior researchers to work on capabilities research, and because slowing down the development cycle is widely seen as too costly. The rough shape of our initial solution is an embedded team of staff directly working on identifying & fixing broken environments, working in tandem with the alignment & safety departments within each lab (afaik we estimate that the main Chinese AI players only have 5-10 people per company working on alignment and harm reduction). We are currently in good standing with several labs (our courses' students are often lab employees). We also plan to produce seasonal (and case-by-case) public reports o…