Robustness: Backfire Risks in AI Safety and Governance

A guide to unintended consequences of AI safety interventions and how you can mitigate them. Introduction My cofounder and I are launching Impartial Projects , a studio for AI safety field strategy. We hope to start the projects most urgently needed in the AI safety ecosystem (which we understand to include AI governance). We think that the founding of new organizations (as well as increasing management capacity) are among the most effective things we can do at the moment to expand the surface area of AI safety. Surface area, it seems to me, is currently the scarcest resource. Another benefit we want to provide is our particular attention to robustly positive theories of change. We prefer to take a somewhat risk‑averse approach to AI safety, whether out of literal risk aversion or because we assign a higher probability to s‑risks than most. Interventions in AI safety are notorious for the backfire risks they harbor. Of course, that does not mean one should turn one’s back on these interventions altogether. Rather, it means that, depending on the intervention in question, more monitoring and mitigation methods are needed to minimize the various backfire risks that are part of the intervention’s risk profile. It is critical to go into this work with eyes wide open, cognizant of its benefits and risks. It’s critical to strike the right balance here. With my last startup , I made the mistake of erring too much on the side of caution and putting the greater part of a year into st…