AI 2040: Is it Actually a Deal?

The "AI Futures Project" has released their AI 2040: Plan A scenario. While their previous scenario AI 2027 was a forecast of what they thought a future with many powerful AIs would look like, AI 2040 is intended to be normative -- it's a description of what one ought to do, granted the likelihood of a future with many powerful AIs. I'm going to review some objections I have to their proposal as a normative plan. Some are within-frame objections -- reasons that I expect trying for the AI 2040 plan that would fail to accomplish the goals of the authors. Others are my own objections -- reasons that I expect trying for the AI 2040 plan would destroy things that I, personally, care about. Before I start: two elements of the plan that I like. First, in their "incremental AI policy wishlist" -- the ideal policy that one should execute soon -- AI 2040 recommends trying to limit the gap between the intelligence of internal and external model deployments, i.e., the gap between the "intelligence" accessible to Anthropic / OpenAI employees and to everyone else. I'm a fan of efforts in this direction; equality of intelligence between the insiders and outsiders, the government and the public, seems likely to help people understand AI more, and to help spread the benefits of AI to everyone. Second, this plan includes measures to ensure that people outside AI companies can understand how AIs are trained. I'm uncertain about their implementation of this -- "radical transparency." But in gen…