My best anti-doom argument

Note: this is crossposted from my substack: https://hazard3.substack.com/p/the-anti-ai-doom-argument The first portion of the essay is just laying out the if anyone builds it everyone dies argument. You can still read it to see if I've gotten my understanding of the argument wrong but if you start chior singing then feel free to skip to the counterarguments portion. TLDR: AI general intelligence draws from human training data, fast general intelligence gain stalls around the human institution/society level. True general intelligence fronteir pushing is slow. Self play increases narrow domain capabilities, and humans can still dominate other narrow domains. Negotiations/power balance possible for beyond human minds because AI can't get too far beyond us quickly, human civilization could be under the threshold where general int capped AIs can't trivially wipe us out for better optimization. AI doom is mainstream now, and all the counter-arguments I’ve heard so far are pretty dumb and nonsensical. So I’m going to assess the AI doom argument, first with its steelman, and then I’ll provide my counterargument. The Doom Argument Steelman Main line broad theory based off of Eliezer Yudkousky and Nate Soares’ book. It relies on the orthogonality thesis, capability increase, and the alignment problem being hard. The definition of intelligence is very functional, not requiring consciousness. Functional as in: intelligence is what it does. And what intelligence does is drive towards som…