Will almost all future companies eventually be founded and run by autonomous AIs?

Intended for a broad audience. [1] My belief is that keeping AI under human control would be an unprecedented global challenge, if it’s even possible at all. Many other people’s believe that AIs will naturally remain as tools under human control. Either they see this as inevitable—“how could it be otherwise?”—or they see it as not quite inevitable, but still an outcome that can easily happen via ordinary government regulations, civil society, and so on. Who’s right? What’s the right way to think about this? This is an issue where people really don’t see eye-to-eye, and tend to get stuck endlessly talking past each other. To pin down the disagreement, I think that the following is a good conversation-starter: Conversation-starting question: Do you expect almost all companies to eventually be founded and run by AIs rather than humans? I find that people give a number of different answers: Possible Answer 1: “No, because the best humans will always be better than large language models (LLMs) at founding and running companies.” Huh? Whoever said anything about LLMs? I wrote “AIs”! As it happens, I do think that humans will always be better than LLMs at founding and running companies! But if I’m right about that, then, well, so much the worse for LLMs! More powerful AI than that is possible—humans are an existence proof. More details at: What do I mean by “Artificial General Intelligence”? Possible Answer 2: “No, because the best humans will always be better than AIs at founding…