Game-Theoretic Disempowerment: Loss of Control to Agencyless AI

Epistemic status: After writing this, I realized that it sounds a bit unhinged and I was being loose with the language. So I asked Claude to write me a good LW post that actually cited the things I was talking about. See the LLM-written section if you'd rather read Opus with citations than my rant. On my read, it faithfully represents my claims, and I do want to work on the things in Section 9. If I missed errors, help correcting them is appreciated. A kind of agency-free AI Takeover could be possible with extant AI, and therefore may have already happened to some unknown degree. A logical first place for loss of control could be a realm where human control is already imperfect, games. Unfortunately, this is a bad place to lose control, and a bad place to have to try to regain it. Most important things are in some ways contested, and the resulting control system involves competing humans and/or institutions settling into equilibria which may be far from good, but no player can unilaterally shift the system to a preferred alternative. So, game theory gives us the language to describe the loss of control that I think may be in progress. The following should be uncontroversial background: If players in a game are receiving private information about a shared random event, and they disagree about the outcome distribution, this can serve as a coordination device to allow the game to reach equilibria that would otherwise be unreachable. If that coordination device (usually called a…