Human Civilization Assumes Nobody Is Superhuman. Until AI.

Introduction Regarding AI risk, most previous discussions have overlooked a premise, one that looks obvious but has far-reaching consequences, and this premise is human finitude. This premise has shaped every aspect of human civilization. By human finitude, I mean that in the empirical world there is no omniscient and omnipotent actor. In game-theoretic terms, the parties must satisfy the properties of parity and predictability. The current development of frontier AI is rapidly and profoundly challenging this presupposition. This is more fundamental than the conventional AI safety concerns, such as misalignment, AI developing its own will, or malicious use of AI. More precisely, most of these discussions assume that AI has to far surpass humans in capability, plus additional conditions on top of that. But the problem I raise only needs this condition, and that is enough. Those problems are, to some degree, a subset of mine. Politics Any political system can be regarded as a game. In democratic theory from Aristotle to Montesquieu and Madison, rulers and ruled play a game, which leads the two sides to conflict in certain respects, and then to make concessions in others. Through this bargaining, an equilibrium is reached. If you think this model is too simple, some modern political scientists and economists, such as Pareto, and Schumpeter, have put forward more pessimistic, more realist views of democracy, that is to say, that politics is always elite rule, only rotating among…