1440p version of the image used in many "The world if" memes :  r/MemeRestoration

What sort of futures can we expect for civilization after AGI?

Ideally, our future satisfies all of these properties:

tazik's blog is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Surprisingly, while current visions meet some of these criteria, none fully satisfy all of them.

Let’s explore some of the most commonly presented post-AGI futures.

type 1: nice but unstable

post-scarcity

  • If labor is no longer scarce (because of robotics explosion), wages will collapse
  • Resources that are currently in surplus will eventually be contested (e.g. land, energy)
  • Even if we (humans) don't fall into a Malthusian trap, AIs (embodied or not) will replicate at a faster rate, leading to scarcity
  • Example: The sun is our primary source of energy. We are 13 doublings away from using up all of the sunlight hitting Earth (1.7x10^17 W). And 44 doublings away from needing all of the Sun's output (3.8x10^26 W)
Source: Post-AGI Civilization Talk, David Duvenaud

Post-scarcity sounds appealing, but is incoherent upon inspection.

laissez-faire

  • Idea
    • We can still have robust trade between humans and AI even when they are abundant in the physical and digital world
    • Concretely, rich humans will want services from only humans. Example: horse-drawn carriages
  • Problem
    • This will necessarily be a niche market, and a tiny fraction of the total economy (to see this: consider what % of the economy is currently occupied by this category)
    • In most of the economy, competition is wont to drive wages below human subsistence
    • This leads to human loss of cultural, political, and hard power


type 2: stable but bad

locusts


File:Foster Bible Pictures 0074-1 Offering to Molech.jpg

Source: Can goodness compete, Joe Carlsmith

  • Problem: Goodness may not be robust to competition (Joe Carlsmith, 2025)
    • The winner of competition is power
    • Power is not the same as goodness (~orthogonal)
    • So every opportunity to sacrifice goodness for power makes you more likely to win (we see this in human society)
  • So if the future involves a lot of competition (does it?)... goodness loses
  • Can goodness compete?
    • Locusts aren’t entirely foreign; human civilization is a locust in some ways
    • Power might be dominant, yet we still managed to select for (or at least, not entirely select away)
      • friendship
      • loyalty
      • art
      • children’s laughter
      • play
    • IMO, some of the above feel like side effects, or local instantiations of phenomena that are globally advantageous (e.g. friendship, loyalty are instances of cooperation, so it’s not that surprising that they’d develop if we select for coordination)
    • But art and play do feel harder to argue for on instrumental grounds (yay for terminal goods)
  • This would be a plausible and stable but not good future

Source: Post-AGI Civilization Talk, David Duvenaud

totalitarian stasis

Idea:

  • Let’s say the government stops or pauses AI development, so as to keep humans competitive

Problems:

  • In the limit, you cannot stop AI development without implementing a panopticon and surveilling all your citizens (it seems possible to stop large scale development, but perhaps not smaller scale ones, without devolving to an Orwellian state. But this is not robust to development: one can imagine underground data centers, or even decentralized AI training)
  • Similarly, if the state has legal precedent for imposing restrictions on progress, we could see this lead to encroachment of other freedoms e.g. reproductive rights (tbh, I’m not sure I fully see the argument here)
  • If it is the case that if anyone builds it, everyone dies, then we need global governance for stability!
    • There will then be no freedom, even in Mars colonies (at temporal limits)
  • North Korea is an example of totalitarian stasis
  • This would be stable, perhaps, but not nice


type 3: stable, nice, but plausible?

benevolent singleton

Source: Coherent Extrapolated Volition, MIRI

Idea:

Problems:

  • You need to load our values correctly and really nail it before “The Handoff”
    • Incredibly hard to robustly validate all possible failure modes
    • “Need to grab power before reflection” (not sure what he meant by this)
  • Powers-that-be will fight challengers
    • It’s not hard to imagine the government, once AGI-pilled, asks/demands to write Claude’s Constitution

Overall, this is stable and good, but may not be plausible (or is at least an unknown unknown).

summary

Existing visions of futurism by philosophers and tech company CEOs are unlikely, non-robust, or not good for humans.

green = yes, red = no, grey = maybe

But we can and should conceive of a plausible, stable, and desirable future for human civilization.

credits

This blogpost is based on David Duvenaud's excellent talk at AI Safety Week 2026 in Toronto. If you enjoyed this, I highly recommend checking out: https://post-agi.org/