The potentially deadly threat of AI output-optimization

“You do not rise to the level of your goals. You fall to the level of your systems.” — James Clear, Atomic Habits “We cannot improve results ‘directly.’ It can only be achieved by improving those processes [...] that produce results.” — Orest Fiume The much discussed “ P(doom) ” question asks about the likelihoods we’d ascribe to the most lethal possible futures of AI. What happens when we invert this question? It transforms into the presumably less urgent, though I’d say still important, form: What are the most likely lethal AI behaviors? Not the destroy-the-entire-world sort of behaviors, but an-AI-has-screwed-up and now a-Boeing-737-has-fallen-out-of-the-air sort of scenarios. (If I’m ever conscripted into a war for survival against Terminators, I can imagine my hypothetical soul resting easy having fought the good fight. If I ever die in a mundane accident thanks to a stupid LLM hallucination, on the other hand, I’m going to leave behind a beleaguered ghost pedantically and disproportionately vexed to an extent never yet witnessed.) My answer to the above question is twofold. If we’re talking about the near future, I think the most likely lethal AI behavior is something I’ve started to call “output optimization” or “output without process” (and a highly related “input over-indexing”). This is the concern I’d like to signal-boost with this post. However—since the lethality of output optimization is at the moment entirely theoretical—it’s probably worth first answering, “D…