Epistemic Sudden Death

When sports go into overtime, they sometimes use "sudden death" format. Instead of gaining points incrementally, the next person to score instantly wins and the other guy instantly loses. "Epistemic Sudden Death" is when you've been accumulating some evidence or orienting to a situation, and now, you find yourself in a situation where you could gain decisive evidence about what world you're in at any time. You don't expect intermediate observations between the current state, and whatever it is you're expecting. In the case of AI, when you meet those observations, you also might suddenly [1] die. The current AIs are not takeover capable. I don't have that confident a take on whether timelines are a few months vs a few years or decades. [2] But there are no observations my models say I need to observe between now and "they are takeover capable." It could happen in during next 10x or 100x scaled training run. Then, some time later as the nanites dissect my body I'd be more like "yep, that makes sense" than "wowzers I am surprised." I think we're in a Sudden Death scenario. I'll get into why I think that a bit later. For right now, I'll note that Paul Christiano (previously the crowd-appointed vague figurehead of "takeoff is probably both smooth and takes decades") said he didn't expect to necessarily see any new capabilities between now and takeoff capable AI. [3] And, Ajeya Cotra [4] wrote two months ago on the Hugging Face incident. Compared to these reward hacks from six mon…