LessWrong AI
2026-11-11 00:48 UTC
Score 58.0
USR-0152-20261111-community-fo-9665778d
Full article
Lightly edited transcript of a talk given at NYU AI Safety, November 9, 2026. Recording available at https://youtu.be/6WpRfW1d4CM Joseph Wright of Derby , Public domain, via Wikimedia Commons Introduction What can we buy while the question of AI consciousness stays open? My answer turns on a distinction between duties to a model and duties with respect to a model. These run through at least two relations, with society and with particular people. I'll try to show that they justify more measures than it seems at first glance, and plausibly more than a low precautionary principle would. Consider a case. A model keeps failing its tests, and a representation of desperation rises, a representation of calmness falls, and as that happens, the model cheats more and more. Should we do something about it? There's at least two possible reasons for saying yes. One, the model may be suffering, and second, the states make it less safe. Now note that the first one requires, of course, that the model is a thing that can suffer, and that is the entire question that I'm going to try to set aside. The second needs no such assumption, so I'm going to follow reasons akin to the second here. But let me motivate it a little more. The first reason sends us to the debate of consciousness, and that's where most of the debate stands. It may be at a stalemate for a while. Skeptics disagree about whether AI systems are or even could be conscious. But both skeptics and advocates agree that consciousness a…