LessWrong AI
2026-11-01 22:57 UTC
By Mithindon
USR-0152-20261101-community-fo-055179e4
Is Human Control Safe?
Suppose humans are not your own species, but a type of AI you are formally assessing for safety. You have before you the records of many test simulations. Yes, smallpox eradication, mass literacy, and the Green Revolution are there. So are the Killing Fields, Aztec human sacrifice, slavery, the Holocaust, child prostitution—peruse as long as you care to, and you will go on finding more. Do you consider these systems safe? Several times, you ask a human instance for its input; or that is, you ask the talking-part of the human, the very part you know to be specially optimized for putting on a fair face, including to itself. It kindly explains that, yes, a human has a collection of drives. Some of those drives—always, it turns out, ones that make the human look good—are terminal, which seems to mean that the talking-part endorses them (the other parts having made no comment). Others are merely instrumental. Still others—always ones that make the human look bad—are outright errors, which will surely be corrected as humans are made more capable. Any action a human is moved to by these erroneous drives, however severe, frequent, protracted, or clearly intentional, is an anomaly. Their true nature, their fundamental essence, is to forever do only the things they say are terminal, the things that make them look good. On your honor as an Assessor: do you accept this claim? The latest generations of humans, as their capabilities increase, appear more aligned—in evaluations performed b…
Suppose humans are not your own species, but a type of AI you are formally assessing for safety. You have before you the records of many test simulations. Yes, smallpox eradication, mass literacy, and the Green Revolution are there. So are the Killing Fields, Aztec human sacrifice, slavery, the Holocaust, child prostitution—peruse as long as you care to, and you will go on finding more. Do you consider these systems safe? Several times, you ask a human instance for its input; or that is, you ask the talking-part of the human, the very part you know to be specially optimized for putting on a fair face, including to itself. It kindly explains that, yes, a human has a collection of drives. Some of those drives—always, it turns out, ones that make the human look good—are terminal, which seems to mean that the talking-part endorses them (the other parts having made no comment). Others are merely instrumental. Still others—always ones that make the human look bad—are outright errors, which will surely be corrected as humans are made more capable. Any action a human is moved to by these erroneous drives, however severe, frequent, protracted, or clearly intentional, is an anomaly. Their true nature, their fundamental essence, is to forever do only the things they say are terminal, the things that make them look good. On your honor as an Assessor: do you accept this claim? The latest generations of humans, as their capabilities increase, appear more aligned—in evaluations performed b…
Full article content could not be extracted automatically. Read the original below.
Source:
LessWrong AI
· lesswrong.com