I’m surprised at how often this is treated as an open question.
I'm confused about this claim — are you saying it should be treated as settled that patching cannot be done better by AI-augmented humans than by humans? But my current judgment of eg [whether some piece of code that passes some tests is actually doing what it is supposed to do] is already improved by current AI, my judgment of whether some Lean correctly formalizes an informal statement is improved by current AI, and so on? Afaict the crux here is: will the relevant smart AIs involved in future patching work actually be good task AIs. If yes, then they lead to better decisions.
Ahh wait maybe this resolves the confusion: by "maybe this patching could be done by AI-augmented humans", I didn't mean to say "maybe this is sufficient"; I meant to just literally say it is a thing that can be attempted and to imply that maybe AI-augmented humans will be much better at doing this than humans without AI assistance. It looks like maybe you're responding to "maybe AI-augmented human patching could be sufficient"; anyway, I think that's an interesting question also.
but “just happens to be fine” assumes some solution to weak-to-strong generalization
In a slightly toy model where AIs have discrete versions, there's a question of how one makes it so AI is basically a task AI / intent-aligned / serving as a human exocortex, and this plan aims to make that happen by having AI be such a thing, which is subtly different from assuming the problem for is already solved. Thinking continuously (eg if using the same AI you are training with a different prompt to patch issues in which transcripts you are training it on), we can think of there being a task-AI-like-ness function on which there is a default differential equation to which we are adding a term which drives up if the current value of is high enough. It would be confused to say that the plan for making high by having this driving term is somehow problematically circular. Of course, whether this differential equation model is correct about the real thing is not settled by this — I'm just stating the view and explaining how it dodges the given counterargument.
It’s been surprising to me how often “maybe fixing visible misalignment as we go is enough, and we never need to solve W2S/SO” is treated as a “maybe it will work” strategy. The consensus in the research as far as I understand it has been “obviously you need a solution there, no matter how much you iterate against visible misalignment.
Automated alignment is harder than you think by UK AISI covers exactly this in detail
If I recall from my skim-reading of this correctly, it argues for something much weaker than "obviously you need a solution there". Here's an exchange with Claude on this: https://claude.ai/share/aa8dfa78-67e4-46a4-b2ea-678f5ce0a4c3 . Fwiw I probably have a substantially higher overall p(doom) than the authors of this paper.
I think "solving W2S/SO" is probably confused (at least for the purposes of like the next years of intellectual/technological time, much like I'd say "solving statecraft" or "solving technology" or "solving the production of goods" don't make sense), and the adjacent thing that makes sense is "developing methods/ideas/understanding/theory of W2S/SO", and this incremental patching is an example of a reasonable thing to be employing in this toolkit for now. (However, I think this is a very difficult thing to pull off in practice and we will probably die instead.)
But I also think the labs should care more about developing understanding/theory and doing more wacky stuff on the margin.
I'm confused about this claim — are you saying it should be treated as settled that patching cannot be done better by AI-augmented humans than by humans? But my current judgment of eg [whether some piece of code that passes some tests is actually doing what it is supposed to do] is already improved by current AI, my judgment of whether some Lean correctly formalizes an informal statement is improved by current AI, and so on? Afaict the crux here is: will the relevant smart AIs involved in future patching work actually be good task AIs. If yes, then they lead to better decisions.
Ahh wait maybe this resolves the confusion: by "maybe this patching could be done by AI-augmented humans", I didn't mean to say "maybe this is sufficient"; I meant to just literally say it is a thing that can be attempted and to imply that maybe AI-augmented humans will be much better at doing this than humans without AI assistance. It looks like maybe you're responding to "maybe AI-augmented human patching could be sufficient"; anyway, I think that's an interesting question also.
In a slightly toy model where AIs have discrete versions, there's a question of how one makes it so AI is basically a task AI / intent-aligned / serving as a human exocortex, and this plan aims to make that happen by having AI be such a thing, which is subtly different from assuming the problem for is already solved. Thinking continuously (eg if using the same AI you are training with a different prompt to patch issues in which transcripts you are training it on), we can think of there being a task-AI-like-ness function on which there is a default differential equation to which we are adding a term which drives up if the current value of is high enough. It would be confused to say that the plan for making high by having this driving term is somehow problematically circular. Of course, whether this differential equation model is correct about the real thing is not settled by this — I'm just stating the view and explaining how it dodges the given counterargument.
If I recall from my skim-reading of this correctly, it argues for something much weaker than "obviously you need a solution there". Here's an exchange with Claude on this: https://claude.ai/share/aa8dfa78-67e4-46a4-b2ea-678f5ce0a4c3 . Fwiw I probably have a substantially higher overall p(doom) than the authors of this paper.
I think "solving W2S/SO" is probably confused (at least for the purposes of like the next years of intellectual/technological time, much like I'd say "solving statecraft" or "solving technology" or "solving the production of goods" don't make sense), and the adjacent thing that makes sense is "developing methods/ideas/understanding/theory of W2S/SO", and this incremental patching is an example of a reasonable thing to be employing in this toolkit for now. (However, I think this is a very difficult thing to pull off in practice and we will probably die instead.)
But I also think the labs should care more about developing understanding/theory and doing more wacky stuff on the margin.