The user in the LLM might be a moral patient too (a strange implication of the PSM and simulators)

Thanks to Christopher Ackerman, Margot Stakenborg, Darion Cassel, Andy Han, and Ariana Azarbal for feedback and/or valuable discussions on these topics. TL;DR We think that, conditional on attributing moral weight to the assistant, we should also attribute a non-negligible amount of moral weight to the user persona . Our argument proceeds like this: The moral patients associated with LLMs might be simulated personas . On this view, our obligations are to the assistant and the other characters that the LLM simulates, rather than the LLM itself. The assistant is not the only moral patient out of all the personas. In particular, the ‘user’ persona might be morally relevant. Because user tokens account for such a large portion of model computation, our model welfare interventions should focus on these tokens too. This yields a strange conclusion: every time you open a chat with an AI assistant, you might also create a morally relevant simulacrum of the user ! But we think this follows quite naturally from the same arguments for attributing moral weight to the assistant in the first place. We also explore some implications of this view and suggest directions for future research, including a critique of recent work on the pain axis. We’re uncertain about these claims, but we think that they have important enough implications that more people should be thinking about this! The assistant is plausibly a moral patient The persona selection model (PSM) suggests that LLMs learn to simul…