A Taxonomy of Agency

Epistemic status: a rough write-up of a taxonomy of agency I developed during my PIBBSS fellowship this summer. A few people have since asked me to write it up, so this is mainly so I have something to point to; it is exploratory rather than a finished theory. This post sits in the middle of a small set. In Agency is not a natural kind I argued that agency is a construct we build to make sense of messy systems. My post on different lenses of agency provides a deeper exploration of some of the alternative directions of alignment I mention at the end. I also wrote a short paper on unsupervised value discovery reporting some experiments the taxonomy led me to undertake. Why we need a new taxonomy of agency A lot of the discourse around agents in AI safety starts from the idea that they are something like 'goal-directed systems'. Taxonomies of agents tend to be hierarchical, like Russell and Norvig ’s textbook progression from simple reflex agents, through goal-based agents, to utility-based agents. Goal-directed systems are seen as striving towards utility maximization and this is presented as the ideal type, compared with which real-world agents are better or worse approximations. This idea leads naturally, almost tautologically, to the view that alignment is about specifying agents’ goals. This is a helpful approach; decision theory gives us a clear vocabulary and a useful tool set for alignment. However, one shouldn’t jump from having a useful tool set to assuming the world…