LessWrong AI
2026-11-10 07:18 UTC
By aak
USR-0152-20261110-community-fo-b2cd0c5b
Measure Optimal AI Share of Labor Costs Instead of AI or Human-Only Performance
In a research post , Cunningham et al. of METR propose the “‘expenditure horizon’ over an optimization problem as the dollar value at which the improvement to the goal metric is equal to the improvement by a human with the same budget.” Though their work focuses on individual optimization problems (with NanoGPT as the main subject), the expenditure horizon can be meaningful for any task distribution: it’s the budget at which human-only and AI-only work reach the same performance. Though I like expenditure horizons as an operationalization of AI capability, I have a few critiques which suggest an improved metric, which I’ll call the “X% expenditure share horizon”: at what labor budget is it optimal to spend X% of the budget on agents rather than humans? In this essay, I’ll explain why this metric is both more useful and likely easier to measure than the expenditure horizon. I think this is an area that’s ripe for concrete follow-up work. Figure 1. Performance as a function of expenditure on humans or agents. Source: METR . Pros and Cons of the Expenditure Horizon Metric Let me begin by saying a bit about what I like about the expenditure horizon metric. First, as we brace for the economic impacts of AI, I think it’s good to begin proposing measures of AI capability that are tied to the human cost of comparable work, and the expenditure horizon takes a shot at this. Another metric of this kind that I am partial to is “autoflation” from the Remote Labor Index : “The percentage…
In a research post , Cunningham et al. of METR propose the “‘expenditure horizon’ over an optimization problem as the dollar value at which the improvement to the goal metric is equal to the improvement by a human with the same budget.” Though their work focuses on individual optimization problems (with NanoGPT as the main subject), the expenditure horizon can be meaningful for any task distribution: it’s the budget at which human-only and AI-only work reach the same performance. Though I like expenditure horizons as an operationalization of AI capability, I have a few critiques which suggest an improved metric, which I’ll call the “X% expenditure share horizon”: at what labor budget is it optimal to spend X% of the budget on agents rather than humans? In this essay, I’ll explain why this metric is both more useful and likely easier to measure than the expenditure horizon. I think this is an area that’s ripe for concrete follow-up work. Figure 1. Performance as a function of expenditure on humans or agents. Source: METR . Pros and Cons of the Expenditure Horizon Metric Let me begin by saying a bit about what I like about the expenditure horizon metric. First, as we brace for the economic impacts of AI, I think it’s good to begin proposing measures of AI capability that are tied to the human cost of comparable work, and the expenditure horizon takes a shot at this. Another metric of this kind that I am partial to is “autoflation” from the Remote Labor Index : “The percentage…
Full article content could not be extracted automatically. Read the original below.
Source:
LessWrong AI
· lesswrong.com