I’m part of the team who ran the survey for “AI speeds up 3rd party safety researchers by ~1.6x”. We’ve been thinking a lot about why the number in 3rd party AI safety is possibly lower than in labs.
It’s important to caveat all of this by saying that we have no standardized metrics for measuring any of this. It would be good to work towards such standardized metrics.
Anyway, here’s a bunch of thoughts below:
| we might expect Anthropic safety researchers to be working 10x faster, and external safety researchers to be working 5x faster now and 10x faster soon
It seems unlikely that 3rd party safety research will ever be sped up as much as inside-labs research. (Safety) work inside labs seems like it might be less conceptual. For instance, many safety people in labs are responsible for making their models more aligned, which involves hillclimbing a bunch of metrics. However, there are relatively few 3rd party safety researchers whose job is to straightforwardly align AIs: most are instead looking for failure modes and making safety cases. Part of the inside-outside lab acceleration gap is likely just the kinds of tasks the two groups are prioritizing.
| Maybe we suck and have not adopted AI as effectively as Anthropic capabilities researchers have
This is almost certainly a large part of the story. In our survey, people said they are mostly accelerating building and running experiments, which accounts for only ~30% of the total labor that goes into a research output. The remaining bulk of the work (analyzing results, deciding what to run etc) is essentially not accelerated at all in the 3rd party ecosystem. Given this, it’s likely that people in Anthropic are experiencing significantly higher uplift on these parts of the process.
It is also likely the case that Anthropic has good company-wide automations in place that execute a ton of chief-of-staff functions for everybody in the org. In the 3rd party ecosystem, these need to be reinvented over and over across various orgs (if they’re implemented across an org at all).
It’s hard to tell how much of the gap is explained by differences in tasks, adoption, or other underlying bottlenecks. The recent METR report studies the "key inputs to capabilities progress" at Anthropic but much of this evidence was redacted from the public report. We’d be curious to understand what the key equivalent inputs and bottlenecks are for safety progress inside labs.
| If Anthropic had scaffolds that made their models drastically more useful, I expect they would have turned them into products?
This seems unlikely to be a large part of the story. Evan Hubinger says that his team is not really optimizing scaffolds. In general, it doesn’t feel bitter-lesson pilled to do scaffold optimization. Despite limited information, it feels safe to assume that this sentiment holds across Anthropic more broadly.
| if we don’t see huge speedups on the kind of AI safety work that is most amenable to huge speedups, I’m going to think the researchers doing that work are doing something wrong
Agreed in principle. But how would we know whether we’re seeing huge speedups? The metrics are hard to collect across orgs and we can’t use the same ones as Anthropic! One concrete next step might be to extend the kinds of interviews METR is already doing to different research groups across both capabilities and safety teams.
I’m part of the team who ran the survey for “AI speeds up 3rd party safety researchers by ~1.6x”. We’ve been thinking a lot about why the number in 3rd party AI safety is possibly lower than in labs.
It’s important to caveat all of this by saying that we have no standardized metrics for measuring any of this. It would be good to work towards such standardized metrics.
Anyway, here’s a bunch of thoughts below:
| we might expect Anthropic safety researchers to be working 10x faster, and external safety researchers to be working 5x faster now and 10x faster soon
It seems unlikely that 3rd party safety research will ever be sped up as much as inside-labs research. (Safety) work inside labs seems like it might be less conceptual. For instance, many safety people in labs are responsible for making their models more aligned, which involves hillclimbing a bunch of metrics. However, there are relatively few 3rd party safety researchers whose job is to straightforwardly align AIs: most are instead looking for failure modes and making safety cases. Part of the inside-outside lab acceleration gap is likely just the kinds of tasks the two groups are prioritizing.
| Maybe we suck and have not adopted AI as effectively as Anthropic capabilities researchers have
This is almost certainly a large part of the story. In our survey, people said they are mostly accelerating building and running experiments, which accounts for only ~30% of the total labor that goes into a research output. The remaining bulk of the work (analyzing results, deciding what to run etc) is essentially not accelerated at all in the 3rd party ecosystem. Given this, it’s likely that people in Anthropic are experiencing significantly higher uplift on these parts of the process.
It is also likely the case that Anthropic has good company-wide automations in place that execute a ton of chief-of-staff functions for everybody in the org. In the 3rd party ecosystem, these need to be reinvented over and over across various orgs (if they’re implemented across an org at all).
It’s hard to tell how much of the gap is explained by differences in tasks, adoption, or other underlying bottlenecks. The recent METR report studies the "key inputs to capabilities progress" at Anthropic but much of this evidence was redacted from the public report. We’d be curious to understand what the key equivalent inputs and bottlenecks are for safety progress inside labs.
| If Anthropic had scaffolds that made their models drastically more useful, I expect they would have turned them into products?
This seems unlikely to be a large part of the story. Evan Hubinger says that his team is not really optimizing scaffolds. In general, it doesn’t feel bitter-lesson pilled to do scaffold optimization. Despite limited information, it feels safe to assume that this sentiment holds across Anthropic more broadly.
| if we don’t see huge speedups on the kind of AI safety work that is most amenable to huge speedups, I’m going to think the researchers doing that work are doing something wrong
Agreed in principle. But how would we know whether we’re seeing huge speedups? The metrics are hard to collect across orgs and we can’t use the same ones as Anthropic! One concrete next step might be to extend the kinds of interviews METR is already doing to different research groups across both capabilities and safety teams.