The review bottleneck
As coding agents help us produce more code, the bottleneck shifts toward review. Writing a change gets faster, but understanding its effects and deciding whether it should ship still take time.
At Sourcegraph, we had a stamp group for small PRs. Someone would post “Can someone stamp this?” and a teammate who was available could pick it up. It helped keep small changes moving, but every request still meant finding a person and asking them to switch context.
Our GitHub setup required an approval before merging. A routine change and a change that needed a detailed discussion both had to clear that same step. We had different expectations for review, but we were fitting them into one approval requirement.
We wanted to make those expectations explicit and spend less time chasing routine approvals. That became our PR routing experiment.
Ship, Show, or Ask
We thought AI could help us decide whether a change needed another person's attention, and when. An agent could inspect the diff and surrounding code, assess the change against our rules, and provide evidence for choosing a review path.
A 2021 article on Ship/Show/Ask describes three ways to handle review. We use the same distinction:
- Ship: no additional review is needed under our policy.
- Show: someone should see the change, but their review can happen after merge.
- Ask: someone needs to review the change before it merges.
We started with recommendations only, while human review continued as usual. We used feedback to refine the rules and the evidence required to establish familiarity, then rechecked the affected recommendations. This gave us a way to inspect and challenge the bot's decisions before enabling approvals for eligible changes.
Research has explored similar choices. A simulation study comparing reviews before and after commit found that the tradeoffs depend on the development context. A 2026 preprint on Meta's RADAR describes risk checks that choose between human review before landing, deferred review, and automatic approval. These are useful comparisons as we work out what fits our workflow.
How the agent works
The agent examines the PR diff, relevant code, and the author's previous contributions. It returns findings with supporting evidence, and application code applies our rules to select a review path and post the result on the PR.
To assess familiarity with each component, our current policy looks for two merged PRs reviewed by another person, plus a relevant contribution of code, tests, or technical documentation in the past 30 days. The workflow runs in GitHub Actions.
The rules consider both the change and the author's familiarity with the affected components. Changes to authentication checks or to how sensitive data is exposed require review regardless of experience.
When a PR needs human review, we use Sourcegraph's Compare page to inspect the diff and navigate the surrounding code. The bot helps decide when another person should look; the compare view helps us understand the change.
What we learned from feedback
We learned that review serves several purposes. It can catch a problem, share context, or check what we tell customers. Being explicit about those purposes helped us decide when another person's attention would be useful.
Familiarity also needs careful boundaries. We can use contribution history as evidence, but we need to check how closely that experience relates to the code being changed. Working in one part of a system doesn't imply familiarity with all of it.
Showing the reasoning gave us something we could question. We could challenge the assumptions behind a recommendation and work out whether the rule or the assessment needed to change. That feedback became part of how we developed the policy.
We also had to account for the bot's demands on people's attention. Comments and repeated assessments can become interruptions of their own. Deciding when the bot should speak, update, or stay quiet is part of designing the workflow.
What we're still figuring out
We're seeing fewer requests for a quick stamp, which is encouraging. We're still treating this as an internal experiment. What we want to understand next is how it changes the way we work together.
With Show, removing the wait means relying on someone to return after merge. We want to know whether those changes get useful attention and whether colleagues stay informed after the author and reviewer have moved on.
We also need to account for the bot's own interruptions and keep its view of familiarity current as people and the codebase change. Fewer stamp requests tell us little about review quality.
We'll return to these questions in a follow-up post, including what happens to reviews after merge and how the bot affects interruptions. Subscribe to the blog to follow what we learn.
A special thanks to Erik Seliger for his contributions to this blog post.