Before a software-engineering agent can fix a GitHub issue, it must locate the relevant code, inspect surrounding functions, identify APIs, reproduce the failure, and run regression tests. Research often evaluates these capabilities separately, leaving developers without clear evidence about which one most affects patch success.

Researchers from Microsoft and Georgia Tech address that question in the ORACLE-SWE paper. They give an agent a perfect, benchmark-derived version of one intermediate signal at a time, then measure the resulting improvement. Reproduction tests produce a larger gain than the other four signals combined.

Five artifacts behind a patch

Software-engineering agents such as Agentless, SWE-agent, OpenHands, and Claude Code repeatedly generate five kinds of intermediate information. The paper’s taxonomy groups them by source and purpose:

• Reproduction test: A Fail-to-Pass test that checks whether the patch satisfies the requested behavior.
• Regression test: A Pass-to-Pass test that verifies existing behavior remains intact.
• Edit location: The files, functions, and line ranges that require changes.
• Execution context: The call stack and surrounding code near the failure.
• API usage: The internal or external functions needed for the fix, along with their definitions.

Each artifact reflects a distinct capability: interpreting requirements, preserving behavior, locating code, tracing program execution, or retrieving implementation details. Measuring them independently reveals where better information produces the largest improvement.

The researchers build rule-based extractors that derive an oracle, or ground-truth version, of each signal directly from SWE-bench data. Edit locations come from the reference patch. Fail-to-Pass and Pass-to-Pass cases provide reproduction and regression tests, with test bodies recovered through syntax-tree parsing. For Python tasks, runtime coverage identifies relevant APIs; for Go, static type analysis resolves function calls and definitions.