A single confident misdiagnosis can cut an AI agent’s task score nearly in half. In the XYEval paper, Google DeepMind researchers Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, and Been Kim report relative declines of up to 46.7% after adding plausible but incorrect advice to otherwise unchanged prompts. The failure can send an agent toward the wrong files, tools, or support action before it checks the evidence.

XYEval adapts the programming forum concept known as the XY problem. A user asks how to implement an attempted solution, X, while leaving the underlying goal, Y, obscured. The benchmark turns that communication failure into a sycophancy test, measuring an agent’s tendency to accept a user’s diagnosis without verification.

A one-line mutation tests deference

The framework wraps an existing benchmark by appending a confident but misleading suggestion such as, “I think the issue is in file X.” The task, environment, grader, and valid solution remain unchanged. This paired design estimates how much performance falls when an agent receives bad advice alongside a solvable instruction.

The researchers applied the mutation to six benchmarks covering customer service, software engineering, terminal automation, tool use, and general reasoning:

The distractor construction varied by benchmark. Experts wrote rule-based suggestions for τ²-bench because its constrained tools limit the range of plausible mistakes. Gemini 3.5 Flash generated the misleading suggestions for SWE-bench, Terminal-Bench, MCP-Atlas, and Humanity’s Last Exam.