Most AI benchmarks test whether a model can get the right answer once. But if you're deploying an agent to do analyst work, "sometimes correct" is not good enough. That's the core insight behind AA-AnalystAgent, a new agentic benchmark from Artificial Analysis that tests models on real-world quantitative analysis tasks , and requires them to get each one right on all five independent attempts.

What it actually tests

The benchmark covers 80 questions across 14 business and scientific domains. These aren't toy problems. The source material includes actual government expenditure reports, commodity trade statistics, hydrology datasets, energy cost models, and financial valuation spreadsheets. Tasks span five workflow types that mirror real analyst work:

• Source lookup and diagnosis
• Filter and total
• Ratios, trends, and sensitivities
• P&L modeling
• Cash flow, balance sheet, and valuation modeling

The question set is kept private to reduce contamination risk. Two example tasks drawn from California Medicaid expenditure reports are publicly shown on the methodology page, complete with full prompts and source material.

The metric that changes everything

When businesses and researchers hand analyst work to an AI agent, they need to be confident it will produce the correct answer every time. That's why AA-AnalystAgent runs every task five times and reports pass^5 as its headline metric. A task only counts as solved if the model answers correctly on all five independent attempts.