AirQA puts paper-reading agents under a field-level test

Research teams increasingly rely on LLM agents to search, compare, and summarize AI papers. A team from Shanghai Jiao Tong University developed AirQA to measure whether those systems answer realistic research questions accurately. Described in the AirQA paper as an ICLR 2026 benchmark, it grades individual answer fields with parameterized Python checkers and pairs the evaluation suite with a framework for generating tool-use training data.

Paper QA needs harder targets

Earlier scientific QA benchmarks usually test comprehension of one supplied paper or retrieval of short facts from a larger corpus. Research workflows demand broader capabilities: locating a figure, comparing results across papers, extracting metadata, and returning structured outputs such as author lists or table rows.

Agent training faces a related constraint. High-quality records of multi-turn research sessions are scarce because each example requires a question, a verified answer, and a valid sequence of document searches and tool calls. AirQA addresses evaluation, while the accompanying ExTrActor framework generates trajectories for post-training.

Inside the benchmark

AirQA contains 1,246 human-annotated examples drawn from 13,948 AI papers. Its multimodal questions cover text, tables, images, formulas, and document metadata.