AI agents are getting better at tackling complex, multi-step tasks. But there's a hidden cost: every tool call, file read, and intermediate decision adds to their growing context, making each subsequent step more expensive.
New research from Nvidia challenges the assumption that agents need all that history. And the findings have important implications for how we build, optimize, and serve AI applications.
Today we break down what developers should know.
Why AI agents don't always need long contexts
In agentic tasks, the context of the underlying LLM might grow into hundreds of thousands of tokens. The agent spends many steps searching repositories, modifying files, and running tests.
Each of these interactions becomes part of the next request to the model. The agent’s context continues to grow, even when much of that history is no longer useful.
In a new study, researchers at Nvidia tested whether AI models really need those long contexts. Across different long-context tasks, they found that LLMs can often work with much shorter contexts while maintaining their accuracy.
For developers, the findings suggest that instead of stuffing everything into the model’s context, only keep the minimal information the agent needs for the next step.
Why long contexts are expensive
When an LLM generates a token, it must attend to all the tokens that precede it. Even when storing attention values in the KV cache, the attention tax increases as the context grows because attending to previous tokens takes more computation and memory.
The researchers measured the impact on three open-weight models running on NVIDIA H100 GPUs. Reducing the configured context window from 256,000 to 16,000 tokens increased output-token throughput by 3.6 times for GLM 5.2, 5.6 times for Kimi K2.6 and seven times for Qwen 3 Coder Next.