I’m building an AI agent designed to find the best results for with a search tool. The agent is essentially a Large Language Model (LLM) connected to a Search API via function calling. The goal is for this agent to use the Search API to identify the most relevant results. Let's imagine that I have an evaluation function f(x) that allow to score a specific result. How can I evaluate the performance of my agent ? For instance, how can I tell between two versions of my agent (v1 and v2), which one is performing better ? Here is a specific constraint due to the fact that in a real world search, I don’t have access to the entire database of results due to the sheer volume (millions of results) and the cost involved. Since I can’t purchase all these results, it’s difficult to know if a specific record should be included in the search results. So I can't use recall like metrics. Given this constraint, how can I effectively evaluate the performance of my AI agent?

Full article content could not be extracted automatically. Read the original below.