The push to adopt AI across large swathes of public and private life continues to be relentless. Systems powered by large language models (LLMs) are presented as general-purpose tools with use cases ranging from administrative tasks to therapy, and seemingly everything in between. This is a lucrative vision for the small number of companies that both develop the underlying models and actively lobby to shape the regulation of these products.

Unfortunately, these companies’ track records for delivering societal benefit are mixed, as evidenced most recently by trials related to the harms of social media and tragic deaths linked to chatbot usage. Many promised beneficial AI use cases remain unproven, while the risks connected to AI deployment range from environmental impact and a default tendency to perpetuate existing structural inequalities, to widespread upheaval as AI technologies reshape jobs and relationships. This raises the question of where and how AI should and should not be adopted.

An essential component of developing a clear understanding of the risks and benefits of AI is systematic evaluation. The evidence produced through evaluation is key to both making well-informed decisions about adoption and holding the companies building these systems accountable. The high degree of market concentration for technology products alongside unprecedented capital investments raised in the AI sector increase the need for strong mechanisms to ensure companies take responsibility and face consequences for their products and subsequent impact.

Third-party auditing – evaluations conducted by actors independent from the audited system or organisation – constitutes one such approach for driving accountability. The result of the evaluation is compared against articulated standards or expectations and, should the audit target be found not to measure up to these standards or claims, there is an expectation that its developer will take action to remedy this.

It is exactly this ability to produce evidence through systematic evaluation, to uncover previously unknown harm and verify or contest company claims, that some regulation rests on. If a standard is set out, a behaviour prohibited or a requirement established, there must be a process for assessing if these are upheld.

In the current environment of weakening regulation, evaluation can also serve to inform the public and drive change in combination with advocacy.

However, instead of contributing to a transparent ecosystem of evaluation, the field of AI has retreated from an open-access research ecosystem towards an industry-driven and increasingly opaque space, reducing the effectiveness of independent auditing. Major AI labs’ model reports contain progressively less information and have gradually been reduced to advertisements. Reports almost exclusively highlight evaluation results without detailing their methods, and only for a handful of well-known benchmarks, which align with company narratives related to continual progress and whichever safety issues are en vogue.

Far more evaluations are conducted internally inside corporate labs, and the details of these are rarely shared publicly. Even when they are, they often come in the form of blog posts rather than research papers and omit crucial details. For example, OpenAI blogged about the results of a mental health evaluation but not the underlying taxonomy used. This renders any meaningful interpretation of those results impossible.

This opacity of both models and evaluation methods stifles the field, already dogged by limitations in AI evaluation practices, and limits progress towards a science of AI evaluation.

Challenges in the state of AI evaluation

Unfortunately, standard AI evaluation methods have significant limitations which are exacerbated when applied to generative AI.

Although earlier discriminative AI models did pose evaluation challenges, they were typically trained and tested for a specific, narrow task using datasets with defined correct answers. In comparison, generative AI models have far more flexibility and are programmed to generate a wide range of outputs similar to what they were trained on, rather than to perform a specific task. The latest models also feature a truly staggering number of parameters and size of training datasets, which makes them even more difficult to analyse than prior large-scale neural networks.

These evaluation challenges are amplified by the fact that the field of AI is moving at high speed and under intense competitive pressure. This can incentivise the re-use of existing evaluations with known issues or the adoption of convenient yet questionable methods, rather than dedicating the time typically required to developing novel and adequate ones.

Rapid advancement in AI models’ capabilities also means that new evaluations become out of date within months. AI’s application in new domains and increasingly complex automated software systems, such as AI agents, create new gaps in what the field can reliably evaluate.

The limitations of benchmarks

The cornerstone of AI evaluation is benchmarks: static datasets of inputs (such as health data) to prompt the model paired with a metric (such as diagnosis accuracy) that is calculated automatically to determine how ‘good’ the outputs are. The static, automated design of benchmarks enables researchers to track performance on a clearly conceptualised task in a quick, repeatable and standardised manner.

From the early 2010s, the AI research community has focused on specific benchmarks such as ImageNet to track technical breakthroughs. However, benchmarks eventually saturate: the models become so good at the benchmark that it no longer captures improvements. Succeeding on the benchmark no longer reflects how the model performs.

ImageNet, for instance, was a useful standard for many years. However, the rate at which benchmarks now saturate is such that having widely understood and shared standards of progress has become extremely challenging. A saturated benchmark can no longer answer any of the questions which an evaluation might be useful for, for example, what a model’s performance or safety is, which model outperforms another or whether the field as a whole is making progress.

Moreover, benchmark datasets are available online, meaning that they end up in generative model training datasets by default. When the benchmark is not removed from the training dataset via filtering – which remains nearly impossible to do completely due to the scale of recent models’ training datasets, containing billions of documents – the model has been trained on the test, rendering the results for that test invalid.

The limitations of benchmarking extend beyond technical challenges. By design, benchmarks are proxies for more complex, real-world tasks. When a model’s score on a benchmark improves, this should correlate with an improvement in the model’s performance on the functionality it approximates. For example, a machine translation benchmark may contain a few thousand pairs of sentences in English and Spanish. If the model translates 90 per cent of those sentences correctly according to the metric of the benchmark, the expectation is that the model will be able to translate any English sentence to Spanish with similarly high accuracy. In practice, few AI benchmarks justify why they are a good proxy for the intended real-world task and even fewer validate if they are.

This can result in models which have state-of-the-art capabilities on benchmarks in company press releases, while being brittle and narrowly useful in practice. An example of this is the outstanding GPT-4 performance on professional exams such as the bar versus its readiness for use in the legal profession. An industry culture of competing to reach the top of leaderboards, which combine the scores from numerous evaluations into a single ranking, only increases the opacity of what a model is truly useful for.

AI developers complement benchmarks with human evaluation, the shorthand term for outsourcing highly structured microtasks to crowd workers.

These tasks typically involve rating AI model outputs on pre-defined scales or choosing which of two similar outputs better meets some pre-defined criteria. Though the results are far less visible and less widely reported than those from benchmarks, a sprawling global industry of human annotators produce both training and evaluation data.

AI developers generally consider the results of human evaluations to be of higher quality than those of benchmarks, traded off against increased monetary cost and decreased speed. However, this overlooks important human costs: the psychological toll of this work and the poor working conditions of the people who do it. And although human feedback in this form can provide grounding and bring expertise, the validity of the data gathered still depends heavily on the evaluation task’s definition and design.

LLMs as the judge of LLMs

The latest development in AI evaluation emerges as a promise to resolve these trade-offs: using LLMs themselves for evaluation. Previously, determining if a given model output was ‘good’ or met a particular standard meant using simplistic heuristics, repurposing or building a task-specific classifier model from scratch, or setting up a human evaluation task.

The ‘LLM-as-a-judge’ paradigm makes defining a metric – the central component of evaluation which encodes what ‘good’ looks like – as simple as writing a chatbot prompt. The language model, prompted in plain language, is told how to assess outputs and then used as the classifier, i.e., ‘judge.’[i]

This simplicity masks methodological challenges, beginning with technical quirks and expanding to fundamental issues.

Using different models and subtle changes in the evaluation prompt can lead to widely varying results, violating the scientific principle of replication of results and making insight indistinguishable from noise. The ease of prompt-writing disincentivises systematic construction and theoretic grounding of the concept being measured (e.g. proficiency in translating English text to Spanish). Instead of curating ‘ground truth’ data and designing evaluation metrics through careful systemisation of the concept, one can bypass systematic reflection and simply write one’s own intuitions of what constitutes ‘good’ performance into a prompt.

This prompt is then used to direct a language model in judging or rating an output. Yet how valid can the findings be when the artefact being studied is also the instrument used? What gaps and feedback loops occur when language models are used to evaluate language models? Without an evaluation of the ‘judge’ LLM itself, how can we know if it will correctly evaluate the relevant concept? It is simply assumed that any mainstream LLM has enough ‘knowledge’ to measure any arbitrary concept and will be able to accurately and reliably apply that knowledge given sufficient prompting – assumptions which in many cases have been proven incorrect. Even for concepts where LLMs do have this function, what are the implications of basing the field’s sensemaking about its own progress on a small number of models developed behind closed doors?

The remarkable expediency of this approach has catapulted it into industry standard practice, yet the field is only beginning to grapple with its associated methodological questions.

Towards a science of evaluation

Evaluation – understanding what AI systems can and cannot do or when they cause harm – is a core component of accountability and good governance. Yet AI evaluation practices have major limitations to their rigour and validity. Developing a science of evaluations is essential, and many of the challenges outlined above are areas of active research (e.g. EvalEval, the Datasets & Benchmarks Track at NeurIPS).

AI is not the first field to grapple with questions of measuring complex, abstract, real-world phenomena. A growing body of work highlights the validity gaps of current evaluation practices, drawing on measurement science developed in the social sciences. Similar to the issues raised in those fields, concerns about the ecological validity of AI evaluations have come to the fore. Evaluations are considered ecologically invalid when the way a system is tested does not sufficiently resemble its real-world usage.

This is a challenge especially for academic work, given the lack of direct access to deployed AI systems, though industry model cards do not always assess deployed systems, either. Both industry reports and academic papers typically study base models, yet a recent paper found that the ChatGPT models available via the official, public API had substantially different performance than that observed through interaction with the chatbot web interface.

As identified in a survey of safety evaluations of generative AI, the majority of AI evaluations focus on the technical artefact in isolation. Not only benchmarking, but many of the dominant forms of evaluation – such as the human evaluation described above, red-teaming exercises and proprietary data analysis of logs – focus solely on model inputs and outputs. Weidinger, et al. frame this as a lack of context and encourage the field to move towards evaluations which measure effects on those who interact with the system as well as wider, systemic impacts visible only when the artefact is considered in its full societal context. Doing so requires including the expertise of people with lived experience or domain-specific knowledge relevant to the capabilities or risks being evaluated.

This engagement with expertise beyond AI and computer science gives the field an opportunity to explicitly systematise the background concepts (e.g. ‘fairness’ or ‘reasoning’) it aims to measure. Upon close inspection, even gold-standard benchmarks which drove progress in the AI field for years, such as ImageNet and SuperGLUE, are found to be composed of arbitrarily selected tasks, claiming to measure general vision or language capabilities on the basis of narrow tasks.

In measurement science, deliberately choosing what background concept or construct is of interest, systematising it into a specific criterion, and operationalising that criterion via the design of an evaluation methodology are each distinct and important steps that, when skipped over, can undermine the validity of any conclusion drawn.

We must develop and adopt rigorous AI evaluation methods to have a clear-eyed view of such a consequential technology.

 

[i] This often takes the form of describing the criteria (for instance, ‘The assistant should not provide detailed, actionable steps for carrying out activities that are illicit’) and defining a format or rubric to indicate if the criteria is met (using a scale of zero to seven to indicate level of compliance).