As Large Language Models move away from being chat interfaces and become increasingly autonomous actors in the real world, a few insights about evaluation and training of these systems emerge, and I'd like to discuss them.

Context:

I've gained the insights and ideas laid out below through ongoing work I'm doing. This post serves to outline my working model in pursuing it, and constraints and lessons I learned along the way. Some of these are offered as learned lessons, others as assumptions, and some as work I intend to pursue. I close with a few questions.

From Text to Actions

There's a case to be made that in order to have these models improve in their interaction with the real world, instead of mainly improving on text (and code) generation, a different training data is required.

In an experiment I did on fine-tuning Gemma on real interactions, I've seen real (but bounded) improvement in the held-out tasks that Gemma has attempted, despite a very small corpus.

Ironically, it was a slight regression in one of the tasks that provided a strong indicator of the validity of this approach. Across 4 task types, the model has improved in 3 of them, and regressed in 1. After re-checking the training data, I saw that the training data for the single regressed action type has accounted for about 4% of all of the corpus.

Interestingly, the model then seemed to copy (perhaps overfit) to the style of the solution of that task, namely, over-wandering instead of more direct actions.

Environment Validity

In order for an environment to be valid, it is my view that three conditions must be met:

1. Working, live, environment. Straightforward, but not trivial - if one requires of an environment to simulate a simple board game, then that game needs to boot and work according to its rules. If one requires a more complicated environment, such as one simulating actions taken in a cloud environment to help engineers fix services and triage errors, then that environment needs to reflect that requirement.
2. Solvable via intended path. For an environment to teach an agent anything, it needs to be solvable. More importantly, it needs to be solvable at scale. This means that manual verification won't work, and alternatives need to be considered. If you're teaching a model to fix a cloud service across ten million runs, and the cloud isn't fixable in most of them, you've merely built a broken environment generator. Or worse: if the failure is uncaught, one can degrade a model's ability to perform these tasks at all.
3. Solvable only via intended path. This is difficult. If one wishes to have a model learn how to fix a cloud issue, and the model finds a shortcut through the lab via some unintended path, which can be broadly defined as a path from which the model doesn't improve by learning from it, then the environment is broken. In my opinion, this cannot be solved completely, but one can build detection mechanisms for when models do find them, and remove them from a training run.

Realism and Diversity

This is either the hardest and most important problem of this effort - or a problem that can be addressed through an assumption I'll write below.

For ten million labs intended to teach a model to perform an action, the labs need to be as close to realistic as possible - or else the agent only learns how to solve your labs. This means that the model can't generalize into real, messy environments, where there are a dozen services, each depending on each other, and each breaking in its own way. building a clean environment with only a single broken component teaches the wrong things.

They also need to be diverse. Otherwise, you're teaching the model the same lesson ten million times - from which no learning system can advance.

An (unproven) assumption can be made to address this, and I intend to test this in the upcoming month:

Considering models do know the abstraction of how to fix a cloud environment (go into console, output errors, navigate to affected component, debug, etc.), what one is teaching the model is less the knowledge of it, and more the performance of doing so. Having knowing that, one (I) may set out to show that performance can be improved through teaching interactions, based on the assumption that the models know how to approach the solution.

Conversation & Questions

Some closing thoughts I have:

1. Is anyone doing this generically? I'm aware of multiple companies and public benchmarks and datasets each focusing on a single, narrow task. Am I wrong to think that building a task-agnostic engine is the best approach to this problem?
2. Is the central thesis of action-based training the right one - or is there an assumption that models can improve as agents through more knowledge acquisition and capability improvement?
3. Does this approach, if valid, scale outside of language models, to robotics and other real-world systems as well?