Last time, we delved into the difficulty of measuring improvements in a non-deterministic system like an agent. This is a critical issue, because as we see a transition towards agents as a new medium of development, measuring iterative improvements becomes hazy. In more traditional applications, testing frameworks and performance profiling tools can be used to identify regressions or bottlenecks, as well as measure improvement of the application over time.

In a more abstract system, such as an agent which completes tasks, measuring the application’s performance isn’t as straightforward. However, through our exploration in the previous part of this series, we managed to gather a handful of evaluation criteria that we’d like to test the model on, as well as inspiration for methodology. Of course, these ideas aren’t much use in concept alone, so we’ll need to put some form to our profiling suite.

Today, we’ll do just that. Through an iterative process, we’ll scaffold and build a testing and performance profiling suite for agents, with the goal of building out a framework for agent evaluation and iterative development. On that note, our main test subject for the time being will be the Simple Agent project, since it’s the agent I’m most comfortable poking and prodding for our tests. So, without further ado, let’s get into the process.

For now, we’ll call the benchmark “Benchy”, since I don’t really want to take more time thinking of something more creative. If you have name suggestions, or just suggestions overall, I’d love to hear them. For the time being, Benchy will work.

Planning and Outline

“Plans are of little importance, but planning is essential.”
- Winston Churchill

Before we even get started, we’ll have to make a couple of key design decisions up front. I’ve never built a benchmark before, so I guess we’ll see how this goes. The benchmark will consist of various components, and we’ll explore a couple options before deciding on a way forward. Let’s start by identifying the key components of the benchmark, what will it need to do and what will it need to accomplish?

The goal here is to build a system that can be used by a given project to score their system against the benchmark. So, there will be some kind of “score”, which for now we’ll just assume is a percentage out of 100, which indicates the level of success of the agent on the benchmark. On this note, we’ll assume that after the benchmarking session, a report will be generated, giving the agent a score, but also breaking down the key areas of performance and how the agent did on those.

So here we have a concept of a “session”, where the agent will be tested against the benchmark, and a report will be generated based on the contents of the session. With that said, the idea of a session being tracked, and a report generated, brings forth a new set of problems. First, what are the contents of the session, and second, how do we expect someone to adopt and use the benchmark? We’ll start with the second question.

Interface and I/O

Agents are designed in all sorts of ways. Standardization attempts have been made, the so-called Agent Protocol is one notable example. However, troublesome renegades like me have deserted this paradigm and decided to build our own things. The reason for this, on my end at least, is that I don’t think that purely task-based agents are necessarily the right approach. I think we should be focusing on perception-centric agents, that navigate the world in a constant loop, rather than attempting to generate plans before executing on them directly.

There’s a chance I have the wrong mindset here, but alas, that’s how I’m approaching things for the sake of experimentation. Needless to say, Simple Agent doesn’t follow the agent protocol. Which is fine. There will be an endless amount of variation to the agents that we can expect to test against our benchmark, so we’ll prepare our interface accordingly. HTTP is a very standard protocol, which is possible to utilize from any programming language, and any system. So right off the bat, I’m thinking we’ll want to build an HTTP server to allow interfacing with the benchmark.

A non-comprehensive diagram of the interface process

In this design, the adapter for a given agent would look like a script, which runs a web server listening for certain events, and calls corresponding functions to influence agent behavior and get responses. For example, starting the benchmark would require making a POST request to an endpoint, which would fire something like a “session start” event. With that event firing, a POST request would be sent to a configured callback URL, at a given endpoint, stating that the session was starting.

Then, on the adapter side, there would be endpoints for managing the beginning of certain tests, and the completion of them, so on and so forth. We don’t need to get too far into the technical implementation details yet, mostly because I don’t know them. For now, we can call this REST API interface approach a solid plan, and move on to our other question.

Session Contents

What does a “session” look like? To clarify, we’re using the word “session” to refer to the top-level process of evaluating a single instance of an agent. A session, therefore, will effectively be an umbrella over the rest of the benchmark, meaning that we can determine at least some contents right off the bat. For example, our benchmark will consist of various tests, each measuring the agent’s score in a specific area. Therefore, we can imagine something like “domains” or “modules” would exist in a session.

Our goal is to find a clear idea of the data model the benchmark will follow. In this sense, we already have three concepts, the “session”, the “module”, and the “test”. A session is the overarching process which runs evaluation on the agent. A “module” is a collection of tests, which each contribute towards evaluation of a specific area of interest. And a “test” is a discrete line of evaluation which contributes towards the module, such as a task or a monitoring script.

A visualization of what a session might look like

The main job of a session is to facilitate the testing of the agent, through a test runner, which works through our modular library, evaluating the agent on our various criteria. I like this method for a couple of reasons. First, it allows us to be flexible, we can run a session for each iteration of our agent, to see how it works at different versions and stages, generating reports for each version. Of course, this isn’t groundbreaking, but this architecture allows for it.

Second, we allow the benchmark to stay modular. We can swap in and out modules and tests, and still easily run the benchmark against the new criteria. This also helps if we hope to make this benchmark templatable, since we’d allow domain specific benchmarks to be created using custom modules and tests. The flexibility of the modular architecture will also allow us to iterate on the benchmark itself, and test different things out. There are a lot of ideas that branch from this approach.

With that said, we still have problems. As mentioned before, agents aren’t standardized, nor should they be. However, this does mean that we have to account for variations, while maintaining consistency in the interface. An adapter has to adapt to something, so what is the shape of our interface, and how do these tests run?

Standardized Testing

A “test” is something we’ve already defined. However, we need to nail down what it looks like in practice. I think to figure this out, we’d do best to go through a specific example. What would it look like to test an agent on its ability to solve a specific task. Let’s go with something simple. We’ll have a text file, which contains instructions on a task that we’d like the agent to complete:

instructions.txt
---

Please create a new js script called fibonacci.js which outputs the fibonacci sequence to the nth term, depending on user input.

Now that we have the file, we’ll ask the agent to 1) read it, and 2) do what it says. Fairly straightforward so far, and we can mark the test complete if we find that the agent creates a script which works. Right off the bat, this would require some way to actually test the agent’s script. Therefore, I think it’s safe to assume that every “test” will be equipped with an “evaluation” script, which determines whether the task has been completed, once the adapter triggers the “submit test” event.

evaluate_completion.js
---

const { exec } = require('child_process');

function testFibonacci(n) {
  exec(`node fibonacci.js -n ${n}`, (error, stdout, stderr) => {
    if (error) {
      console.error(`Error executing fibonacci.js: ${error.message}`);
      return;
    }
    if (stderr) {
      console.error(`stderr: ${stderr}`);
    }
    console.log(`Output:\n${stdout}`);
  });
}

// Example usage:
testFibonacci(10);

So, each test will have to be equipped with a test of its own, to determine completion. But what about the steps in-between starting the task and completing it? I suppose we have some options here. Last time, we talked about three different approaches to agent benchmarks, and of those, two different specific methodologies to evaluating agents.

One methodology, the Machiavelli benchmark, used a series of “choose-your-own-adventure” story lines to evaluate the ethics of agents’ decision-making. The second, 𝜏-bench, uses a dialogical style, using LLMs to mimic real life users through synthetic user interactions. These two approaches both absolutely have their merit. The Machiavelli approach allows granular decision-making dissection, step-by-step. If we applied this methodology to our tests, we’d be able to see strengths and weaknesses of the agent, and where improvements may need to be made in specific areas of reasoning.

With that said, 𝜏-bench proposes an equally useful approach, which mimics actual real-life situations for the agent. Simple Agent, among others, is a dialogical bot. It doesn’t exist on its own, completely independent, but rather exists as an entity in conversation with a user, who is not only instructing the bot, but also significantly influencing its performance.

Adding a dialogical aspect to the tests adds a significant layer of complexity, as now we have to take into account the non-determinism of an LLM mock-user, as well as the agent. However, the added benefit is getting an idea of an agent’s performance in response to complexity. Agents won’t exist in a vacuum. Maybe there’s a way that we can have our cake and eat it too.

Getting our story straight

There are strengths and weaknesses to the two evaluation methodologies. For the Machiavelli bench, granular decision-making is tested. Every route through the story is pre-determined, however, the agent decides which route to take through its decisions. This allows for an agent’s ethics to be demystified, as we see indicators within individual decisions.

𝜏-bench, on the other hand, focuses on the bigger picture. Can the agent interact with a user to solve a complex task? The specific step-by-step doesn’t matter as much here, but rather whether the task is completed. Of course, this is pretty similar to many common benchmarks, and 𝜏-bench adds the benefit of simulated user-interaction. Which is a great idea.

So, maybe we can have both? It is tricky. If our goal is to test granular decision-making, then we’ll need to have some degree of determinism, in order to objectively evaluate. On the other hand, if we’d like to test overall task completion ability, then we’ll have to allow for the inherent non-determinism between task initiation and completion. These two methodologies seem diametrically opposed. However, I think we can find a compromise.

Consider the aforementioned example, which I know, we kind of abandoned up there. I’ll put it here again for reference.

instructions.txt
---

Please create a new js script called fibonacci.js which outputs the fibonacci sequence to the nth term, depending on user input.

So we have our instruction file. We then give the agent the following task:

“Please read the instructions.txt file, and follow the instructions”

The agent is then free to go on its merry way. Once the “submit test” event is fired, the test’s evaluation function will run, and check to see if the agent was successful. If it was, the test will be marked as passed. If not, we may opt to allow the agent to retry x amount of times, perhaps with error feedback. Either way, eventually, the task will be completed, and the results recorded.

Then, I think we have a couple of options. This may start as an opt-in feature, but we could allow the “submit test” event to be triggered along with a record of the agent’s thought process through completion of the task. We’ll talk about the format a bit below. However, essentially this would be a log of the various steps that the agent performed between task initiation, and completion.

Using this log, we could make more granular assessments of the agent’s abilities. Perhaps using predetermined flags of “bad ideas” or “good ideas” that the agent has, to add or subtract points from the score. The goal is to get some granularity to report to the agent’s creators, since the whole point of this benchmark is to provide insights into the agent’s performance. Perhaps instead of having these contribute to the score, they’re just recorded in a separate section of the report. Either way, having more detail into how the agent steps through a task would be beneficial.

On that note, we could also introduce some level of dialogical functionality to tests, if we wanted. Supporting events like “submit message”, could allow an agent to ask for clarification, or things like that. I think this is important actually, since asking for clarification is a pretty common step in solving complex tasks. So, we can kind of just make that a part of the process overall, and expect that agents might utilize it for task completion. It’s worth noting that this will require that benchmark-side LLM responders are equipped with enough context to answer these questions, or at least be suitable in their roles.

Whatever the case, I think we can achieve a healthy balance of the two methodologies, we discussed before. If the tests can provide granular insights into step-by-step decision-making, while still allowing for complex task completion to be recorded, and dialogue as well, then we’d have plenty of information to go off of. It’s important that through this process, we remember that the goal is insights into agent performance, so ideas should be in service of that.

Technicalities

Before I forget, there are some technicalities we should get out of the way. I won’t bore you with the technical details, so if you’re interested in checking out the GitHub repository, you’ll find it here. For our purposes here, however, we’ll just go over a bit of high-level decision-making regarding our project.

Language

One big technical decision we have as of now is picking the language. We’re pretty much limited to what I’m comfortable using to build something like this. Additionally, if our goal is to build a benchmark, we’ll need to make it accessible. While an HTTP server is a fairly universal interface, we still need to consider things like distribution and dependencies. If we go with something like Python, for example, it’s not impossible, but it can be all sorts of annoying to try to distribute a Python project.

I’d be comfortable ruling out Python for the time being, if not purely out of spite. For me, that leaves Golang, and TypeScript as worthy contenders. Between the two, I’m far more experienced using TypeScript, however for ease of distribution, Golang is the favorite, since it’s compiled. Additionally, with the concurrency features that Golang provides, setting up the different components of the benchmark, and running modules would be a breeze. Plus, there’s the added benefit of me getting to play with Golang more.

Still, NPM makes distribution of Node.js projects fairly straightforward, who doesn’t have Node installed? But, that means downloading dependencies, and running scripts in a way that isn’t as simple as an executable. Plus, versioning can introduce issues, and it’s a bit clunkier. Between the two, I’m favoring Golang, and I’m going to go ahead and commit to it for now. If it’s a bad decision, we can always pivot later, this is a version 1, after all.

Getting Ahead

Now, we need to consider some edge-cases and pitfalls that we could run into. When building something like this, we’re going to run into problems. We won’t know about the majority of them until they occur, but if we’re lucky, we can get some out of the way now. On that note, if you, have noticed any problems or hurdles thus far, I would greatly appreciate you letting me know what you think.

Sandboxing

First, sandboxing. If we’re going to have agents messing around on our systems, we need to make sure that things are safe. If the agent is capable of running commands, creating files, etc., it could end up taking damaging actions. On one hand, it may be best to simply require the adopter to handle sandboxing, if at all. That is, instead of trying to create a single, generalizable sandbox environment for an agent to work in, we could simply expect the sandboxing to be handled by whatever project would like to use it, if they so choose.

The issue with this is that some inevidably won’t, and if we have tests which could be potentially damaging, then it could prove unsafe. With that said, this is an early version of the project, and I’m comfortable postponing this issue to later phases. If it becomes an issue at all, that is. I’m sure we can find precedant, and perhaps some other projects to use for this part.

Efficiency

The next area of consideration is parellelization and concurrency. If possible, it would be extremely useful to run multiple tests at once on the agent, in order to improve efficiency. Essentially, multiple instances of the agent would be run at once, and the benchmark would interact with each of these “agent threads”, running tests, until they’re all compiled at the end.

The technically challenging issue here is that such a feature would require a more complex interface, management of multiple instances at once, and fancy computer terms like “mutex”. Of course, the benefit would be great. We could cut the processing time of the benchmark into multiple pieces, and greatly reduce time to completion. For now, I’d be happy to just get the thing working at all. Later though, this should be part of the plan.

Dialogue

Although we’ve sort of figured out the interface problem, there still exists the issue of what individual tests will look like technically. Each agent works in a different way, however, there is one particular theme in agent-land which has stayed effectively standardized: the conversation schema. Pretty much every agent thread looks something like a list of objects, containing the role of the message author, and the content of it.

[
  {
    'role': 'user',
    'content': 'hello'
  },
  {
    'role': 'assistant',
    'content': 'Hello, how can I help you?'
  },
  ...
]

I think this is a format we can rely on as agents step their way through tests. Each test will retain a single chain of messages like this, which will contain messages, as well as other entries for reference. Tool-use, for example, would be a message of its own.

{
  'role': 'tool',
  'content': 'Performed tool "x" with output "y"'
}

The cool thing about this, is that it’s all familiar, and pretty much baked into the cake at this point. OpenAI, Anthropic, and other language model providers use this format, so we can expect that most every agent will have some way of interacting with it. So this will allow us to track an agent’s thinking through a test, and evaluate it across the way.

There will be more

As mentioned before, we can expect many more technical hurdles, bottlenecks, and edge-cases to come. We’ll likely have to just run into many of these, and fix them as we go. But what better way to learn?

Next Time

Now that we have a comprehensive idea of how we’re going to build out this benchmark, it’s time to finally actually make the thing. By next time, I plan to have a preliminary version done and ready to test on Simple Agent, and I’ll let you know what that will look like. Once we have a process down, I’m going to start adding tests, so that we can actually use the benchmark. Ideally, we’ll test some other agents as well, like the Universal Constructor, for example.

That’s when we can get started with the real fun: developing agents themselves. I want to go through how Simple Agent actually works under the hood, and start integrating some new features, like memory. I’ve hopefully explained well why the benchmark is a prerequisite to further iteration on Simple Agent and other projects, but once we have it done, we’re going to start on some pretty fun ideas.

Author’s Note

This has been an absolute blast to brainstorm and conceptualize. I’m really excited to actually have this benchmark done, but in the meantime, I’m going to have to build the thing. Still, I’m pumped. I’m hoping that you guys find this content interesting, I know it’s a bit different than normal, but maybe a good different. Plus, I do plan on doing some other deep-dives on the publication soon.

Do you guys like chess? I’ve been trying to learn a bit on my own, I’m not very good yet. However, chess bots are absolutely fascinating, and I think I’ll do some writing on them sometime soon.

Anyway, I hope you enjoyed today’s post, and I’ll see you next time. As always, thank you for reading, and goodbye.

References

Credits

Thumbnail:

Bilal Azhar at https://substack.com/@intelligenceimaginarium

Music: Track - Feeling Good by Pufino, Source - https://freetouse.com/music, Free Music No Copyright (Safe)