Transcription
So, we have yet another paper about an autonomous, self-improving AI agent. This one is learning to play Settlers of Catan. What's not to love?
So, I know not everyone loves this term, AI agents. And I totally get that. It's probably not the best term for it, but we don't currently have a better one, so we're just going to use it here. The paper uses it. These are large language model-based agent architectures. So we're taking a large language model, creating some scaffolding around it, some architecture that allows it to play the game, to write code, to take notes, etc. And this kind of approach is very popular. That's basically what Alpha Evolve is, the big breakthrough out of Google DeepMind. The Darwin Goal machine is also very, very similar. It's a self-improving coding agent. You can think of it as a large language model with scaffolding. Same thing with any of those LLMs that play Pokémon, for example, Minecraft Voyager AI game, GPT-4 that played Minecraft and got better, wrote its own code to improve its abilities, etc. That was kind of the same thing: LLM plus scaffolding, where scaffolding is basically stuff that we give it to make it better. We give it tools, the ability to write code, maybe some documentation, etc., so that it knows what it's supposed to be doing. And the results for some of these things have been very good. And here we're looking at an approach of if we can get these AI agents to get good at playing Settlers of Catan. Here's kind of what that game board looks like. It's a fairly complicated game. There's a little bit of strategy, math, negotiations, etc. I actually haven't played it, but I've seen people play it, and it seems very interesting. Seems like there's a lot of depth to it.
So, this paper is called Agents of Change: Self-evolving LLM agents for strategic planning. So they're saying that recent advances in language models have enabled their use as autonomous agents across a range of tasks. This is true. We're seeing more and more of that. But there's a problem: They continue to struggle with formulating and adhering to coherent long-term strategies. We often see papers where these AI agents are really good right out of the gate—superhuman level, better than humans, faster—but over a long enough time period, their ability to kind of, you know, stick with the plot, it kind of falls apart. It's that idea of long-term coherence. It kind of falls apart.
So, this is a very interesting approach because they're trying to figure out if they're able to get these large language model agents to self-improve when placed in environments that explicitly challenge the strategic planning abilities, right? So, that long-term coherence—can we create a scaffolding where they get better with time as opposed to getting worse? And we're going to be playing Settlers of Catan using the open-source Katanatron framework, right? So, some sort of an open-source way to play this game. I believe it's a Python-based open-source thing that allows you to simulate the game, to play the game, etc. Here's kind of what that looks like, or at least the terminal, right? So, you're able to simulate these games. They run pretty quickly, it looks like. You got four players, it looks like, and it quickly runs it through the game. So, you can have bots playing it.
And so the idea here is to benchmark the progression of LLM-based agents from a simple game-playing agent to systems capable of autonomously rewriting their own prompts and the player's agent code. They introduce a multi-agent structure. So we got analyzer, researcher, coder, and player. And these work together to analyze gameplay, research new strategies, and modify the agent's logic or prompt.
So interestingly, Nvidia's Minecraft Voyager was similar to this. It was guided by, at the time, GPT-4. This was really early on, one of the more mind-blowing papers, way before it was apparent where stuff was going. Nvidia released this, you know, their team of AI researchers. And it really was kind of ahead of its time because now, a few years later, now we're getting a lot of the stuff that's basically similar to this framework, but, you know, more advanced, with more advanced models. What's interesting here is this was before the vision models, right? So this was when large language models—I mean, they were text only. There was not a visual component. They couldn't see images. So this whole thing, it was basically kind of like the Matrix where it could just like see the matrix, see the code. That's kind of what it was doing. It would work with something called Mineflayer, which allowed you to write little JavaScript code to interact with the game. It would only give you the abilities that a player would be able to do. It wasn't like a god mode or anything, but instead of having to click to mine something or chop down wood, you can create a little script in JavaScript that would do that for you. So, for example, here was the code to fight a zombie, right? And if it worked, it was tested in the game world. If it worked, then it would be added to the skill library. And this thing did really, really well, much better. And it—it didn't plateau. Notice here, it keeps going, keeps improving.
So, right off the bat, this seems very, very promising because we know that approaches like this can work. And they're saying, by comparing manually crafted agents to those evolved entirely by LLMs, we evaluate how effectively these systems can diagnose failure and adapt over time. Our results show that self-evolving agents, particularly when powered by models like Claude 3.7 and GPT-4, outperform static baselines by autonomously adopting their strategies, passing along sample behavior to game-playing agents and demonstrate adaptive reasoning over multiple iterations.
So this was similar to the Darwin Goal machine where they compared it to, you know, there were some AI coding agents that were made by humans through human innovation. Somebody sat there and kept adding code to improve how well this thing was able to help with coding tasks. And some researchers tried to see if they took a baseline AI coding agent and the task that would just sort of improve itself with this evolutionary search. Each time it would, you know, add some modification, it would be tested against a coding benchmark. If it improved on the benchmark, that modification would be added to its codebase. So it was like attempting to prove itself, testing if that piece of code would improve its abilities to code, and if it did, it would add it to its own sort of codebase, improving itself. This is very similar, but with Settlers of Catan and also multiple sort of agents similar to Nvidia's Voyager.
So Settlers of Catan is a prime example of a multi-agent strategy game requiring players to plan resource management, expansion, and negotiations over many turns. And here's kind of the important thing: the traditional game AI methods, right? So things that were done with just reinforcement learning, not ones that were LLM-based. Well, they have achieved superhuman performance in perfect information games such as chess and Go. So in chess and Go, you see everything there is to see. There's no chance. There's no probability. There's just the entire board, and you're able to kind of observe where all the pieces are, etc. With Catan, you have dice rolls, right? So you have probability, you have partial observability. So you don't have all of the data at a glance. There's things that are hidden from you. You have to kind of guess at what's there. So this can be a little bit more difficult for kind of this reinforcement learning approach. It can be done, of course, but it's not as easy as these perfect information games like chess.
So here we have four agent architectures. One is the base agent. So it maps an unstructured game state description directly to an action. We'll break that down, what it means in a second. We have a structured agent that receives a representation of the game state, available actions, and a basic strategy and natural language for better parsing and guidance. So, here's a prompt example, right? So, you are playing Settlers of Catan, right? It gives you the rules, the board understanding guide. So, it kind of spells out some ideas behind this game and by, you know, the game state, the board state. This is what that looks like, right? So, game status, longest road, largest army, you know, the various resources, etc. This, by the way, seems to be a big deal because when these papers provide something like this, they kind of keep reminding the agent with every prompt kind of its state. That seems to work really, really well. Minecraft Voyager did that, and it never sort of lost track of what it was doing because it kept being reminded. Papers like Vending Bench, right? Where it was tasked with running a vending machine business, it broke down over time, but it didn't have the sort of reminder. It didn't keep telling the state of, you know, the game that it was playing. It didn't update it every turn. So over time, the large language models often just lost the plot. So this seems like a big deal. Like this is a key to making stuff like this work.
Three, we have a prompt evolver. So, it's an evolver agent and a player agent interact for up to 10 iterations to refine, test, and evaluate a prompt for an LLM to play Catan. Then, we have the agent evolver comprising of evolver, analyzer, researcher, coder, and player roles, which can autonomously rewrite its gameplay code between games. So, again, this very similar to Minecraft Voyager in that you have multiple agents kind of taking care of different parts of the game. One is sort of playing the game, one is trying to improve it, the other one's testing it, one is, you know, writing descriptions. It's this—this is a proven approach, and uh, I would guess that it works incredibly well. If you've ever thought about building something like an AI agent for business purposes or something else, I mean, these are really—it seems like these are the recipes for how to do it right. And you know, also some papers show you how to do it wrong. This, so far, it seems like it's doing everything right.
And how they test these models is they compare it to Katanatron's strongest heuristic-based bot in head-to-head games. So there's a logic-based bot, kind of like a simple scripted bot, or here they're saying an alpha-beta search AI, but it's not AI in a sense like with large language models, kind of not like this, right? So it's a—I guess you could say like a more basic version of it. And we're tracking relevant metrics such as average victory points, numbers of settlements and cities, largest army, and other development indicators. So with Catan, victory goes to the first player to reach 10 points, and you get points for settlements, cities, etc. Every turn starts with a dice roll that produces resources, and the active player may trade and build. If a seven is rolled, the robber is activated, blocking a tile and stealing a resource. Yikes.
All right. So the kind of self-evolving part of this whole thing is the agent evolver. It starts with a blank template and over time evolves the ability. So this is the self-improving aspect. And so we have the evolver agent. This is the central coordinator. It reads the analyzer reports, right? So the analyzer agent evaluates the player's gameplay, identifies weaknesses, summarizes key areas for improvement, right? The evolver agent can read those reports. So this is almost like a tool for the evolver agent, right? Right. So this one like analyzes the game and hands it over to the evolver for the evolver to read and figure out what to do next. The research agent—that's interesting. So it handles specific queries about the Katanatron and a broader Catan strategy utilizing local file access and web search, right? So this is kind of like a deep search, and again it provides its deep search results to the evolver agent. That's interesting. So it's—they're combining not just one agent that analyzed the actual gameplay. There's one that can do deep research to maybe find new strategies to attempt. So it does a web search and, you know, goes on Reddit's like what's the best strategy for Catan, and then based on that writes up a report, hands it to the evolver agent. This I got to say is kind of brilliant. We have the strategizer suggest high-level gameplay strategies or critiques past strategic choices. The coder agent translates proposed changes into concrete code modifications. So every time it gets the existing code and, you know, writes, adds changes to it, etc., and then outputs a new version along with a summary of what it did, and then the player agent is the actual AI player that actually plays the game, right? So and this is the thing that gets improved over time.
So this data was collected on a MacBook Pro 2019, 16 GB, and a MacBook M1 Max 2021 with 32 GB over a total time of 60 hours. So, this is fairly accessible if you wanted to run an experiment like this. This isn't out of reach really. I mean, MacBooks aren't cheap, but they're also not tens of thousands of dollars. And the models that we're trying is GPT-4, Claude 3.7, and Mistral large. So, that's the open-source large language model, Mistral. I would love to see this played. We'll see what the results are. But I wonder if you take the latest, like A100 80GB, obviously it's going to be a lot more expensive to run it, cuz you're also—you have a lot of different agents running, each one, you know, having an API bill. Unless you're doing it locally with something open source, they're probably not. They're using Mistral AI API. I wonder why they wouldn't use a DeepSpeed R1. It's very, very cheap.
So the baseline agents, they just play 10 full games against Alpha Beta. So that's that kind of more simple AI approach to playing this game. And random seeds are fixed for initial board state generations to ensure consistency across runs. So they're basically minimizing how variable the games will be between runs, trying to keep everything, you know, as consistent as possible. So it sounds like they're mainly for the initial board state generation, just so you're not completely rolling a brand new world every time. That could be very different, like where you start could have a massive impact. Then the prompt evolver runs, so across 10 evolutions, and then the agent evolver is playing similarly to the prompt evolver. So one is trying to improve the prompt. One is trying to improve the actual agent.
And finally, the results. How well do you think our LLM agents did? Did they beat the heuristic-based alpha-beta, kind of the baseline? Were they able to improve? I'm assuming they started out worse, but over time did they get better? That's the question. And here are the results. So agent performance across models, right? So GPT-4, Claude, and Mistral large. So in all cases, the base agent is here right around 3.6, right? So that's kind of the unevolved, unimproved agent. Structured agent improves 6% for GPT-4, 11% for Claude 3.7, 31% drop for Mistral large. I'm not sure why that is. So, this seemed like this was the worst performer. The Mistral—Claude 3.7 was the best performer. Go Claude Prompt Evolver—22% increase, and for GPT-4, 95% increase for Claude 3.7. This seems huge, right? And then agent evolver, 36% improvement for GPT-4, 40% improvement for Claude 3.7.
So the point here is this system successfully improved agent performance by autonomously iterating on strategic prompts. Although the degree of success heavily depended on the model used, keep in mind we have probably better-performing models now, or you know, if you're willing to pay more. Claude 3.7 exhibited the most significant strategic advancements, systematically developing detailed strategic prompts that outline clear short-term and long-term plans, including precise settlement placement, resource prioritization, development card usage, and robust response strategies against opponent actions. So, Claude 3.7 did great, right? Up to a 95% improvement over the base. GPT-4 was mid, Mistral large was the least effective. The biggest limitation is the underlying LLM, right? So the better the LLM, the better the outcome is. And this is kind of that whole thing when Sam Altman was talking like if you're building a startup, you don't want to be trying to fix something that's currently wrong with AI. You want to have some sort of a system where the better the models get, the better the performance of your entire business. And it's the same thing with papers like this, with systems like this, right? So imagine the improvements we're seeing now with these models. Now fast forward a few years, imagine kind of those next-generation models. If you just plug those in and keep everything else the same, it's likely that those improvements that we've seen, right, a 95% improvement, we're probably going to see even better ones if we just improve, you know, the model, the underlying model. That's an important point to grok, so to speak, right? These models will get better at self-improvement just, you know, as they're improving, as they're growing, as they're getting better. They will also get better at self-improvement or improving systems like these autonomously.
One of the things is they've only done 10 steps of evolution. And as you can see here, you know, especially with Claude 3.7, our best model, right? It—it keeps improving even—let's see, what is this? This is, you know, evolution 7. There's—there's still improvements here and here. So it plateaus after seven. Although it might get better, you know, if you keep going, and if you projected 20, 30 evolutions forward, but as you can see, it's not like all of the improvements are early on and then just nothing. Even towards the end of the evolutionary steps, it keeps improving. I wonder how well it would do if we allowed it more time, more evolutionary steps. Like is it just going to plateau from 7 to, you know, 100, or is there going to be, you know, some modest improvements if you just keep going? Obviously, the researchers likely had limitations in terms of funding, right? I don't know how much those—this would cost, but it could be somewhat expensive.
So, absolutely phenomenal work. I love this project. I wonder if it's open source somewhere cuz I'd love to play it. Now, the Catan—I forgot it was called Katanatron, that is open source. That's on GitHub, so you can download it and use it. You can probably attach your own API, your own large language model to it. But this is yet another example of what we probably would call an AI agent that's able to recursively self-improve. And we're seeing more and more of these. So kind of like the recipe for building these things. We're seeing more examples of it. We're beginning to understand what works, what doesn't. Really exciting time to be alive. And I love the fact that they're using games to demonstrate some of these things. And I can't wait to see where this goes. Thank you so much for watching.