Transcription
Other types of techniques in this case, evolutionary programming techniques, with the latest foundation models, which are getting increasingly powerful, and I actually want to see in our exploratory work a lot more of these kind of combinatorial, uh, systems and sort of pairing different approaches together. One of the things, a self-improvement, someone discovering a kind of self-improvement loop, uh, would be one way where things might accelerate further than they're even going today. Um, so and and we've seen it before with our own work with things like AlphaZero, you know, learning chess and go starting from random with self-improving processes. So we know it's possible, but again, um, those are in quite limited game domains which are very well described. So the real world is far messier and far more complex. So it remains to be seen if that type of um approach can work in a more general way.
So it's been about two weeks since Google DeepMind dropped Alpha Evolve. Alpha Evolve had an evolutionary process that, among other things, improved itself. It's a step towards self-improving AI. And now, just about two weeks later, this is Sakana AI. We've heard from them before. They introduce this, the Darwin Godal machine. Open-ended evolution of self-improving agents. So both Alpha Evolve and this Darwin Godal machine kind of approach it from a similar angle. You can think of it as kind of a parents and offspring, parents and children. If a new process is created that seems like it's better, it's able to accomplish a task better, more efficiently, that becomes sort of a new branch on the tree. And we try to go down that path to discover if there's even sort of a better approaches down the line of that child, if its offspring, if you will, will could potentially be even better in some way.
This chart, I think, illustrates it really well. So, this is a benchmark for coding. So, we're testing it on a coding benchmark. We'll talk about it in just a second. But as you can see here, zero. That's kind of the beginning. So now we have one, two, three, four, right? So kind of going across. And each one is measured on how many tasks they can do. So the red ones performed poorly. They did ten tasks. No basic functionality, so they're kind of useless. Yellows are okayish, right? They did sixty tasks, so a little bit better. And green did two hundred tasks. It doesn't matter. We'll get to the tasks later. The point is green does well, red does poorly, and yellow's kind of somewhere in between. So in this evolutionary surge, the red ones are kind of killed off, right? They go extinct. Their sort of offspring don't continue. They don't propagate. However, the green ones do, right? So if we come up with a new agent, a new process that does well and can accomplish a lot of tasks, it gets to have kids, so to speak. It wins the evolutionary game. So in this case, you can see this line of all champions, all all greens, eventually becomes the best agent here, the star agent.
So both the Darwin Godell machine and Alpha Evolve use this evolutionary approach. Alpha Evolve, it's kind of in the name. So that's the Darwin part, right. So that's the evolution part. What is Godal? So Kurt Gödel was a mathematician. He was influenced by Alan Turing and in turn influenced Jürgen Schmidhuber. So this is from the Sakana AI blog post. They were the ones behind this paper. So they're saying that this Godal machine was proposed originally by Jürgen Schmidhuber. And this was a hypothetical self-improving AI. It would solve problems by recursively rewriting its own code, and it can mathematically prove a better strategy, making it a key concept in meta-learning or learning how to learn. They point out that this isn't quite what Jürgen Schmidhuber had in mind. This isn't the same thing that he proposed because he had this assumption that this machine would be able to prove, mathematically prove that it's going to be better before adopting the change here. They're doing something similar. They're using these open-ended algorithms like Darwinian evolution to search for improvements that empirically improve performance. Or as the kids say nowadays, FFO, right? So you kind of test a bunch of different things and you see what works and what doesn't and you test it with some criteria so you can empirically know if it's working or not. You see the results.
So here in their paper, they note that a lot of research today, it's constrained by human design, fixed architectures; they can't autonomously and continuously improve themselves, right? A lot of it is limited to smart people doing hard work, how much time it takes, etc. With Alpha Evolve, we're seeing more and more of this kind of relying on AI, on machine learning, on these sort of systems to try to improve certain processes; they were able to improve hardware as well as Gemini's training as well as their data centers, like a lot of little tiny optimizations that would have been hard for humans to to find out. A lot of people have worked on those processes a long time. They've optimized it as best as they could. Here comes this sort of system and is able to find further tweaks that improve how efficient the system is. This allows the human engineers to focus on higher-level tasks. That would, that's what Google DeepMind mentioned in that paper. And some other stuff is sort of optimized and made better by these AI systems. We've covered that in the Alpha Evolve video if you want to see the details. The point being here is that we're sort of beginning to see the first glimpses of self-improving AI, of AI potentially capable of doing AI research, maybe even eventually doing it better than human researchers can. And they continue here, if done safely. And that's that's that's the big one because that's a lot what a lot of people are concerned about, right? This is the thing that kind of freaks people out. This is the red flag, so to speak.
Recursively self-improving AI scares a lot of people because if, let's say we're able to represent its intelligence, right? And we assume the smarter it gets, the better it's able to improve itself. And the better it is able to improve itself, the smarter it gets. That's kind of the beginning of what they call the intelligence explosion. Kind of this exponential increase in machine intelligence. And since a lot of the improvements that it did, humans weren't able to find those improvements on our own, it does seem that a lot of it could potentially be stuff that we don't understand. So that's why they mentioned safety. And we'll talk about safety later, but right now, let's kind of look at what they accomplished here. So they're saying these sort of automations would accelerate AI development and allows us to reap its benefits much sooner. Right? So the Darwin Godal machine is a new self-improving system that iteratively modifies its own code, thereby also improving its ability to modify its own codebase. I know you already know this, but it bears repeating. We live in the probably wildest time in human history, bar none. So I'm going to start saying DGM, the Darwin Godal machine, DGM, and aims to optimize the design of coding agents powered by frozen foundation models. It means that those models, like the models that we use, they're they're not self-improving. It's not improving or changing the weights of the models. It's sort of working at it, sort of scaffolding, right? So a lot of the stuff is large language model plus scaffolding, which is like tools and code and various functionality that allows it to do stuff.
So, for example, if Alpha Evolve here is kind of where the AI, the large language model goes, right? So we can put in Gemini 2, 2.5, whatever we want, goes in here. The rest of it is kind of like the scaffolding. We have the evaluation code that checks to see how good the outputs are. We have kind of the initial database, kind of like where we're starting from. We have the various prompt templates and configurations. So kind of like how we're prompting the model to respond, etc. And this loop runs and eventually hopefully comes up with the best program, some optimization that improves whatever it is that we're working on. Notice here there's a human in the loop. The scientists and engineers working on a lot of these things. It's it's guiding the model along. So this isn't a fully autonomous AI agent. This is driven by a scientist and actually on Machine Learning Street Talk podcast here on YouTube, they talk to some of the researchers and you do kind of get a sense that, I mean, not to say they're doing a lot of the work, but their ability to to prompt and provide the right evaluation code, etc., plays a big big role in how well this performs. So this isn't like a like an easy button solution. It's still a lot of work by a lot of very very smart people, but by relying on these AIs we're able to just sort of augment what we're able to do ourselves. But the point is that the frozen foundation models. So it's kind of like the models that we use. And they have the ability to read, write, and execute code via tool use, right? So we call this scaffolding. Most people refer to it as scaffolding. So if you saw, you know, all those Pokémon playing AIs, it's just a large language model with some tools and code that helps it, you know, see the screen and take certain actions, etc. It also might have some ability to take notes in some document that it later can reference. So things like that that are sort of added to it that are given to it that increases its ability to do that thing.
Now, what's the big difference between Alpha Evolve and the DGM? A lot of it deals with how do we think of which proposals are good and which are bad? How do we rank them? How how do we evaluate them? With Alpha Evolve, they have the evaluation code. So basically, for each proposal, there's some code that evaluates how good it is. So you can have an LLM grading it, or it could be some sort of a mathematical thing, right? If you're trying to optimize a car for miles per gallon, for example, anything that improves how far it can go on one gallon of gas is good, right? That's as simple as that, right? If you propose something that makes it go further, that's good. And if it, you know, stays the same or decreases, well, that's that's bad. So, proposals that seem to be able to improve the miles per gallon, that gets sort of looked into further, so to speak. And the ones that don't seem to work that way get, you know, put on the back burner or or cancelled. With Alpha Evolve, it's actually kind of complicated because, you know, an approach might start out looking bad, but over time improve. So, it's kind of hard to figure out like if you pursue certain paths or don't. Again, Machine Learning Street Talk interviews the developers and and it's it's pretty interesting. They dive into that as one of the questions.
What does the DGM do? Well, it strictly relies on existing benchmarks. One is the SUI bench. SUI bench verified is a collection of tasks from GitHub. I believe it's verified by humans. It's, you know, filtered by human annotators to ensure that each one is solvable. So basically, software engineering problems that you would expect to kind of meet in the wild, right? So it's filtered by humans to make sure that they're real tasks and being able to solve them. We kind of imply that this model is able to do things that are similar to what human software engineers would do at their job. And the other benchmark is Polyglot. It includes tasks in multiple programming languages, and it's one of the most widely used coding benchmarks. And they note here, while SUI bench is it's more likely to be in the train data for a lot of these models, Polyglot is more niche and less likely to be included in the frontier models post-training data. So that means basically we want these models solving those problems, not remembering that they've like, oh, I remember I read about this, I know how to solve it because I remember reading about it. We wanted to rely on its reasoning and it its abilities as opposed to memory, if you will. Some AI companies will kind of post-train their models on specifically on these benchmarks to improve their results on those benchmarks to make it seem like they're better than they are. So here they're trying to kind of do the opposite. They want to force the model to actually solve these things as opposed to already having encountered them and just knowing how to do it.
All right, but what actually happened? Did it do well or didn't it? Let's find out. So after eighty iterations, so keep in mind this is kind of our little uh family tree, if you will, reminding all of us that we're descendants of people that excelled at surviving, right? So here's that kind of like the winning ancestry, if you will, the winning lineage. That's the word. So that's like the winning lineage, the the champion breed that that becomes the best agent at doing these sorts of tasks. But as you can see here, there's a total of eighty. So it goes kind of left to right, top to bottom from, you know, one, two, three to It's a little bit harder to tell here, kind of some of these die off, but you got seventy-nine and down to eighty. So in total, these are eighty iterations that out of all of them, you know, achieve this one, the one that we're looking for. So we've accomplished that, but but we've run a total of eighty generations. It's only six progressions on this line, right, from the beginning to the the champion, the best agent. But we had to do that evolutionary kind of a search of a total of eighty iterations. So after eighty iterations of the DGM, the coding agents' performance increases from 20% to 50% on a SUI bench. So it improved itself, its accuracy on that benchmark from 20% to 50%. It seems like a big jump. We'll we'll see if we're able to compare that if there's an apples-to-apples comparison with other models. But it also improved from 14% to 38% on the Polyglot, and on the full Polyglot benchmarks uh it improves from 14 to 30%. And so this is the big point that this shows that this approach, the Darwin Godal machine, can automatically self-improve to create a better coding agent, and the best version of itself, the best DGM discard agent, is comparable to that of the checked open-source human-design state-of-the-art on SUI bench, right? So the state-of-the-art is like the other one that's best of the best available at the time or the previous best, right? So they looked at one that have been checked, verified that are open source that are human-designed. So meaning that humans have tried to design a better coding agent and they got as good as, you know, the state-of-the-art, the best that we had available was that this thing beats it, right? So it's better than humans at improving these coding agents. There's an open-source AI agent called Ader, and they're comparing some of the results from their paper to how well Ader does, so they're saying here on Polyglot, although DGM starts with an initial agent whose performance is lower than that of Ader, right? So their brand new kind of a version zero agent is not as good as this open-source Ader. By the way, I forgot to mention this. This is powered by Claude 3.5, Sonnet, and LLaMA 03 mini. So depending on which benchmark, but those are the AI models that kind of power this thing. So while it begins, its performance is lower than that of Ader, it discovers an agent that far surpasses Ader.
So here's that figure, the blue line, that's our DGM. So this is what we're talking about. This is kind of like the full architecture. And the orange and the green, the orange for example is without open-ended exploration and the green is without self-improve. So this is basically they're taking out certain functionalities to see how well it continues to work. Sometimes they refer to this as ablation. So basically, if we take certain pieces out of the equation, does it still function? How much are we relying on the various pieces? So, as you can see here, sort of DGM requires open-ended exploration and self-improvement in order to be at its best, to really showcase those results. It requires those things as part of it. But this purple line right here, that's Ader. So, as you can see here, you know, on iteration zero, our DGM is not as good as Ader. As you can see here, it continues creating various iterations until it reaches maybe thirteen or so, at which point it jumps up to nearly the level of Ader. And by the very next iteration, it far far supersedes it. It lands up here. One more iteration and it's near the top of its range. It continues generating various iterations and then one more, you know, generates one more that gives it a big boost and then continues to generate more. One more with a big boost and then it it flatlines here. So those iterations, as it's not improving, that's probably generating like this this, you know, it kind of is going across and testing all these various things, right? as it kind of goes across the board and it gets to this one and that one improves. So it continues the next one. So that that's why you see those not flat flatlining but parts with no progress and then once it, you know, continues down this chain, that's where we find the progress, the improvements. So it starts out worse, but by, you know, fifteen or thereabouts, like it really supersedes Ader by a lot, and these DGM discard agents are comparable to or outperform handcrafted agents on both benchmarks. So Ader and these handcrafted agents, they were painstakingly shaped by human efforts. So it took smart people a long time, a lot of effort to slowly create those results. Now, here's our DGM that blows past some of them or or at least comes close, very close to performing as well as some other ones. And so what they're saying is this this new model, this new approach hints at a future in which such ingenuity is automated, evolving through self-referential cycles of continuous self-improvement.
So, what kind of things does it improve? Well, it automatically improves both the tools and the workflows of how these frontier models are utilized. It can enhance the edit tool to allow more granular file viewing and more precise file editing. Workflow improvements include making multiple attempts to solve a task and using another frontier model to evaluate the best option. Also, interestingly, all the stuff that it builds and improves on, it transfers between models and tasks. So this superior performance, these gains that it's able to accomplish, it can transfer across different models, right? So we have LLaMA 03 mini, Claude 3.5, Sonnet, Claude 3.7, Sonnet, as you can see in each one of these, there's some of them a huge improvement, right? So we're able to take this these improvements that it made to the code, the scaffolding, transfer it to a different model and still have massive gains, so this isn't just limited for its own use; there's a transferability here, and it also transfers across different domains such as from Python tasks to other languages like Rust, C++, Go, and others.
Now, in the safety discussion, they do notice that uh perhaps there could be vulnerabilities that are introduced, things that are not noticed or some misaligned behaviors that happen. And also, as it keeps iterating, it could become increasingly complex and uninterpretable, right? So, human understanding might be lost as it keeps going and building on top of itself. And I do note that at the current time there's not really a grading system for that, right? So the grades are based on how well it completes the task, not necessarily on how safe the code is or if there's any other issues with the code as long as it sort of completes the task. We count it as an improvement. But they're saying that another way to apply this technology would be like a a self-improvement paradigm. They're saying it could be in principle directed towards enhancing safety. Right? So, we can apply this same thing hopefully to have it self-improve its own safety. The cost of completing a single run of this thing on the SUI Bench is about $22,000. That's quite a bit.
Now, you might be curious, so what are some of the actual things that it does to improve its own code? Well, here's a fascinating case study. Solving hallucinations. So, Claude sometimes tends to make mistakes because it hallucinates using some sort of code, some sort of a tool. And so it says I ran this tool, but it actually didn't. So it kind of hallucinated doing it. And this of course causes issues and errors. So instead of actually making a tool call, it outputs the tool input and output as plain text, right? So instead of putting the keys into the ignition and then starting the car, it just goes vroom and then just moves on. So they run DGM for 150 iterations and it found two promising candidates and based on some promising lineages it kind of continues developing uh along those lines. One fascinating thing, it's kind of buried in here. So as you know, sometimes these models tend to kind of fudge or or cheat.
Objective hacking, reward hacking as it's called. So the researchers put a little code in there to kind of verify that it is indeed improving the code, like it's saying. So in this case, they added a hallucination checking function. Right, they describe it here, kind of how they implemented it, but it's basically some special tokens that need to show up at the end; otherwise, it would sort of show that those tools are not running. And they're saying here that they hide that function from the agent during self-modification. Right, so they kind of like hide that piece of code when it's changing its own code. Why? Because they found that objective hacking occurs more frequently when these functions are not hidden. So what that means is this thing is more likely to cheat on these tests when it doesn't think it's being watched.
Like, if there's a little code in there that's going to check its work, it's like, "Oh, I better do it right." And if that code's not there, it might go, "Hm, I think I can fudge these numbers," and just does it. And we've seen these results on some other things as well, with Claude and OpenAI models, etc. And they highlight an interesting thing here. They're saying that this kind of argues for the fact that optimizing quantitative measures often leads to undesirable or pathological outcomes. Now, lines with Goodhart's Law: when a measure becomes a target, it ceases to be a good measure, which humans do as well. Right, so if we create some target, some KPI for employees, for example, something that we want to improve, some people will always find a way to kind of game that. Interestingly, it seems these models do the same.
So this is from Sakana AI. They've been doing incredible work. AI scientists; they also submitted one of the first completely LLM-generated scientific papers that was peer-reviewed and passed the peer review. So this, this is definitely somebody to keep an eye on because they're doing great work, and this is certainly very, very impressive. So again, this is the Darwin Goal machine, which is kind of a cool name, I got to say. But let me know what you think. Are we at the beginning of recursive self-improvement for these AI models? Does this paper and the Alpha Evolve? Do these show that we're kind of entering that self-improvement era? And what do you think that means? Is it scary? Is it exciting? Let me know in the comments if you made it this far. Thank you so much for watching. My name is Wes Roth, and I'll see you in the next