The Fractured Entangled Representation Hypothesis

So it goes back to these observations from Pickreeder, um, which is this picture-breeding website where neural networks—which were a special kind of neural network called compositional pattern-producing networks—were bred or evolved by people to produce images. Um, and this is a really unusual and exotic form of training that's very different from modern deep learning; it's completely different from it, which is one of the important points about it.

Um, and so when people were able to actually breed interesting-looking pictures, there's a lot of lessons we learned from that. Actually, the novelty search algorithm and things like that came from those lessons. But there was this one dangling lesson that just hasn't gotten out. And that's what this paper does. And that lesson was that the representations—the underlying representations of these images—which are basically represented, encoded by these neural networks—are absolutely incredible, amazing. And there was like no good explanation for how they could be as good as they are.

And we noticed this very early on, um, that like they have unbelievable modular decomposition, which means that it's almost like it was engineered by a person. You know, it's like there's a there's a network that generates the image of a skull, and it—the network—has decomposed it such that like there's a component of the network that's responsible for the mouth and can do things with the mouth, like open and close the mouth, or there's another dimension that can make the mouth smile. Um, and things like that, which are not—what's really amazing is not data-driven. Like we're talking about dozens of iterations in terms of like the amount of search that's involved to get a representation like this.

And so, um, I've always—there's—we've had many discussions over the years like why—why is the representation so amazing in these Pickreeder images? Um, why does it have this unbelievable modular decomposition? Um, and and and extremely efficient and compact. Um, and so one thing that, like, at one point I did with Joel Lehman was we were kind of just um playing around to see like, well, what if we tried to get SGD to produce the same images? Like what—what would it be different internally? Uh, the image—the way that it represents these images—and it was just dramatically different.

Um, but then we did nothing with that. Like we just kind of like left it. We both—Joel and I—knew this thing, but we didn't really do anything with that information. But I've been just mulling it for years that like there's something there that's really important. Um, that like with this weird kind of open-ended search process, you get these incredible representations, and then with conventional, like objective-driven SGD, which is like the backbone of all of machine learning right now, um, you get a completely different kind of garbage representation, just total spaghetti. Um, and we—we—we used—we came up with some terminology that we put in the paper to—to more like clearly articulate what these differences are. But basically, you're talking about just amazing versus garbage.

Um, and so like the question that the paper addresses is what does this mean? Which is something I think that has endless repercussions and potential implications. Um, like the fact that we're basing the entire field on something that produces this complete garbage under the hood. Um, and does this mean anything? Now, I think if—if you just saw this—by the way, the fact that it's images makes it easier to see like that the—that the representation is garbage, like the fact that it's a network that produces one image, because what that means is that we can visualize every single neuron, what subcomponent it's actually computing. It's very easy because it's just a single image. So, it makes it very easy to see all at once how it actually computes the image. So, we can see the internal representations.

And if I just showed you that for SGD, like if I said, "Here's a skull. I made this network. Reproduced this skull, and um, the internal representation is crap," I think it would be not like hugely resonant in the field because it'd be like, "Wow, it's not super surprising that it's kind of hard to understand this like really complicated entangled representation. It's just emergent from how SGD climbs these gradients." But the thing that I think makes this really intriguing—like the reason that the paper is—goes beyond just that—is that it gives you something that otherwise could never exist, which is a counter-example, that there actually do exist networks that don't have that issue. You would think that that's just intrinsic to neural representation, that somehow they just look like kind of entangled messes, and that's just the way life is. But clearly, is not how life has to be.

Um, and so because we have now actual concrete examples from Pickreeder that show beautiful internal representations. And so I think that the paper's trying to get at what does that mean? Like how should we interpret this? Should we perhaps aim for algorithms that actually do achieve these kind of, um, you know, really amazing kinds of modular decompositions? Um, or should we—or should we just say, well, no, that's—that's just not important; uh, let's just forget it and dismiss the fact that we know they exist. I think that would be—that seems to be a pretty premature move to say that. So, so there's something here, I think, a deep, deep lesson for us to learn in the field um about internal representation which just is totally off the radar. It's way, way off the radar of anything that's being discussed.

Um, you know, mechanistic interpretability is not on the same page with what we're showing here. Um, and so, of course, it—it can play a role in—in further like interpreting this result. But this is a new thing to look at and for us to contemplate and try to understand what the implications are. Don't we just need to have humans in the loop? I mean, wouldn't it be great if we could just have these evolving topologies right without humans? Wouldn't it be great if the systems could themselves learn the categories, the abstractions? And is it just a matter of missing data or is there just a fundamental gap in capability? Yes. So it's clear that part of the explanation for these amazing representations in Pickreeder is that humans were in the loop. So that's part of the explanation.

Um, but the question is what's the deeper lesson? Like what did the humans do that caused this to happen? It clearly wasn't intentional. I mean, the humans weren't thinking about the underlying representation at all. They're just choosing things they like. So, why does that lead to this virtuous type of representation? And I think, you know, there's one—one really interesting lesson that this shows is that it matters not just where you get, but how you got there, you know, and that—that's like something missing right now, you know, because we tend to just care where you get—like we look at the benchmark score, you know, in the field, and it's like that's the result. It's doing really well. It just passed the math olympiad. But what if it matters how you got there? And you could say, "Well, why does it matter how you got there if it performs well? It's like the same either way." And that's what the paper's all about, basically, is that like, you know, you can have these two things, these two networks, they both output a perfect-looking skull, exactly the same skull. One was found in this completely different open-ended way with people guiding the search. Another was found objectively through SGD.

Um, and so very different trajectories through the search space were taking radically different—and which means that under the hood is radically different representation—and so you say, well, what does it matter then? What's the underlying representation? Well, it can matter a lot—how you represent the world—and so—so then like the question is, with humans, like is that the essential ingredient? And I think—can't be—it can't be that the only way to get good representation is to have a human guiding the search; there's a deeper explanation—like why were the humans successful at getting to these amazing representations? And it has something to do with the underlying open-ended nature of what they were doing. Like, in other words, on the road to getting an image of a skull, they were not thinking about skulls.

Um, and so like when they discovered a symmetric object—like an ancestor to the skull—they chose it, even though it didn't look like a skull, but that caused symmetry to be locked into the representation. You know, from then on, symmetry was a convention that was respected as they then searched through the space of symmetric objects. And somehow this hierarchical locking in over time creates an unbelievably elegant hierarchy of representation. Um, and it's—it's plausible, I think, at least for me, that there are algorithms that don't have humans that could take similar trajectories through search space. Maybe not quite as like perfect as the human trajectories, um, you know, I—I wouldn't be surprised if we can't actually hit that ideal, but there's probably a continuum where you could come closer—and then what are the implications? You know, if the order that you encounter the principles that lead to your final understanding of the world matters for how you represent the world and therefore for your ability to be creative in the future, then does it matter the order in which we allow these large models to encounter the different principles that they encounter on the road to total understanding of everything in the universe? And I mean, I would guess like this implies probably yes, it probably matters. And that opens up a huge range of possible creative opportunities for alternative ways of thinking about training that would lead to better representations.

You brought in the open-endedness aspect, which is fascinating because you're saying they weren't looking for the skull, right? So, what were they looking for? They were um composing these primitive basis functions that they have in their mind. So they know that symmetry is good. Where did they get the symmetry idea from? Well, it must be somehow gleaned. And so actually our function space is restricted in some very important way. We know that we have certain things we can compose, and we know that we can compose them in certain topologies, and we know that invariably if we follow that trajectory, we will land on interesting things, even though we don't necessarily know exactly what we will land on.

Yeah. Yeah. That—that—that—that's how this happened. I mean, the people—people in general have intuitions about potential, not just about where we are right now, but that this might lead to something. And in fact, it's very complex because like the more people play with Pickreeder, the more they get intuitions about Pickreeder itself. So they start to understand what might lead to what—what's promising, which is different than what's aesthetically pleasing in—in the moment. Um, and so that they're—they're—they're both enjoying the image and predicting like, well, where might this go? And it's not like they're predicting they're going to get a skull, but they're predicting symmetric things are really interesting and beautiful and—and—and let's see what other kinds of symmetries come out of this. Like they're thinking that, um, and so that helps to—to get this kind of virtuous ordering um which causes like a sequence of lock-ins of different conventions of—of increasing complexity—which then creates this like amazing representation underneath the hood.

But I want to—I want to point out though that like this view of the world is so radically different from the data-driven view that we live in right now. Um, you know, it's—what's—what's really fascinating to me about it is—not data-driven. Like, you know, we—we think about, you know, when you say like eventually if you experience enough of the world, like you say, of course, you might expect that your representations start to mirror just the way that the world is. Um, and you get this kind of isomorphism maybe between like the—the—the—the organization of your brain and the organization of outside reality. Well, that's a data-driven view, like—and basically almost everything we think about is data-driven. I mean, the bitter lesson is sort of like a data-driven philosophy. Um, but like this is like—what's so interesting is this is totally contra bitter lesson, you know, cuz like what you're saying here is like we see almost nothing of the world—Pickreeder knows nothing of the world; there's no pre-training at all—start with some blobs—over a few dozen iterations. I mean, dozen is crazy—like it's like peanuts. Like we're used to millions, billions. Like we—we don't—that's not what we do in our field. But here we have dozens of iterations—not enough to be exposed to almost anything—it actually somehow finds within this—uh—newly constrained space of dimensions that has been discovered by humans things like the difference between opening and closing a mouth or smiling and not smiling.

Um, and like those dimensions exist now in this space of the network but not because of data. They're whole cloth denovo discoveries which are not data-driven. So there's no bitter lesson. It's just out of nothing. Um, and there's even more crazy ones—like there's like—like the apple one which is in the paper. It's in the appendix. This apple has this unbelievable weight, a single weight in the apple representation which is a single continuum that if you move along that continuum, you swing the stem of the apple back and forth—like from left to right. Maybe someday we'll put an animation over this so we could show—like I could give you the animation. But you can see the swinging stem is one dimension, and it's—it's like three-dimensional, you know, it's not just like a two-dimensional thing. Like it moves as you would expect in a rotation in three-dimensional space. It has a shadow underneath it. It's like a green leaf. The underlying apple, which is a symmetric object, is not disturbed at all. It's totally independent. It's been decomposed. And then there's this one thing which is this stem swinging.

And so, you know, I'm—what I'm saying is it's absolutely incredible. It's mind-blowing that if you think about that as a world model, like it's an actual true hypothesis about the world. This is the way that stems look when they swing. But this model has not been trained on anything in the world. It's never seen swinging stems, let alone apples at all. Um, I can almost guarantee you that in the training uh trajectory itself, there was no swinging of the stem. After all, if the stem started swinging, that would mean it already had that ability. So, it's just circular to argue that this is just something that arose out of the fact that the—that the representation is so elegant that it somehow has an internal—I think of it as a hypothesis about the world, which is correct. And you have to ask yourself, how many of our hypotheses are like that instead of data-driven hypotheses? Because we do sometimes have these unbelievably elegant underlying representations of the world that often are unique—unique to an individual, not necessarily universal across human beings. Like everyone's representation is unique. And so this is a totally different way of thinking about representation and knowledge and how it comes to be.

Yeah. And the thing I'm trying to understand is we can agree that humans have this incredible abstract model of the world. We have a language of thought, and the way we understand things is through these compositions, these topological compositions of these little basis functions, right? Yeah. I mean, I think it's true that [__] is a kind of psychology experiment that—that is a dimension of what we're seeing. Um, and it's true that intelligence and compression often equated, and they clearly have a relationship. I think in the—in the paper we—we talk a bit about this and even speculate that, you know, there may be something more to say than just compression. You know, in the sense of like the factored aspect—how you factor matters. It's not just that it is compressed. You know, it's like if I know that a face is composed of eyes and a nose and a mouth, and I actually factor those out, um, even if you had a greater compressed version of the face that didn't factor those out, I would prefer the factored version. That's still in some sense better. Um, then I can, you know, generate new faces in a principled way. So, I'm not—I'm not sure that it's always like the maximally compressed version is like the most so-called intelligent, depending what we mean by that. Um, there's—there's multiple factors to consider, but obviously compression is a virtue, um, you know, and so these extremely inefficient uh like representations that—that you see in just like a regular SGD—is obviously that's part of the problem.

Um, and so I think though that um like speaking about the—the human aspect of it um just to go back to that for a second—it's just one—one thing that I think is important to—to think about is just that you can extrapolate outside of Pickreeder um like this—this principle of searching through regularities or—or finding um good isomorphisms with the world in some kind of sequence—like that's a just general way that people do discovery. So in the sense like that—so in other words, it's not just in—in this kind of like very almost psych-test-like environment that—that kind of stuff happens—like that's just a general aspect of human exploration—intellectual exploration—and so when we do that, you know, you can imagine like the difference between somebody who learns calculus from a textbook versus someone who invents it for themself because they were curious, but they both end up knowing the same thing, you know, so they both take the test, and they both get a good score on the test, but it's just pretty obvious that the person who found it through their own independent exploration is probably going to be do much more interesting math like after that test.

Um, but why? It's because like they—they went through a Pickreeder-like exploration process. Of course, this is easier said than done. Most people won't do that. Um, but it's just like the way that you got there matters a lot—like all throughout life. If we did have a large language model that could understand abstractions at the level we do, why couldn't we just run it autonomously, and—and why couldn't it step by step know what's interesting and just traverse this fogyny and find interesting things and self-reflect? You know, it—it—it feels to me that the missing link is having the correct level of abstraction and being able to do this iterative open-ended search. They can't do that because they simply don't have the abstractions. We have the abstractions, but could we actually build an autonomous open-ended system that could find the abstractions the way we do, or do you still need the humans? Is—is there still something missing in this respect from large language models—like their ability to leverage abstractions to—to really think creatively out of the box and so forth? There's different ways you could say this, but that's basically the big question. I think—and I basically agree that—I don't think that what we have now can really match the very best of human creativity. Like I think that's safe to say. I think a lot—a lot of people would probably agree with that. Um, it can do some level of creativity, what I would call derivative creativity, um, you know, which is sort of like the—the bedtime story version of creativity. It's like you ask for a bedtime story, you get a new one; it's actually new. No one's ever told that story before, but it's not particularly notable. It's not going to win a literary prize. Uh, it's not inventing a new genre of literature. Like there's basically nothing new really going on other than that it's a new story.

Um, and so that's derivative—like in my nomenclature, I call it derivative. Um, and so that's—that's pretty much I think where we are. And so when we do things like uh try to uh leverage these models to explore really important creative spaces, I think that's a—that's a um an obstacle. Um, it doesn't totally stop us because there are ways of getting out of distribution even with that limitation—which like, for example, wrapping evolution around it—like—like we did with evolution through large models—like uh recently DeepMind does with Alpha Evolve—like you can actually get out of its own distribution, but I think it's an inefficient way, you know, compared to the human mind which leaps through levels of abstraction that it—that it's encoded basically in its representation, which then points back to I think the lesson in the paper, um, which is that it could be that part of why we're sort of trapped in this box of derivative creativity is because we—we don't have these really uh nice style of—we call them—unified factored representations. Um, uh, in other words, like concepts are unified, and it's well factored into the different components that actually corresponds to what's interesting within the domain. So they don't—maybe they're lacking. So they—in other words, they have what we use the other term for the bad kind.

Of representation, which is a fractured, entangled representation. Concepts are fractured into pieces and entangled with each other in ways that are inappropriate. And so maybe it's because SGD naturally does produce these fractured, entangled representations. At least conventional SGD, the way it's being applied here. Not necessarily saying that no version of SGD can actually create the more elegant form, but the way that we apply it conventionally, it tends to produce this fractured, entangled representation. Maybe that's part of why it doesn't have access to the abstractions, as you put it, um, that are necessary to do truly transformative type of uh creative ideation, um, that would require like a very well-factored version of the world, which is just lacking.

And there's lots of evidence of the problem uh with um having entanglement and fracture in representation. Like we go through some of it. In one section, we give a lot of examples from more recent large models. We don't just look at PiqRay and images, and we give hints of it because it's hard to get direct evidence because you'd have to look under the hood at the actual representations, which is quite complex to do, of course, like for things that are not just images. Um, like con concepts and knowledge, of course, is very complex, and the whole field of mechanistic interpretability is trying to grapple with this, but we give some kind of hints that like there is evidence of this kind of entanglement and fracture. And so, so maybe it's a result of the fact that we're doing this conventional type of objective-driven search that generally kind of gives you the lowest common denominator version of representation, which then is going to be a creative straightjacket eventually when you're trying to get outside of the box in a way that it isn't for PiqRay networks, which can imagine things that are almost unbelievable, like the swinging stem of the apple, um, even though it's not part of a world that it's ever experienced.

Um, and so I think there's a lot to learn in representation that can help to address what we intuitively perceive as a limited ability um to be creative and abstract. Basically, there is a spectrum of representations. The problem with neural networks is it can't distinguish the good ones from the bad ones. And what Kenneth is basically saying is that we don't have to do it the the the lottery ticket hypothesis way, which is where you start with an infinite number of degrees of freedom and then like a block of clay, you kind of carve it down afterwards. The really smart thing to do is to build it up from first principles at the beginning. So, you know, you actually build these neural networks piece by piece. But, you know, he's he's advocating for a non-stochastic gradient descent, possibly an evolutionary method of doing this. But honestly, um, Keith, the the the the thought that I was left with is he thinks that there is an algorithm which can do this.

And the reason why it works in the real world is because we have this epic panplay of computation. The universe is executing all of this computation at at the at the sort of like minus minuscule level, right? And that is that is the physical process which gives rise to evolution and intelligence and all of the interesting phenomena we want to capture. And even though Kenneth is absolutely on the mark when he points out the reason why neural networks aren't creative and indeed what creativity is, no one has better intuition of evolution than than Kenneth. He still seems to be slightly off the mark when he believes that we can construct an algorithm completely disconnected from how the physical world works or largely disconnected, and we could still capture some of the phenomena. Right. Well, I mean, in the spirit of his title, you know, questioning representation optimism, I think uh, you know, Kenneth has engineering optimism, and and I, to a degree, I share that optimism. So, I mean, you know, cuz cuz you and I both have spent enormous amounts of time in our Discord server arguing about things like this. So, I'm well intimately, at a gut level, aware of just the absolutely cosmically insane amount of computation that goes into powering the the algorithm of evolution with a capital E. You know, it's happening on the earth in many forms, not just life but mimedics and information and, you know, whatever else, right, higher-order phenomena. So I'm just saying there are probably very clever hacks, at least I hope, maybe Kenneth hopes that there are clever hacks that will get us somewhat towards a better method. You know, Ash had a very good take on on kind of, you know, some of these evolutionary aspects in particular, you know, it it not being a um an optimization but a constraint, and so it's good to see, you know, new new researchers, right, like kind of taking up taking up that mantle. And I think a big part of what the point of this paper is and their point is we need to think more about these processes because I think there are, you know, like you brought up a lot of interesting, you know, sort of takes just now and intuitions on how poet and these algorithms might work, and it's I think the building up is is essential.

So I think what we need to do, like I imagine some future training neural networks will will look like this: like you'll start with a neural network that has 100 parameters and you'll train it on some some subset of the data, some simple subset, and then the next phase you'll train one that has 1,000 parameters, um, and in some way it's kind of expanded from that lower-level network. Like we'll have algorithms that will identify subnetworks in there that do something in particular, let's say these 12 neurons, and what we'll do is like we'll now make those 120 neurons and then train them and and keep them sort of isolated as a group, right? So that they grow that abstraction like sort of further. Um, and then it'll be 10,000, 100,000, like you'll kind of keep scaling it up, right? And then what you'll end up with is almost like a neural network that has a very specific kind of um not sparsity but modularity, like a very specific kind of modularity because it was grown from like seeds, you know, up rather than starting at the massive scale and allowing it to do all this like really obnoxious and and counterproductive entanglement. Yeah. And this is the I mean, I think we're getting to putting the pin in in the center of the dartboard here. So um, he said the the problem is that neural networks have too many degrees of freedom. They're like a pile of sand, right? And we are different from that because, you know, the the very basis of how we think is correlated to how the world works. So there's something about building up, you know, you said building up, not not tearing down. And and he said it's not about what you know, it's about how you got there and how evolvable the knowledge is. So where you can go with that knowledge. And it's not necessarily that stochastic gradient descent is a bad thing in of itself. It's simply that it's statistically intractable to use anything other than stochastic gradient descent. Otherwise, it simply doesn't work. So, he's not necessarily saying, I mean, maybe he is, but he's saying that these evolutionary algorithms are really, really good when you're building up when you have a very, very sparse search space. And and then the other thing which is very important which we didn't get to is this whole building-up thing like the NEAT algorithm, and you can bring that in um, is this idea that we need to have a form of training or learning about the world which monotonically increases information and complexity. So we're we're adding this thing and we're adding this thing, and every time we compose things together we're looking at the evolvability. So we're not just saying can you do the thing now, we're saying is it actually likely to be able to deal with future things that I might encounter. When children are born, they still have two legs basically, so you shouldn't just think of it as a random uh, you know, crossover operation; it's an operation that respects the the provenance, the topology, the structure, right? So there are, you know, in practice there are um crossovers and mutations within certain topological frames, but some things are held constant because they need to be held constant.

But on your comment about degrees of freedom, I think Kenneth was saying that we actually want to reduce degrees of freedom. So he's not kind of saying, oh, we just need degrees of freedom for stochastic gradient descent, but degrees of freedom are okay. I think he was saying that there's actually a magical Goldilocks zone when it comes to degrees of freedom. So we want the degrees of freedom to be at least a representation of how the world works. But necessarily there should actually be more flexibility than how the world works. But if there's too much flexibility, you get a weird kind of mode collapse and you don't see the emergence of evolution. Degrees of freedom. The way I think about it is like it's like memory with an algorithm. You want to use as much as necessary and no more. Um, and it's really hard to it's really hard to uh figure that out. So, yeah, I mean, you just imagine Einstein. Of course, there's an element of serendipity, right? You have to be in the right place at the right time, but there are still just mental degrees of freedom, and simply if there are too many degrees of freedom, then relativity wouldn't be wouldn't be conceived of, right? You you need to have that that that spark. What's funny is so he added one additional parameter, the cosmological constant, right, and then was kicking himself for like lots of his life that that was his greatest blunder, right? So so he considered his greatest blunder adding an additional parameter. Like think about that, like you know how like simplicity is such a driver of of successful models. Now it turns out though, in the spirit of just as many parameters as you need and and no more, now we need the cosmological constant, you know, for very valid scientific reasons, right?

On on your comment about degrees of freedom, separating them from training and how things are enacted in in the real world. I like thinking of them more in the same frame. This is actually related to what we've been talking about when we when we uh discuss agency and autonomy. I mean, Kenneth is using the frame of of creativity, but it it's all the same idea, right? It's when we have agents that are acting in the real world, we need them to make intuitive decisions that make sense. And that is precisely the reason why current AI doesn't do intelligent things when it's autonomous. That is the reason why we have to supervise current AI every single step of the way because it would do stupid things. So I think if you did reduce the degrees of freedom to actions that actually made sense, the probability of it working autonomously would be much higher. Yeah, that's that I think maybe that's a bit unknown for me. Um, I'm not sure because I think there's there's tradeoffs there. Um, if if there if it's reduced again, it's this Goldilocks thing. It's like um if you reduce it too much, then it ends up sort of not being able to do the things that you need it to do. Um, and if there's too many parameters, it's able to do the things you need it to do. Plus, it does a bunch of stuff like you definitely don't want it doing, you know, and so there's just this weird, you know, this weird balance in there. And and yeah, so I think that's a very difficult question and and very hard engineering problem. Um, but this paper is pointing in in directions that we need to go.

And actually one really great thing about this paper which you mentioned earlier is how visual it is. So like if you look for example at the um at the the intermediate layers um for the evolved uh, you know, compositional neural network um versus the SGD trained one. Okay? Like I mean, come on folks, like I I don't care how much of an SGD fanboy you are out there. Like just look at the images and come and tell me with a straight face, okay, that there's any merit whatsoever to the entangled, convolved, contorted, convoluted nonsense that like you end up with like with SGD. Like seriously, like I don't even know if we can have a conversation if you think there's like value in there, but if you because if you look at the evolved ones, they're like a stencil that you would could imagine an artist would have like, well, here's an oval thing and some, you know, gradient lines and and this sort of thing. And the point is, it's not just about appeal to humans, okay? It results in extremely powerful transformation capabilities like these parameter sweeps where there's a parameter that controls the mouth opening and closing or the mouth being wider or winking or the apple's, you know, leaf literally rotating as if it was on a 3D axis. Okay, so these are neural networks that were trained solely on 2D images and they have evolved a concept of like 3D rotation. Like if you're not impressed by that and kind of blown away by how cool that is and if you can't appreciate that that's relevant to the real world and agents operating in the real world with an effective world model, like I don't know. I can't really pull you out of that hole. Yeah. And I wrote some software to visualize that so we can show that on the screen.

But this actually comes back to the famous episode 61 of MLST when we had Randall Bellisero and and Yan LeCun. It was called uh interpolation, extrapolation, something else. I can't remember now. But um, yeah, we we showed this neural network visualizer. You know, it was it was the TensorFlow playground, I think it was called. And it it would show you like um the the evolution of of training, you know, for all of these different types of neural networks with different activation functions and learning rates. And one of the data sets was a spiral manifold. So it show you a spiral and you could kind of see it, and it was it was trying to fit the the thing over time. And one of the reasons why it can only interpolate when inside the training support, which basically means roughly speaking inside the, you know, there's this high high-dimensional space of data and it can roughly, you know, go inside the space, is because the activation functions are are ReLU, right? They're they're piecewise linear functions, which means they just chop up the space. Now what's interesting about the the evolved CPN, the um compositional pattern-producing networks, is that the functions can actually be trigonometric functions. These are continuous functions because most of the time in a neural network, like if you if you give it a test sample which is outside of the training support, it you're in no man's land. It gives you nothing. Whereas with these CPNs, if I understand correctly, they actually do um extrapolate outside of the training support because these trigonometric functions, they just they just keep going, right? The problem isn't just the activation functions, it's the training um methodology. And and what would happen, for example, let's say we just keep ReLUs. I think if you still found a way to do this kind of evolutionary building up from simpler um, you know, mappings, what you would end up doing is there would be there would be kind of an a a simple higher-order layer that say took the spiral and chopped it up into four quadrants, like nice nice quadrants, and then one that did sort of like some 45-degree angle chopping and whatever. So it would still end up being a piecewise linear division of the space. But rather than if you go and do this over at the TensorFlow, you know, site, it's all messed up weird, you know, lines, funky angles and whatever. Instead of that, you would end up with the type of chopping up that you and I might do, right? Like quads and then hex hexagons and like whatever else. So the point is whatever activation functions you have, there's a better way to train them.

Yeah. I mean, there's there's a few points here because I guess like what I was where I was going with this before is you take y = x^2, and the reason why we think of it as robust is for any value of y it kind of does something; it does something reasonable, right? It's not it's not just memorizing little little regions in in the training support, and we often speak about the the limitations of neural networks in a variety of different ways. So there's there's a complexity limitation, there's a computational limitation, there's a training limitation, but it's quite interesting to play with this and pull it apart a little bit. So yeah, as you say, the SGD thing that is actually a serious limitation because there are neural networks that can do a hell of a lot more if they are not trained with SGD. I mean, Schmidhuber gave an example of, you know, the um it's pos it's possible to have an RNN which is Turing complete under certain conditions, but it's not trainable with um with SGD. And there's also the other matter that when you start using these like weird and wonderful activation functions, you can do things outside of the training support. And then there's the other matter that when you start doing non-SGD training, you can actually build hybrid systems as part of the training process. So imagine if you if the way we recognize faces we we do some kind of composition, splitting um iteration, recursion, what you could just train that entire system with another form of training process, so it rather opens up the whole game, right? Yeah, 100%. I mean, like and you can handcraft neural networks, like for example, um, you know, there was that uh was that um a DeepMind paper, right? Did the one where they handcrafted sort of a universal Turing machine in a in an LLM form, but you also have to add to it, you know, expandable memory. So, they had to do extended generalized autoregression rather than just, you know, autoregression. So, that little technical detail aside, like you're saying, you can handcraft things um, you know, you you can construct them, but they're not trainable. Um, and so it's the core thing we need to improve is the training methods. And I think nobody knows what they look like really like at the moment. Like there are some hints, right? Like poet, um, maybe what you're talking about these kind of hybrid systems where things are trained in different ways or the curriculum learning or some type of building up, you know, from seeds that are somehow kept slightly isolated. Um, I don't know, like I we don't know because everybody's just stuck on SGD. SGD, scale, scale, scale is enough. SGD, attention's enough. It's like until the industry kind of fails enough to achieve anything real like with those methods, I don't think they're I don't think sufficient resources are going to go into the alternatives.

Yeah, but they're getting close. They're getting close cuz even the even the doomers are having conversations about like why aren't we all dead yet? Like we thought, you know, once GPT 4.5 came out or 4 whatever, it's like I thought we were doomed then, and like what's what's going wrong? So they're kind of re-evaluating their, you know, apex of rationality over at LessWrong to figure out what went wrong, you know, why why aren't we dead? I know. But even this actually, I think this paper is the perfect in it's the perfect thought pump for people to understand why we are not doomers, right? Because we're not we're not horrible people. If we actually believed, as the doomers do, that this technology is autonomous, that it has agency, that it could just go and do things on its own, we would be doomers as well. I mean, we would be regular guests on Lyron's YouTube channel. But it just doesn't work like that, right? That it's not even close to working like that. Well, let me just let me let me caveat something there. So, excuse me. I'm on the record as having like, you know, a and again it depends on how you define doom, but sort of let's say an all-cause doom of, you know, one-third or whatever it is. And I also am extremely concerned about AI harm. Like AI is causing harm today. Massive harm. It's causing massive harm. It will continue to cause massive harm. People are using AI to do heinous things and cause cause lots of harm. And I absolutely think we need to protect ourselves from that. I'm just not like I'm not a AI-triggered AI is going to become super intelligent and kill us all because we're we don't yet have the right architecture for that. So like we haven't discovered any architecture that, in my opinion, is machine-trainable that will lead to um AGI.

Yeah, I agree with all of that. I mean, the way I think about it is there are two types of AI. There is the agentic, monomaniacal, you know, separately intelligent form of AI that Lyron is talking about. I don't think AI is intelligent because to be intelligent requires creativity.

This is what chalet says. It's about being able to invent new knowledge given a novel situation. AI doesn't do that. But the AI that we have now, it requires humans, right? It it's it's it's a tool that humans use, and and it makes us more creative and smarter, and it has all sorts of ramifications. And I agree that there's a lot of harms to be discussed there, but I do make that clear separation between the two again, and I I make a similar separation; I just use different words. So I talk about the AIs we have now as being narrow intelligences, um, not general intelligences, and and I considered those two things like categorically different, not just on a spectrum, different. Um, so and I know we have some differences on that, and you know, lots of, but the similar things we we discovered this incredible book, *Why Greatness Cannot Be Planned*, and it it's got this butterfly on the front cover. And the reason Kenneth put the butterfly on the front cover is there was this Fogyny, which was created by Pickreeder, which was basically a cross between Tinder and Flickr, where you log in and you you select two images that you like and you breed them together. And behind the scenes, it's using the NEAT algorithm, which is a way of evolving neural network topologies. So essentially, the the topologies of of the two CPN neural networks are mixed together, and you get a new image. And a CPN is is simply a neural network which has a broader array of activation functions; so trigonometric functions like sin and and you know, cosine and whatnot. And um, it also takes an input of an X and a Y and also a couple of other things. And it has an output of a hue, a saturation, and a luminance. So if you want to generate an image with it, you just basically enumerate a bunch of pixel values. And it's resolution independent, which is quite cool. So you can like make very high-resolution images or low-resolution images. And um, and it will just generate you an image of something. It might be a skull or an apple or whatever.

And yeah, so essentially the whole system is you could humans could supervise this breeding process, and when you looked at the Fogyny, even though the humans weren't looking for anything in particular. So you you you got these weird intermediate steps that did not resemble the amazing thing that was discovered. But what's interesting is that amazing things were discovered in surprisingly few steps. It's really fascinating on on so many on so many levels. I think um, you know, I highly encourage people to to check out the NEAT paper and and Pickreeder and a lot of related things. So I think um, and you you pointed this earlier, the crux of the the difficulty is all right, cool. We did Pickreeder. We got a skull and a butterfly um and lots of other things, but I don't need a skull and a butterfly. And I I actually I need something that that generates, you know, I don't know, knights' helmets um or or um or dragonflies, like not butterflies, dragonflies. And and therein is the crux of the problem because we did this open-ended cool thing, and I mean part of the let's say downside of an open-ended search is you don't know where you're going to end up. And I think um, in in the interview, you know, Kenneth referred to this like cone of inevitability, right, where where um as as you progress further and further along the the let's say the the time dimension or the step dimension of evolution um where you end up becomes less and less certain.

And the crux of the problem is how can we do open-ended evolution um in such a way that we end up with the things that we need? Like we need something that detects school buses and something that finds you know pedestrians in the crosswalk and something that can generate um movies about samurai and you know, like there are things we we need, and so there's this this this uh conflict, right, between we do have goals, goals like we do have certain goals. We need to protect pedestrians and and monitor school buses and like whatever else. So, we do have these kind of goals, and yet we want to try and achieve them with open-ended evolution. You know, I know it's it's such a it's such a paradox that, you know, you you you find what you need when you're not looking for it. But even there are so many um forms of deception in open-ended search. This deception word is is very interesting, and it's it's all about what you think you want isn't actually what you want, right? And even you just said now um we want school buses to be safer, and there are many discoveries in the search space that would make school buses obsolete. In fact, YouTube is now it's completely taken over Hollywood; you know, if you're trying to be a traditional filmmaker, you don't make any money anymore because everyone's watching it here on YouTube, but my god, YouTube could be out of business in in a few; is we could easily discover something that what that wipes YouTube away. So, it's about this epistemic gap between what we really want and what we think we want. Yeah. Yeah. That's a good point. It's kind of like um why do we need all the kids to go to a physical location every day of the week? You know, maybe maybe there's some other pure virtual or or more local, you know, system. I you know, you're totally right. I mean, and that's that's the the really the crazy thing about life and the universe. And we need more variety. We need not all our eggs in one basket. We need to explore alternate alternate methods, folks. GPUs were invented for like video games. Um, they're also happen to work really well for a certain kind of machine learning. So, let's just do that, you know.

Yeah. And and that by the way, in the hardware lottery, that is also a manifestation of Kenneth Stanley's *Why Greatness Cannot Be Planned* because nobody thought when they were developing the GPUs that it would have anything whatsoever to do with AI, and YouTube started as a video dating website. It had nothing whatsoever to do with you know um a new type of media. Wonderful. Well, Dr. Dugar, thank you very much. Yeah, it was a pleasure. It was it was so great to be to be with you guys again, and you know, it was four years ago that we interviewed I think about four years ago that we interviewed Kenneth the first time, and it was eye-opening then and and it and you know, it's it's opened a lot of great open-ended exploration for me personally. I think he he won you over in the end, didn't he? Sure. Yeah. I I I'll concede to that. Guys, welcome to MLST. Today we have um Kenneth Stanley, and Kenneth is my hero, as folks of the show will will very much know, and also um Akash Kumar from MIT. Um, Akash, since you haven't been on the show before, can can you um give us a bit of an introduction?

Yeah, definitely. Yeah, my name is Akash. I'm a third-year PhD at MIT. I'm working with like Philip Isola, and uh, I'm interested in researching like emergence, open-endedness, and all the cool stuff around like artificial life and metal RL, these kinds of anything that's really emergent uh process that creates intelligence or creates some complexity um I'm very interested in, and I'm super excited to be here. And you've had the great fortune to be writing a paper with Kenneth Stanley; I think not many people would uh would have that accolade on on their resume. Yeah, I'm super privileged. I'm super happy to be with him, be here with him. As I understand it, you you guys have just written this paper called *Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis*. And and the basic idea is that the types of representations that neural networks learn are chaotic. They're just like spaghetti, right? And what we want to have in artificial intelligence is a way of carving up the world by the joints, right? A way of actually understanding how the world works at an abstract level. And it seems to be the case that us humans, by some kind of osmosis or magic or god knows how we do it, we have a very powerful abstract model of how the world works. And neural networks don't do that.

Yeah, it it does seem like we we do this in a very sophisticated way. Um, but I think I think you know the paper isn't so focused on saying, "Hey, look how good humans are, and then um look how bad LLMs are," for example. It's not really that story. It's more that there's an observation of the underlying representation that you get uh from SGD um and just how terrible it looks um which is which is just really visceral; like you you don't need quantification to see that there's something wrong there. Um, and I think that that observation on its own would make only half a paper, you know, because like a lot of people would think, including maybe I would think, that's just the way it is. Like it it looks the way it looks, but it works, so it's fine. Um, and of course, people quibble about like, you know, what are the differences between human-level cognition and LLMs, and I don't doubt that humans do something beyond what LLMs today can do. But you can't really look at that by itself and just that when you say you observe the underlying representation and just say, "Oh, clearly it's it's it's messed up," or something like that, because we don't know what it should look like. There's no baseline to compare it to. Um, but what's really I think unique here in this paper is that we do have a comparison point from a very unusual set of circumstances that led to a very different-looking representation. So actually, you can compare it to something, and indeed the thing that we compare it to looks like it should look intuitively. It's not necessarily clear that like this is how it looks in the human mind or in the human brain. Um, we don't know. There's no way to go in and check. But it definitely would be nice if it did. Like it looks the way you would think it should look. And so it just really I think it just raises questions. You know, we could still you could still take a position that, "Oh, well, I'll just dismiss that, and I still think it's fine the way it is with all this horrible-looking entanglement." Um, but it definitely raises questions like, "Should it look like that?" I mean, now that we know that there are representations that don't look that way. Um, and so I think what's really valuable about this is not necessarily that it shows that there's a fundamental flaw. It may or may not, but that it raises really deep questions that we can pursue now that we wouldn't otherwise have pursued. So there's lots of questions um that we can now go after to try to understand the nature of the representation and maybe if it is potentially flawed in a way that could be rectified.

So, so um gosh, I mean Kenneth was just saying that um this shortcut rule or Goodhart's law, you know, we refer to it via many different terms. I mean, the reward hacking and things like that, that there's this problem in machine learning that you put an objective into the system, and generally the objective is created by us. It's a proxy because we don't understand reality. So, we create a proxy. Initially, it works very well, and then it actually gets much much worse if you continue to train on it. And that is a really fundamental problem, right, throughout all of machine learning. So I just want to break it down: like you mentioned SGD a minute ago, is why is it SGD that's the problem as opposed to something else? So we in our paper we don't term it SGD; we term it conventional SGD, and we do that on purpose because we're not trying to we're not 100% sure if it's SGD by it's like if we just need to get rid of SGD or what the exact problem is. We're just saying the current paradigm, which encapsulates a fixed architecture, a fixed objective, and SGD as a target as as the thing that's doing the chasing of the target. That entire paradigm, there's something wrong with it because it's susceptible to shortcut learning. And people have known this for a long time; like I think Melanie Mitchell talks about this a lot, which is basically just Goodhart's law for representation learning, right? It's basically just you can if you try to solve a task, you're going to perfectly solve that task, but what you really wanted was a good representation of that task. And as our paper kind of shows, there's many different ways to solve the task. You can solve it with a bunch of heuristics and a bunch of if statements, or you can solve it with the right abstractions. And uh, there's those are the two ways to solve it, right? And what you pick, which one you end up like employing and using, it really matters for what you really want, which is adaptivity, uh, generalization, creativity, out-of-distribution generalization, and continual learning especially. Um, and we can we going to talk about that a lot more later, but yeah, I guess the point is we don't just care about the training loss. That's not like we train it on the training loss, but that's not what we care about. What we care about is the down all the downstream stuff that we're going to use it for later, which is much harder to quantify, much harder to formalize um than uh just a training loss.

Yeah. And let me add one thing. Um, I think one of the really interesting things about the observation in this paper is it pokes a hole, I think, in a very deep assumption that we have um that if the results are good, then what's underneath the hood is also good. It's a kind of assumption that's unstated most of the time. Um, but it's clear here like when you look at the underlying representations in this paper, it's not necessarily the case. Um, and so like you can't just look at benchmark performance, which is most of what we look at when we talk about is this model good, and know that actually things are okay under the hood. Um, and like Aar said, that does have implications because you may think, "Well, who cares? If it does well in the benchmark, then it's good. What what else matters? Like why should I care?" But as Aar says, there's all kinds of downstream implications if your underlying representation is terrible. Um, and I'm sure we'll get into those. Um, but so like just this idea that like you can't be confident just because things look good on the surface um is thought-provoking. I think it it leads to a lot of questions about what's really going on. Uh, one thing I really want to add quickly to Ken's statement is I think at the end of the paper we have a quote from or like something uh like a quote or a paraphrase of Ken Ken's dad, and I think he says something along the lines of um two mathematicians, they can both like ace like a math exam, of one can go on to become like a great mathematician that discovers a lot of things in the field, and the other one can go on to discover nothing. So the test is just like it doesn't give you a picture of what we really care about, which is downstream like um how they influence the field and how their research progress carries out.

Yeah. I mean, this might be a good time Kenneth to talk about the story of your book a little bit. So on the front cover of the book there was this butterfly, and I'd love for you to tell the story of of the butterfly, but you know, in in the book you went to great lengths to talk about this concept of deception, which is simply that when you have a complex objective, again, there's this um epistemological gap, right, that we were saying why we have proxies with Goodhart's law is because we don't actually understand the thing that we're modeling, so we have a proxy, so anything that's ambitious, we we create a proxy, we optimize towards it, and we actually get stuck in this deceptive search space, and we find anything but the thing that we actually want to find. What's the story of that butterfly? Right. So, um, and this is a lot of people know my work for this point that's related to the butterfly um which comes from this system called Pickreeder um which is also relevant to this paper. Um, but just for context, like we're actually going to draw a new lesson in this paper from the usual one that people have heard before from me if you're familiar with my work. Um, like the original lesson had to do with the fact that it seems to be the case that there are some things that you can only find by not looking for them. Um, and so setting an objective and saying like, "I want a butterfly," for example, can actually cause you to fail and not be able to get to the thing that you want. Um, and this was true in this Pickreeder system, which was basically a big crowdsourced experiment in people searching through exploring uh image space generated by little neural networks called CPNs or compositional pattern-producing networks. And this uh this this Pickreeder system allowed people to effectively breed pictures. Um, and it was it would we found inside of the system that the people who would decide they want a certain image and try to evolve that image would fail. Um, and then people who are not looking for anything particular would discover all these amazing things; like the butterfly was kind of the symbol um that we used because we put it on the front of the book of the one of these serendipitous kinds of discoveries. Um, and this led to this idea that, well, you know, there's many things in the world that we're not going to be able to find if we directly search for them because of deception. And that's the underlying reason. And deception means that the stepping stones that lead to these interesting artifacts that you might want to find don't resemble them. Which means that if you have a if you have an algorithm that's trying to follow a gradient by matching closer and closer and closer to the objective, getting a higher and higher score um or lower and lower loss, that would mean that uh you're going to get stuck in a dead end um because of deception um because the things that lead to the thing you want actually don't look like the thing you want. And this is true in the lineage of many of these images in Pickreeder. Um, and so it led to, you know, the novelty search algorithm, um, which was an idea that, well, hey, if that's the way Pickreeder users succeed, maybe there could be an algorithm that's actually not driven by where it's trying to go and instead just tries to do something new and interesting all the time, which then led to the field of quality diversity. You see quality diversity inside of major systems like now like for example, very recently, Alpha Evolve. It has map elites under the hood, which is a quality diversity algorithm. So, and these are systems that try to get out of deception and try to illuminate spaces of many many interesting things. Um, and so it led to a lot of stuff um this insight, and then the butterfly is kind of symbolic of this insight. But there's one further insight that was missing from all of this, which is another deep insight which just shows like this this this crazy Pickreeder experiment had all of these really unexpected uh discoveries underneath the hood. And it was this this this point that what about if you do get to something as an objective, you know, so like what was interesting with Pickreeder was we did experiments later to confirm the hypothesis that you can't get to these things by trying with the same algorithm. The underlying algorithm in Pickreeder was the NEAT algorithm. So like someone found a skull, for example, the skull is used all over this new paper. So someone found a skull, and so we we then said, "Okay, well, we have this hypothesis that like you wouldn't be able to find the skull by actually looking for it." So we tried to automate the um the the NEAT algorithm to just try start from a random position, a random configuration of the network, random weights, and then to just try to get closer and closer and closer to the skull. Um, and this generally would not work. Um, it was too hard for the NEAT algorithm to do that because it is deceptive. The things that lead to the skull don't look like skulls. So image matching is not a good heuristic. However, there was this one caveat to that which was kind of interesting, which was that with relatively simple images, like some of the easy images from Pickreeder, like a crescent or something like that, just something relatively simple, we could evolve to it as a target. Um, but we noticed something weird about it when we would succeed, which was that it was always triple

The complexity of the network. And it's it's important to note that with the neat algorithm, the networks grow in size as they optimize. And so like we could see the difference in size of the networks. And so if there was a success, it would be much worse because it was obviously inefficient compared to the original discovery. This was a just a kind of a um a side observation. We didn't really push on it too much, but it's really interesting that it seemed like the open-ended process, like the process where someone's not looking for something would lead to a more efficient representation.

Further down the line when deep learning, you know, became dominant, you know, we we started to think Joel Layman and I started to think, you know, let's just see what SGD would do um if we just try to train a network to output a single pipreer image. Um, and then we could see that like it's not you don't get a bigger network in effect because like you just you get whatever size network you start with. I mean it doesn't grow, but we we found we couldn't start with the same small network that was found through pickreeder; that there's too small too few degrees of freedom for SGD because it's deceptive in that space. So like you need a lot more degrees of freedom. This is one reason that you can train these huge networks like and and to do amazing things. And so if we started with a big enough network with SGD in this case, we could reproduce pigreeder images. No matter how complex they are, we could do it. But when we look under the hood, the representation is is terrible compared to the underlying representation that we see in the original discoveries from Pickreeder.

This is a really weird principle if you if it actually generalizes, because what it means is that the story doesn't end with the question of whether you solve the problem. It's that if you solve the problem in one way, it's worse than if you solve it in another way. Like we never talk about this in this field. Like if you solve the problem like to be able to output the picture of the skull in a serendipitous way, it's a beautiful underlying representation, which is what the paper shares. But if you solve the problem through brute force, like I'm just going to move deliberately towards that image through uh stochastic gradient descent, it's terrible in comparison.

You may say, well, how do I know how terrible? Why do I say it's great versus terrible? Like what am I why would I know this so well? Well, the the really cool thing about 2D images like pickreeder images is that it makes it easy to just see directly explicitly what is represented at every single intermediate node. You know, because we can query every node in the network, not just the outputs, but every single neuron. We can ask for every XY position, what are you computing? So we can draw a picture for every single node in the network and literally see what the representation is, how it builds it up piece by piece to get that final image. And it was just stark that what you see from SGD is horrendous. So in the sense that, for example, it has no recognition of the symmetry of the images like the butterfly or the skull. It has no idea that they're symmetric. It computes these horribly entangled and fractured representations where pieces of the left side and the right side have no coherent organization and and are have to be re-represented multiple times when they should only be represented once because it's effectively symmetric, and you can just see that explicitly. And you can see in the the version from Preeder this beautifully captures the underlying representation. It's not just that it captures symmetry. It has a an unbelievable modular decomposition. Like it knows what a mouth is. Like there's separate controls of the mouth. The mouth can open and close, which is absolutely incredible, mind-blowing to me because it's like there's a world model of what a mouth is there without being data driven. Like how is that even possible? It's it's like Contra the bitter lesson. Like there's not a lot of data here, but we're getting world models out of this thing.

And so I think um like this this observation that it matters how you got to the solution, how it's represented under the hood just hasn't gotten the light of day until now. Um and it's it's kind of a companion to the old insight from the book, you know, which is that sometimes the only way to find something is by not looking for it. But now there's this caveat, but even if you do by looking for it, you may pay a steep steep price in terms of the underlying representation. Yeah, to to play back a couple of things there. So there was this paper called the lottery ticket hypothesis which was by Jonathan Frankl and and he said that SGD only works when you have many many degrees of freedom. When you start with this kind of dense monstrosity you train it with SGD because it can use those degrees of freedom and then you spify it afterwards. And we should be clear to say that this isn't a limit of neural networks as a representation. I mean there exists a neural network representation which does I mean a CPN is is a neural network, but the question is how do we start sparse and build up rather than starting dense and using SGD and and coming down and then I want to pull apart a couple of things so you're framing this just as an optimization problem right, and I think there are other interpretations for what's going on here because it might not be contra bitlescent; it might be the fact that us humans we've had the benefit of billions of years of evolution. So, we've had all of the data. We have learned all of these abstract motifs about how the world works and through a form of interactive imitation learning. I certainly think this is what happened in Pikreeder. We're unwittingly imbuing the the model with the representations that we know about the world. And that explains why in so many intuitive, you know, so few intuitive steps. If you look at the fogyny for pigreeder, you can get to the butterfly in something like was it five or six steps? So is the optimization process just kind of cleverly like is it telegraphed from the humans or is it actually learning it in some fundamental way?

So, I've had this this discussion or you might even call it debate with a number of people and and I don't I don't want to claim to to know the answer because I think there's a a deep almost philosophical question at the heart of this that that where the answer is actually not known. Um, but I think at least it's important to to to convey that there is another side to to that argument that that that's valid. I think at least reasonable to consider which is that no the human has not supplied enough information for that to be a satisfactory explanation. Um like the the humans generally were talking about from dozens of steps to hundreds in the entire run. I mean, think in terms of deep learning, we think about millions or billions of steps. Like this is absolute peanuts. Um, and so like in the dozens or or hundreds of steps. Um, like it was dozens to get to the skull, for example. There just isn't enough time to show it things like a mouth opening and closing, whatever that would even mean. It would be a strange also, it's a strange trajectory to take. Like the human is not thinking I'm that they're training. It's not the thing in their mind. So like you know you would think like from a training perspective you would show examples of closed mouth open mouth closed mouth open mouth like let's show the space so you can understand that that dimension of freedom. They would not do that. That's not how people are thinking. People are choosing things because they look good. And so like we don't know for sure like maybe you know it could be that like inadvertently somehow miraculously they just sort of like trolled it through the space of all of these dimensions of freedom in just the right way. But I think that that's probably implausible and there needs to be another explanation. I think it's it's actually um that the the open-ended process causes the human to actually move instead through a hierarchy of regularities like starting with things like the discovery of symmetry just because humans like symmetry. They see a symmetric object, they like the symmetric object that locks symmetry into the system. Then from there they continue to search and find symmetric things that now vary on the symmetric convention that's been established and add more to that. And that it's a side effect of that process because it develops this hierarchy of representation which is very virtuous in its structure. Um a side effect of that is you get some kinds of degrees of freedom or dimensions that just happen to align with things in the real world almost like miraculously. Um and we see multiple examples of that in pigreeder underlying representations like the my favorite is this is the apple like the apple which is in the paper in the appendix has a stem which has a single weight one weight in this network of I think in this case hundreds hundreds of of connections one of the weights actually can control the swing of the stem it looks like a 3D projection of a stem and moves back and forth like this there's a shadow that moves underneath it and this is a single dimension that you can sweep to cause this stem to move back and forth. Um, again, I think of it as like a hypothesis about the world or a world model which has absolutely no data behind it. So, it's it's insane. Um, you know, it's like the exact opposite of move 37. It's like play millions of games and then, you know, maybe it's not a surprise that something might happen that humans have never seen before that's actually quite valuable. It's an origin of a creative insight. But this kind of insight that we're seeing here is not from millions of trials or seeing all of the possible ways the world could work. It's from very very limited amount of interaction and just somehow it's a coincidental alignment with the way reality actually works but it's happening over and over and over again you know so so I think it makes me wonder like you know when humans have really deep insights something like theory of relativity something like this you know which one is it more analogous to is it more like move 37 like Einstein did 10 million simulations in his head of what might happen when certain physical interactions take place or is it just that Something about the order of the hierarchy of the things that he integrated into his model of the world over his lifetime just happened to precipitate this in the right place at the right time that actually does align with the world. I mean many times it wouldn't. So he's kind of lucky and all we all are when we have a good idea but nevertheless because of these this virtuous ordering and the way that integration the way that repres representation's integrated it can lead to these amazing serendipitous alignments once in a while. um which just come out of thin air and that that might be the kind of thing that we're observing here which is a different kind of thing. It's not the bitter lesson style of thing. It's a wholly other type of thing which I think we need to dig into more now that we've exposed it to understand better uh what it actually means because I don't think we have a good understanding yet.

And let me let me jump in because I I didn't get so it's it's great to great to see you again uh kind of it's been it's been a very long time and Akar nice nice to meet you um nice to meet you too and the reason I want to jump in here is is first of all this this paper I mean it's it's brilliant it's insightful it's important uh it's visually one of the most beautiful papers I've seen in a in a long time and I think even people just looking at this um will get the insight that you're talking about right now like with Einstein you know because I mean sure we're not Einstein but I think anybody who introspects the way they think the way they think about solving problems the way they think about a skull the way they think about an apple you know they're going to find that the images in this paper completely reflect the way we think about the world the way we model the world right and so by this kind of open-ended search and interaction with humans And like you said, the neural network is is able to grow in such a way that it captures these fundamental world model aspects. I mean, it's really cool. It's really cool. Thank you. Yeah. And it's great to see you, Keith, too. Like this bring back good memories to that that old interview that uh that was really fun. Um so yeah, appreciate your comments. Um yeah and uh obviously I I I'm not going to agree with the part about the the beauty of the paper because I want that's for others to judge put it but obviously I agree with that that like this is something we need we need to think about like what this means. Um that's that's why we put it out there.

Yeah. Uh I wanted to say something about pigreeder more concretely uh really quickly. It's basically this uh just to drive the point home a little bit. These people who were on pigreeder, they were selecting only images via they only saw like 15 mutations and they just clicked once. That means they're selecting that they're injecting like what like a few bits of information into the system and you end up with a butterfly after a few dozens of uh generations. So like just from an information theory point of view, you're not you're not able to put enough information to cons like describe the regularities of the butterfly like symmetry and the skull. And I think the thing that really makes this uh pig breeder work and it goes back to your point they were talking about in general the creative space of like all human outputs but in pig breeder I think what's especially what was at play was like the evolution of evolvability because people were only selecting for what they wanted right what they would look good in ter in the general sense in an open-ended sense what they thought would be uh cool to follow pursue but implicitly they're also there's also implicit selecture for selection pressure for evolvable things. So if there's like two versions of the skull, which is one is like spaghetti and one is like very modular and composable after a few generations of evolution, the one that's more evolvable will be the one that wins out, right? Just like in natural evolution, the evolution evolvability. And this evolvability combined with the serendipity is what I think gives you these nice representations. Cuz if you just try to get a skull which moves its uh mouth up and down, right? like you're just wanting that one type of regularity, you're not going to be able to one, you're not going to be able to find it, and two, even if you do, the representation is going to be like overfit to that thing, right? Um, what you really want is evolvable in any direction of interestingness kind of. And that's where the serendipity really comes in is you have to embrace whatever the system gives you. And if it gives you a regularity of symmetry versus it gives you the moving of the mouth versus if it gives you the eyebrows winking, you don't know which one it's going to give you. You just have to see what it gives you and go from there. And I think the combination of those adaptabil evol evolvability and in general adaptability combined with serendipity is what's giving you these nice representations. And I think that has a lot of um analogies to natural evolution and scientific discovery as well.

Yeah. And building on top of that also uh really quickly is parsimonious. I guess in deep learning people are obsessed with the idea of compression, right? Um they think compression is intelligence, compression is everything. And in some ways it kind of seems like it has to be true, but in other ways I'm not sure that compression captures this idea of adaptability fully cuz um you can compress the hell out of the skull and get a very nice compressed representation. That doesn't mean it's going to be adaptable or evolvable in the way that you want it. Right? In the same way, you can probably compress all the knowledge about the world and your mental model of the world in a very like small zip format or like very compressed thing, but that doesn't mean it's a adaptable in the sense that it's going to be it's going to give you creative leaps and the way to connect the dots. Um, if that makes sense. Yeah, sure. So, we'll be looking forward to your your next paper. compression is not enough, right? That sounds good. That's a good Yeah, copyright Keith Dugger. But some people say um I don't know like poly semanticity or grocking or scale and you know that the it just it just appears like the neural network isn't grocking it, but actually it is. And maybe if you just trained it a little bit differently. Part of what's important is is just to point out that we've we've identified a really interesting phenomenon, but we don't have answers to all these questions yet. And sort of part of what we're doing is saying, let's let's go find out the answers to these things. It's possible that some of them help. It's possible some of them don't help. Um I think one thing just to to think about just to to observe upfront is that um the difference that you see visually is so dramatic and stark. I mean, anybody who looks at the figure of the the two different versions of the skull is that it's hard to believe, although we don't have the evidence yet that there's something that's going to fix this that's just real simple. like a gro like I imagine grocking does something good but it's hard to believe it ends up making something that looks like the the CPPN version of the skull from Pickreeder that's hard to believe but nevertheless it's possible that things like rocking and all kinds of other things that we do like mixture of experts maybe convolution helps in some ways although that's not in the LLMs but it could be in other networks helping like they're all just questions that are not answered um but I think one thing to to think about with respect to things like rocking is just even if you like if you're trying to rationalize and save your faith in how these things work right now and think oh well that's okay groing will fix this just think about this like wouldn't it be nice if you didn't have to do groing like what grocking basically means is that you get this absolutely horrible entangled mess and you clean it up later like once you start figuring things out start deleting all the excess and redundancy all the fracture gets fixed wouldn't it be nice if you don't have to do that and it's just good in the first place like I mean like the The Um what what we see because of the Pickreeder CPN is that that actually is possible. Like without this example, without this weird example from Pickreeder, almost no one would believe or buy that that's even possible. It's like well I mean yeah of course I'd love miracles to happen every day but it's just not going to happen. But the thing is that it did happen and it happens consistently. So it raises this question is like even if groing is helping is there a better way or maybe a complimentary way? because maybe both factor into why like you know human brains get to where they get to. Um so like these are unanswered questions. This is not to say that grocking doesn't help. Um we need to check and find out the degree to which it helps. But one other kind of interesting thing to think about is that if you think about um the efficiency of training, you know how expensive it is. like we're building spending, you know, billions, hundreds of billions of dollars like creating these these giant data centers to do giant kinds of training runs. Um like if if it's true that like there's an incredibly more efficient way of doing things, you know, like so for example, like it could be that like yes, SGD from with a brute force will do basically almost anything at least um in terms of being able to get the training data. Um like obviously there are implications for generalization from what we're observing here but at least like you know if you cover almost all of human knowledge you can do a lot of useful stuff um and so you just brute force it into the system and you get this FE representation which sucks but it still works. Um but the thing that this raises is the question is like but the it's really really hard if you're in a point in search space where all the regularities are broken to

Get to the next point in terms of, like, higher-level intelligence. You might be able to do it, but the the the representation is not doing you any favors in doing it efficiently because all the degrees of freedom that you have are the wrong ones. And so you're constantly overcoming and making up for and overcoming and making up for how much cheaper would this whole thing be if we just did these things in a way that actually worked the way we're seeing in these CPPNs. Like if it was actually like always getting the right dimensions or close to the right dimensions that actually align with the modular decomposition of the way the world actually works, then this could be multiple 10x, 100x, like more efficient in many ways.

So even if, like, you still believe we're going to get there with just SGD, this could at least have implications from an efficiency point of view. I suspect it has more implications than this just that because when we go beyond efficiency to things like creativity, you know what you're seeing is that the dimensions that have been discovered in the skull, for example, align with new skulls, imagining new things in the world. You wouldn't be able to imagine these new things if you didn't have those dimensions already groed, so to speak. And so your ability to imagine something new is highly compromised.

And of course, like the whole thing, like, like you know, the conversation, the public conversation right now is turning to exactly this issue of creativity. Like people are saying, "Well, where is all the new knowledge? Like, isn't that next on the menu?" Like we're we're getting all we're acing all these tests and it's like amazing, math olympiad, like beating the PhD-level people. The people don't get PhDs because we want them to score well on tests. They get PhDs because we expect them to invent something. They're supposed to come up with new knowledge. Where is this going to happen?

And so this suggests that there could be serious implications here because the ability to invent something new requires that as a prerequisite you understand the underlying dimensions of reality. Even if you're going to break those dimensions, like some really innovative people do, intentionally, like break a rule, it's because they know the rules before they break the rules that it's interesting. If they just don't know anything and don't understand how the world works at all, they're not going to be brilliant and have these amazing insights. And so I think I think there's like really big implications here, um, in terms of both efficiency, creativity, um, generalization, and and continual learning, you know, because what I said about like if you want to keep learning, it depends on how you represent the world, how efficient that's going to be. So like I don't think we can just say, "Oh, Grocking might fix it and like everything's going to be fine."

Let me just jump in on two points. So one is, and I I want to know if you think this analogy is fair. This this issue of the problem with fractured representations. I kind of think of it as, you know, if you set out to build a nice UI for an application or something and instead of starting with a stencil that has triangles and squares and ellipses and whatever, you start with a jigsaw puzzle, you know, it's going to be a lot harder, right, to try and build a nice UI out of fractured, weirdly shaped, you know, components. Like that's a fair analogy, right?

Yeah.

Okay. And and I guess I wanted to ask you like I really see, you know, connections between this kind of path dependence that you talk about, like it matters a lot how you got to your state of knowledge, how you got to your representations. I think there's a a direct or deep connection with, you know, poet, like the work, you know, your your earlier paper, right, on this kind of increasingly complex curriculum and environment where you start off training in simple cases and make them more and more complex. Um, I mean, there is a connection there, right, and maybe that's a simple tool that can be utilized.

Uh, for sure. Yeah.

Yeah, I mean that then that's, you know, one of the other factors we talked about that might help to to mitigate the issue is yeah, the the kind of open-ended um search where um yeah, it sort of naturally uh guides the the process through uh a set of tasks of increasing complexity um but but in a way that's divergent. I mean that's like what's happening in pickreeders. So like we could we could extrapolate that that's actually partly or maybe largely responsible for what we see in these representations. Um, but it it it raises, you know, questions about, you know, are there very different training paradigms from just dumping in all the data in the world in a in a kind of a batch um which are more intentionally um focused on the chronology being intuitive or or at least aligned in some way with building good representations. I mean it it it evokes ideas about curricula and things like that. Um, but like there's also a kind of concept of a natural curriculum where like the human being themself uh by their nature tends to learn things in an order that's actually useful for building good representations. Like, for example, like you know, you take little kids learning arithmetic, and if you start trying to teach them calculus, they're just going to ignore it. They're not going to start absorbing it. I mean that's not the same as LLMs. Like they'll take anything you feed them and start to make the connections.

Um, which arguably is really unhealthy, you know, because if you start to learn calculus before you've learned arithmetic, you're going to actually create some kind of heuristic version of arithmetic. You can't avoid it at the same time as you're learning arithmetic somewhere else in your brain. Um, and this is what causes this kind of redundancy or fracture. You get like multiple representations of the same thing. Some of them diminished in their capacity. Um, well, that's shown in the the Mecciner, you know, paper, for example, the the biology paper, right? Uh, about where where you find arithmetic that's done in some bizarre way, like I'm going to add 39 and multiply there, subtract 53 and add to, and you know, it's exactly what you're saying, which is a fractured representation, but I just want to

Yeah.

I mean, one other really important point I just want to add to this, this may also apply to humans. So, I don't want to seem like I'm saying that all humans have unbelievably beautiful uh, you know, unfractured representation. We also, I think, are victims of going through things in in a bad order sometimes.

Absolutely.

Um, and I can give you a personal example. I can give you a personal example of that. So, in high school, when I went to sign up for physics, for physics one, um, they put me in the one that was for people who had not had calculus, and I had had calculus, right? And so, I'm in this class and I'm like, "What the heck are we doing? We're just memorizing all these like long list of equations, you know, for a cannonball in this situation." and they're like, "Oh, actually, we put you in the wrong class. You have calculus. You're supposed to be over here." So, after a week, I switched to the other one. It was so much easier because I knew calculus. So, I could I didn't need to have a formula for this specific cannonball situation. I just could derive it or just calculate it directly. You know, it's radically different different learning mode. And that's that's exactly the

Oh, yeah. I was gonna say I was just going to comment on the biology paper that you mentioned. That's kind of like how it's doing arithmetic. It's like memorizing all these random heuristics. Like if you look at like one of the uh circuits they found, it's like they're trying to add like 23 and 57. It's like, "Oh yeah, it's around 55 plus around 25." So it's like going to be around this, and at the end it all magically works out, but it's just like a bunch of like if statements and patterns get you there. Yeah. So if you if you learn arithmetic like that, how how can you adapt that to like new situations, right? So, and and even with that, I mean, we're not going to get into the computational argument because Keith will go off on one about Turing machines, but you know, there are actually limitations with what you can do. So, so the neural network has has to learn some fractured version of multiplication or whatever it is because it's it's a finite amount computation.

And we were just talking about Grocking. And the way I understand that is as the training process progresses, they start off learning quite simplistic um sort of low-frequency um representations and then you train and train and train to Grocking and eventually you learn very high-frequency representations. And it just so happens that many of those high-frequency representations are more aligned to the natural factorization of the world. But that's basically coincidental. There is no principled way to distinguish good representations from bad representations. I tell them never use GPT to generate anything because it's it's always obvious. But what you can do is you can write something and you can discriminate with a GPT model, right? And so so so it's good at discriminating, not good at generating.

Would it be possible to have our cake and eat it? Could we build some kind of a a sort of like a bottom-up algorithm that does something a little bit similar to Pickreeder and every step of the way it's asking a language model that's been trained on everything in the world, "Does this look good? Does this look good?" Would something like that work?

Yeah, like a pigreeder for AI and intelligence, right?

Yeah. I mean, like creating an open-ended algorithm which solves this issue. I think uh Jeff Cloon on our paper, he calls it like the trillion-dollar question or the trillion-dollar algorithm because that's basically like how you if you think that the UFR, like the unified factor representations are akin to like a human, then that's basically like creating like a human representation, right? Um, so yeah, I mean I guess the answer from my side is that uh some algorithms should exist and we're going to try to find it, but right now we're we're still, you know, just in the preliminary stages of this kind of thing.

Yeah, I mean there it doesn't seem like there's any principle that says you can't do this um algorithmically. I mean presumably it's all algorithmic. Um, so hopefully this is just a nudge to have us start looking into it. Um, and you know, you you can go on your merry way down the path that we're going down and just make things bigger and have more and more data um and ignore this, but then you're at risk of disruption. You know, because if somebody does actually take this seriously and it works, obviously would have profound implications. I mean, you just look at the two pictures of the underlying representations in the skull. If that actually would be translated into the world of giant LLMs, there's going to be implications for that. So, I don't think we can just ignore this question. Um, and and that question of like these magical types of algorithms is on the table now.

Yeah. So, how how how can I get in with you guys on the uh ground floor of this trillion answering this trillion-dollar question? Andre Kapathy posted on Twitter and and he said V3 has just been released and loads of people have been releasing videos of like, you know, hairy gorillas in the tundra doing ASMR videos and all this crazy stuff that there's this ideation and curation. People came up with these incredible ideas and then they they generated images with with V3. Now, Andre Kapath has said, "Well, the obvious next step is that we kind of do gradient search over the creative space." So, what if Zuck built Instagram version two and now the ideation and curation were basically optimized using stochastic gradient descent and the images would still be like in the the mixture of collaborative filtering. So, the users would be kind of like saying, "I like that. I don't like that." My intuition is that would mode collapse and maybe it would work better if it was mixed in with with human creations. What say you?

That yeah, this a very uh complicated question. Um, I think uh you know, you have to disentangle the degree to which the human is part of what you're calling creative and the degree to which the model is. So like with V3, um, you know, the the human came up with the idea of the the ape doing the ASMR or something like that. So so who's getting the credit for being creative? It's not that it's not impressive. I mean, certainly the model did something impressive, but where's the creativity? Um, and so I think though in in the larger picture, you know, if you want to ask like can the model totally autonomously on its own be genuinely creative, whatever that means, you because people are going to disagree about what that means, then you know, is it just a matter of some kind of gradient-following algorithm that's going to uncover some really interesting insightful types of creative scenarios that that would be that would be really valuable for all of us, and yeah, I think I think it depends on the underlying representation. I mean that that's like the answer. It's like what how is the world represented inside the model? Because that's going to be what determines what are the adjacent points where the gradients can actually get you from where you are. Um, and if the model doesn't represent the world in a coherent parsimonious way, then you're going to find your options more limited. Not not so limited that you won't be impressed at all. You'll probably still be impressed. um, but but more limited than like a really creative auteur or something who's coming up with amazing new genres and a new way of thinking about film or something like that's not probably going to fall out of this. Um, and so it just sort of like depends on the degree of of what of of this creativity that you want. Um, and and I do think that like you know, I distinguish between I call like derivative creativity and transformative creativity. You're going to get a lot of the derivative style, but a lot less of the transformative style if you have a bad underlying representation. And so that's the trade-off that we're working with. It's still going to impress a lot of people. But the mode collapse problem, I think, is a real concern here, you know, because as we sort of like freeze pop culture in the year 2025 and just live inside of that bubble for the rest of eternity, like it's going to get more and more tiresome. Um, and so like the ability to have like pathways out of that like again will depend on how we represent the world. And there's a danger with this road that we're going down that that we're going to be trapped. And we're already seeing this. I mean, it's like we're seeing that uh like it looks like like what's on the radio doesn't sound as different from 20 years ago as it did 20 years before that. Um, and this kind of convergence could be just accelerated um by the kind of stuff we're talking about here. Um, and ultimately I think it all boils down to representation. It doesn't work autonomously. It's not creative because it's not built on the foundation of a representation that describes the world.

Well, so as soon as you leave the thing on its own doing anything with any flexibility, it'll just kind of go crazy. But I guess the question is, does it matter? Like we have AI and we use it in tandem and we can do great things with it. Is is that a problem?

I mean, I totally agree that in terms of uh creativity and the amplification of what humans already can do, uh AI is hugely valuable in its present form, like it can amplify things um in really interesting ways um but it's still ultimately the germ of the ideas is coming out of the human mind um and so you know, you could be satisfied with that or not, you could say this is good enough, I'm happy, uh but I think like in terms of what like what we can't do because of that that we might someday in the future be able to do, the things we can't do are the things where the human mind isn't having the ideas, you know, like we're we're worried about things um you know, from a scientific perspective, for example, and like the kinds of things like like the new place that I work now uh Llaya Science is thinking about like automating the wheel of science um where we would like these ideas to come faster uh to solve the problems of the world, but I mean it also applies to art as well, you know, like I mean it's like when are we going to get the next big uh idea in music like that's going to like really shake things up. It's like it's taking a long time here. Um, and so of course we can just wait for humans to come up with these things. Um, and wait long enough and it might happen. Um, but the question is can AI accelerate that process both in the sciences and the humanities. Um, and um you know there's a separate question of is that a good thing? But it's just an interesting question. Can it be done? Um, and you know if it can be done then then that's not what's happening with current models yet. Um, and so like I think like when we talk about representation, it starts pointing us in the direction of how that will ultimately be done is by taking seriously the underlying representations and understanding that to actually see something novel in the world in a genuinely interesting and deep way requires you to decompose the world in a special way that's highly organized um and unique. And uh that's just not what this fractured entangled representation seems likely to do, at least as far as we've understand it right now.

And actually this question came up uh Ken in your doom debate, which is um you know, is survival an objective or a constraint? And I think you know, Ken, I think and by the way I I want to say that was a brilliant interview. I thought your performance was excellent. I really enjoyed watching that debate. I saw it from start to finish. I think um you know your communication of um open-ended search and and kind of these this take on it has has uh been sharpened you know significantly since like four years ago when we talked. I found it like really compelling. So I thought that was a great um a great conversation and in there you say, "Look, you know, survival of the fittest is is not an objective, it's a constraint." It's like it's a constraint on the system. You have to survive if you're going to propagate, but subject to that binary, you know, survive or not survive. It's not part of any objective, you know, I think is the way you kind of communicated that.

Yeah.

Yeah. I think that's an important point here. Like I I don't think that the right mental conception of evolution, biological evolution, is as an optimizer. Um, and this causes endless confusion, you know, because like I mean in the field of AI it causes a lot of confusion um because unfortunately, you know, early genetic algorithms essentially were using selection for explicit optimization. Um, like the the metaphor was broken in my view, like that like basically we had the metaphor for selection, but the metaphor is still wrong because like that's not overall what evolution is doing. He's not selecting just to get to a single target, a single target in the search space that we're trying to get to, like the solution to one problem. Um, but the problem is that in the field of AI like people think of genetic algorithms as effectively like an appropriate metaphor for what evolution is. Um, you know, it's sort of and it's sort of a a bad optimization algorithm. Like people think of it as like the poor man's optimization algorithm and so get like really dismissive of evolution as a useful metaphor. It's like, "Ah, we we played with that that in the 80s, but like it there's nothing really down that path," but I think that's why it's been damaging um that like it actually is a very deep metaphor if you think of it like you said as a constraint, like think of survival as a constraint, is not as an object, like there are many many things that we could have thought of as objectives, like flight, you know, or or uh or like photosynthesis, like these could have been objectives for machines, they would have been single runs where if they succeeded we would have celebrated and said, "This is amazing, amazing achievement. We discovered the ability to fly, the ability to to process sunlight." Like these would have been achievements that were objective, but they're not the objective of the system in the usual sense. Um, and the reason that the system is discovering is not because survive obviously leads to these things. It doesn't follow at all directly. They say, "Well, we have an objective and we're optimizing it." It's

called survive. So, of course, we're going to get photosynthesis. It's obvious. I mean, no one could predict this at the beginning. It makes absolutely no sense at all. It's an orthogonal issue. It's sort of like a side effect. The point is that the things that we care about are actually the side effects of the constraint. It's not directly a consequence of the constraint, but we have to understand it as like the side effect is actually the main event. Like we care about the side effect.

And so what that there's a really important—so when we look at it in that light, like if we change the metaphor, you know, and we think of it as evolution is actually an open-ended divergent process without a final objective, then we we can see that like there are huge consequences for, again, things like representation. I think this is a very important point, like because like it gives us hope for this idea that there might be an algorithm which isn't an evolutionary algorithm, which is more like like what deep neural networks do, but some kind of like learning algorithm in large neural networks that's more analogous to evolution that then would get these unified factored representations, um, because evolution has done that, you know.

So I claim that the underlying representations in DNA are incredible in a similar way to the skull in Picbreeder, um, and the reason is because what you see when you have offspring is that fundamentally you get changes—clearly, like, you know, your your child is not like a clone of you. So you get changes, but the dimensions of variation that are searched almost always preserve the underlying most important regularities. Like in other words, like humans always have bilaterally symmetric children. The problem is that uh selection—um—is captured by a genetic algorithm. It does do the selection part. Um, but what it doesn't capture is this divergent aspect because genetic algorithms are traditionally optimization algorithms which are trying to use selection to converge to a point, which is what optimization algorithms do, but it's a huge digression from what actually natural evolution does because it doesn't converge to a point and it's not trying to. It's just diverging subject to the constraints of survival. So we can think of it as a constraint and then the things it actually finds which would have been objectives if this was a genetic algorithm—like flight or photosynthesis—amazing achievements are just side effects. They're not actually the goal. They just happen to happen as a side effect of having this constraint of survival. Um, it's a very different algorithm from an optimization algorithm in that sense.

And the thing that's that interesting about this is that it then expresses an algorithm which is much more like Picbreeder in the way that the Picbreeder users are actually searching the space of images in Picbreeder. So Picbreeder, in effect, is a much better metaphor for evolution in nature than a genetic algorithm. And that's important because we also notice, I believe, that representation in nature is similarly unified factored the way that we see it in Picbreeder. In other words, amazing. Um, and the evidence in nature is more indirect because I can't just look under the hood and see the underlying representations, but there's evidence of how amazing the representation is from the kinds of variations that you see from one generation to the next. And so, for example, the fact that a human being always has a bilaterally symmetric child. I mean, if you think about it, this is really remarkable. Like it seems like such an obvious thing, like it's not even that interesting, but it's actually really remarkable from a genetic algorithm perspective because preservation of regularities like that is not at all a foregone conclusion. Like in a regular genetic algorithm, you're messing things up on every single mutation. Um, you don't get this really high probability of preserving some deep underlying regularity like that—that would be that would be incredible if you were having such a fortuitous set of offspring that they all preserve some really important aspect—but what's happened in the representation in nature, the way DNA has become organized, the way that the genetic regulatory network works in a as a hierarchy is that it's been—and biologists use the word canalized—or canalization—it's been canalized—it's like it it dug a trench into a mountain side and created a canal so that if there is a change, like a mutation or an earthquake in the analogy, you still get the water to run down through the same canal, and that's the developmental pathway—like with the bilateral symmetry and the other regularities—like you have arms, you have you have legs. It's true, obviously, there's a miscarriage rate, um, but it's incredibly low compared to like the like general destruction of the phenotype that you see in a traditional genetic algorithm.

Um, and there are things you never see—like you never see trilaterally symmetric offspring—and things that could happen like if you were using like an interacting coding or something like an L system uh in a traditional genetic algorithm—like which can express body plans—like you can get—it's very easy to go from bilateral to trilateral or five fingers to 10 fingers like in a single jump—like it never happens in biology. So biology has learned underlying regularities. Um, just like the underlying Picbreeder genome for the skull. Um, there's a strong analogy. And why does that matter? It matters because it shows as a proof of concept that there are forces in nature that are not guided by humans that similarly—because of the divergent aspect of the search, the serendipitous aspect, open-ended aspect of the search—similarly, similarly also do end up with representations that arguably are approaching what we're calling a unified factored representation in the paper. And so it gives hope that there may be algorithmic interventions that are possible independent of human users—like making selections themselves—um—that could could do something similar to representation um because we also see in nature in a in a sense in an automated environment um but not an objectively driven one in the traditional sense.

Um, a lot of people in ML think think that we perfectly understand evolution and that it's just like, you know, a genetic algorithm. Um, I think Ken would agree with the statement that if we really were to understand it then we should have then there should be exist like an algorithm that we can scale up right now that can recreate all the glamour of evolution—right—all the flights, photosynthesis—there's not a single algorithm that can that we can scale up that we can definitively say that will like do what evolution did in the current um state of genetic algorithms in a mouth. Would you agree with that, Ken?

Yeah.

Yeah. I mean, so so if that's true, that means that we don't really fully under like stand and um we understand aspects of it—like Ken's amazing work on like open-ended serendipity, but we don't have a full picture of um evolution—and that's why genetic algorithms are nowhere close to a full picture and we don't really understand everything that evolution is doing.

Agreed.

Yeah.

Yeah. And I Mhm. just for that bell audience out there who claim that we understand evolution fully, and I wanted to kind of jump in, you know, along this line of not understanding evolution completely. I want to put in another dimension which is which is very well known and just almost anything in the natural world, but certainly with life, which is energy constraints. You know, there are these very heavy constraints on resources, on energy, on material, etc., that life has to uh endure in order to survive. And so, again, I'm going to beat on my parsimony/simplicity sort of drum here and just get your thoughts on that, which is that it isn't just open-endedness. It's not just divergence. I think there are other aspects to natural evolution. One of these being the constraint to use less resources, less energy, simplicity of solution. Like do you think that's fair? Like is that another important dimension?

I think it's it's, you know, part of the survival constraint. So I think you could think of it as as part of the function of that compute—survival—whatever survival is—um—like to the extent that efficiency is important for survival then you're going to stay in regions that are relatively efficient. Um, and so that that is having some effect on representation. Um, but you know, similar like in Picbreeder, you know, there's less of this perhaps concern for efficiency—that's at least not as explicit—but it could be implicit in the sense that that you know users perceive evolvability and like it. So like users get a sense that, oh, this thing is really malleable, and so I I'm going to keep playing and exploring in this area, and maybe that's a side effect of being efficient in some sense that it's that it's so evolvable, but it's an it's an indirect correlation. I mean, it's not like exactly explicit that necessarily efficiency equals evolvability. It could be though something implicit and indirect. Um, and so I think I think overall it's it's—like Aar said—uh there's still some unknowns here about uh what actually what actually matters, what doesn't matter. Uh, especially with respect to the representation, you know, cuz like we're we're having kind of two conversations at once. One is like what is the best metaphor for biological evolution, but then it's actually not the key question. The key question is just what leads to these really nice representations. It might not actually have to be a perfect metaphor. So it's a little bit confusing uh to disentangle these things.

Well, that well that's why I was kind of suggesting that because there's other lines of evidence um and and a lot of discussion and just the let's say uh evolution—you you know the literature on evolution itself—on evolution as a general meta-algorithm—not just biological evolution—that um you know in order to achieve the degree of variety and diversity that that you actually get in life there has to be an element of shock and challenge. And sort of it can't just be lots of resources around because then the earth would have just been like—if the earth had just started off as a massive ball of, you know, consumable cheese or something like that—it would have just been covered by gray goo that never had any incentive to actually diversify—crystallize into different solutions and various forms.

Well, you know, I'm not as sure because the like you know that like what you don't want too much pressure. I mean, this is get maybe this might be go going into the weeds, but like you don't want—well, yeah, it's a Goldilocks—a Goldilocks zone. So, so it's like when when you if there's too much constraint then you can't do divergence—like you have to be safe—you have to play it safe, right—so you want to have some opportunity to be opportunistic, but yeah, if if everything survives then you go to the other extreme and then uninteresting things start surviving—like like we've said in some of our publications, you know, about like open-ended evolution that like we think that—we call it the minimal criterion—like the minimal thing you need to do to be able to pass on to the next generation—that it has to be non-trivial. It's like that we think is important—like once it's trivial—like if it's just like if you hit a minimal mass then you get to have an offspring. This is a thought experiment. It's impossible. But like if somehow like God would intervene and give you a child just because you hit some mass. Uh, it might just create a situation where you know that the world yeah would fill up with just like inert blobs. Um, because all they need to do is get bigger. Um, and so uh you you you do want something non-trivial and and you know these these constraints enter enter into the non-triviality of the minimal criterion. Um, so it's a complex—it's a complex question about how to set something up like this. And then now we're like talking about transferring these insights into the way you train a neural network. Um, that like because there's another leap then of of sort of like cognitive complexity for us to think through. It's like what is the analogous thing like in just a training session uh where you're being exposed to data?

I like that Goldilocks analogy a lot um for evolution because I think Joel and Risto had a paper where it's like you need like uh some sort of um catastrophic events to happen in order to get adaptable solutions, extinction events, like near extinction events. Um, so that's like one extreme, but on the other way, like Ken said, you can't have global competition just everywhere, or else you just end up doing like local hill climbing, which is what a generic algorithm does. You need some sort of local um competition and not global competition.

I have to thank you. I have to thank you actually Kenneth because uh you know when we talked four years ago up until that point—I mean ironically I was I was trained a lot in biology and biomedical engineering and but up until that point I really hadn't thought that much about evolution and you know and since since the conversation with you I've been thinking about it more and more and more and reading more and more and just becoming increasingly fascinated at how powerful and amazing this this algorithm of of evolution is, you

Yeah, I appreciate that. Um, that that's great to hear.

Yeah, evolution is is just wildly powerful and amazing. I mean, yeah, it's like to think it's a GA is so underrating it and so missing what it is. Um, it's a real shame like in our field that in AI especially, you get people making this analogy a lot. I see it on social media. Um, and it's like GAs have done a disservice uh to our field's understanding of how profound this process is. I mean, it created all of living nature. Um, and it begs an explanation—like it's biblical—literally.

Yeah.

Yeah. Yeah. And I mean, and also I think there's a vast underappreciation of of the fact that evolution occurs in many scales and many systems—like it occurs in psychology, it occurs in in memes, it occurs in the information space—like it it's a very general almost meta-algorithm, right? And and I would encourage people to look into whatever domain they're in whether there's any kind of evolutionary perspective on it because it can be very fascinating. Like a lot of human behaviors that people dismiss as irrational are actually not irrational if you consider it from the perspective of an algorithm operating on an ecosystem of species or individuals or or things like that. So it's just fascinating.

Yeah. And the the Picbreeder is like that, you know. I mean, Picbreeder has a lot of evolutionary properties, and here it is creating a neural representation that has these fascinating properties internally and that can't be dismissed. I mean, the evolutionary properties of what the human beings are doing in Picbreeder uh is obviously part of the explanation. But it's not that we want to have to depend on humans guiding the search process. It's whatever they're doing that's somehow analogous to what evolution did which allows it then to discover these incredible internal representations. We want to abstract that. So we want to find like yeah the correct level of abstraction to allow this to happen in an autonomous fashion so that the training regimens of these large models leads to a similarly amazing underlying representation.

And I think that the the question that looms is like how far is the representation currently from this ideal because it's it's not totally known. Like we show in the paper a number of pieces of evidence um indirect evidence that there is fracture inside of fracture and entanglement inside of today's large models. We we we go through literature. We also show new examples. Um, but you you know the word—reason I use the word indirect is because unlike these Picbreeder images, we can't just go in and look at a neuron and know what it does explicitly because that's what's so nice about Picbreeder images because they're 2D. You can just look and see exactly what's represented. It's more implicit in like a real giant multi-, you know, multi-billion connection network. And so the easiest thing to do is just look at indirect evidence of of fracture. But there's this overriding lesson here that even when you're getting answers right, you could be messed up under the hood, which then, you know, further confounds our ability to analyze what these models really know. Um, because like even if they got everything right, they could be just like the the network that that outputs the skull, which is basically almost perfect or it is perfect. And so it it could be perfect and every every question we ask is perfect and it could still be an absolute mess underneath the hood, which means that this indirect evidence you know could diminish over time like as the models get so-called better, but it might not mean that they're fixing the underlying representation to be less fractured and entangled.

I think it's super important that you you name this concept impostor intelligence in the paper. So I think it's important that we that we point that out—like the idea that if—sure—if you have enough fractured entangled representations you can perfectly reproduce an output you know that looks identical to the one that you get from uh unified—oh shoot—what are they called again—unified um factored representations—right—can look identical on the surface—the output can can look identical—and that's impostor intelligence.

Correct.

Yeah. Yeah. It just seemed like a like a

Another good metaphor is to think of it as an impostor. It's like this, you know, the the representation of the skull is just somehow a farce. Um, like it's not what you expect. Like if you just look at the output, it's great. It looks exactly like a skull, but underneath the hood, it's it's not capturing any of the underlying uh components or the regularities. And so, in some sense, it's not really a skull. It's an impostor underneath the hood. um it's very misleading. And so if you think of the skull again as a metaphor for all of human knowledge because that's what an LLM is trying to capture. It's not just a single image. It's like an image of all of human knowledge. For any input, it should output something that's convincingly human. Then like it it could be just similarly an impostor. It could be the same underneath the hood. Everything could be organized wrong, not the way you expect. It's like a giant charade.

And again, this is very confusing and counterintuitive like for for for people because like you know people are naturally—including even me—I would react like but should I really care like like when you say it's an impostor but it's getting everything right and it's human level—like what are you objecting to? Um, but the but the point is that it can still be an impostor because it's like what we care about here is not just that it's going to get answers right—like get good test scores—like seem to be plausibly human when you talk about things that are in distribution. We want it to be able to go outside um like to do things that are creative um to be able to continue to learn—like to do to to get to the next level—including learn on its own and get to the next level. Um, I mean, these are like the next frontiers for the field. If it's an impostor underneath the hood, then these kinds of things are are are going to hit a wall or become insanely expensive. It could be that you can always push through that wall, but the expenses just go up and up like crazy, exponential, worse. I don't know what it means, but it could be something terrible. We might already be seeing that. Like, you know, the amount of money that we're spending here raises questions like, is it necessary? like does it have to cost this much in energy and in money? Um and so being an impostor is not a good thing even though the results look good. I mean that's what it means to be an impostor. Um and we would like to know if that's the case. Well, we don't have direct hard evidence yet. What we have is some evidence from these Picbreeder images that suggest there's a concern here, but we don't know how bad it is. And so it's it's something that's, you know, good behooves the field to actually go off and figure this out.

Speaking on the evidence uh part of it, this goes into mechanistic research. People have been trying to dissect like how these models think and operate uh for the longest time now, right? And it's like notoriously hard because they look into these like models and you have like stuff like polysemanticity where it's like one neuron is like activating via like adding circuits and also like black holes or something random and it's like it's like it's so—that's why it's so notorious. I think this is one of the fractured entangled representations are one of the reasons why backprop is so hard is because how are you supposed to even interpret um something uh that messy—right—it's like it's just a hard problem—and this might—I mean I guess it begs the question is neuroscience isn't easy either—right—if you just try to look inside a human's brain—it's not like—it's like

Smooth sailing when you look at a neur human's brain, either. But um, I guess the question is, is there like a better organization for a neural network which is also interpretable? Maybe it's not—I think we say in the paper it's like an aspirational ideal. Maybe it's not possible, but it's a good like direction to go towards, right? At least more going more unified and factored.

One thing that is lingering in my mind a little bit, and we talk about this a lot in many different ways on MLST, is in the real physical world, um, we seem to have this emergent phenomenon of intelligence and even evolution, actually. I mean, you know, like we could all agree as materialists that you have these physical rules, and you have these particle interactions, and you have these dynamics that emerge, and that's evolution, and it's beyond our cognitive horizon. We simply can't conceive of it. So, what we do is we create these abstractions, and we think about it with algorithms, and these are idealizations or abstractions, and it's just—it's too—it's unimaginably complicated.

But there's always this question of: does it even make sense to think that we could, in a disconnected electronic way, recreate something which had many of the properties of the thing in the real world? Even with Pig Breeder, for example, I think we've been couching it as an algorithm, but actually, it is also a constraint. The magic comes from humans, and humans are still embedded in the physical world to a high enough fidelity that, you know, we see the phenomena that we're interested in. But even with Pig Breeder, does it really have the evolvability that you're talking about? Like if humans stopped using it, wouldn't it just mode collapse? So there's always this question of: don't we just need the world, right, to actually give us the type of intelligence that we want?

I mean, I think that it could be that a process that yields representations as good as presumably what humans have at their best would require some interaction with the real world. Um, but I don't know—I don't know if that's really, uh, like that—um, prohibitive. Like, I mean, that we are getting to the point where effectively computers are directly interacting with the world, like—like I think, you know, it's interesting like in the field of AI that if you go back 10 years or so, like most of the interactions of AIs with dynamic training environments would be in simulations—like it's not the real world directly; the environment is an abstraction of the real world, too. But we're now like getting to a point where these models are actually directly interacting with real-world data.

Um, and so I think it's conceivable that they, like Pig Breeder users, could simply explore the world itself. I mean, of course, they're not—I mean, unless they're robots, they're not literally out there exploring. Um, but the internet is a proxy for the world. Um, and so they could explore the internet; they could gain knowledge in some more natural way. Um, which is more like a Pig Breeder user. Um, and you know, the important part of it is that allows them to discover an order or a chronology on their own, the way we do to a large extent. You know, I mean, we develop curricula for students—obviously, that's what school is—but it doesn't account for the first three or four years of life. I mean, which are obviously very formatively important. Um, and then there's lots of life outside school, too. Um, you know, and so like if you take a person who like invented mathematics independently of going to school, presumably that person is going to be a better mathematician than somebody who just read the textbooks and memorized it and gets the same exact score on the test. So it's like what led you to knowing this is going to affect what you're going to do in the future. Uh, these are things that involve real-world interaction.

Um, and so I think it's conceivable that now, with internet access, you could imagine a world where there's more control on the model side in terms of how it goes off and experiences the world. It's not—it's not easy. It's not like—it's not as easy as, oh, well, we'll just dump in all the world's data like and just like train—like obviously that's—that's what we do, and it's because it's—it's easier to think of. It's harder to conceive of what I'm saying. But I don't see it as impossible, and it could end up a lot cheaper, you know, because like—I mean, one of the—one of the offshoots of that could be that it's multiple orders of magnitude more efficient um to happen in a natural way. And so while we might think it sounds like a tough haul, it might actually be easier in some sense because it is so much more efficient.

To that point, then, do you think that the structure of organization of the internet or even language is a great example? It's incredibly diverse, and different cultures have different words for different concepts and so on. You know, we spoke about this before, Kenneth, but one school of thought is that there are kind of natural categories and given, you know, if you if you could press play on evolution thousands of times over, that you would see this kind—just as we have morphological convergence in evolution, there would be a kind of informationational convergence. Do you subscribe to that?

Um, you mean like if if more than one person explored through the world that they would converge to similar categorizations? Is kind of the idea. Well, I suppose like if if we had a thousand parallel universes and we just watched the evolution of language and the internet and all of those different universes, would we see a kind of convergence towards natural categories which are just representative of the world? Or, you know, like the other school of thought is it's very constructive and it's remarkably disconnected from the physical world. But I feel that even though there are more degrees of freedom in the virtual world, um, it's very difficult to imagine how this works because of the complexity of emergence and whatnot. But there seems to be some kind of ground. I think that the closer you are to the origin of an open-ended process, the easier it is to predict what might happen. Um, you know, so so it's like from the very beginning of evolution, whatever that means, like the first cell or something like that, uh, you might be able to say like if we reran this film again, then you know, there'll be something like photosynthesis is likely—like there's some early things that we might get—algae, something like that—but like the farther you go, I think the less likely it is you're going to get the same thing again—so like almost certainly not humans again. I mean, you might get something as intelligent, but not—not what we are; it'd be something different in some different part of the space. And so like sometimes, you know, it's like we've actually—in my old research group, we had a term—a term for this. It's not a known term; it's just like it was an internal term, but we called it the cone of inevitability. It's like the more closer you are to the origin, the more inevitable things are in an open-ended process. And as you go outward, the cone expands, and there's more possibilities, and then it becomes less and less inevitable as you go up. Um, and so this would be true starting at any point in an open enterprise. So, it's like the invention of the internet. What do you think will happen like in the next couple years? Probably it's somewhat predictable and similar to what did happen, but then you say, well, what would happen in 10 years, 20 years, 100 years? I think it becomes less and less predictable over time, and it will diverge in different ways, um, and that that's a general principle I think for open-ended processes. Really quickly, we know that like eyes evolve multiple times in animal species, right? So there are some stuff where it's like just so fundamental to like how the world works where it's like it kind of is inevitable how it comes about—like eyes—but uh, I guess I would still argue that there's so much other stuff that's just—I guess it seems unpredictable. I guess um, I can't say for sure, but I mean, good luck predicting like peacocks would exist, right? Because that seems like it's just like—like their beautiful feathers are just like a one-off in evolution.

Um, that's like… Yeah, there's also, you know, one thing we said in our—in our—our book is we talked about the fact that you can think of searching for novelty as a form of information accumulation. Um, and so, you know, so this applies to any kind of divergent search, like something that continually finds new stuff. Um, and so like to find something new, you have to exploit some property of the universe that hasn't yet been exploited. Like that—that seems to be a principle. Um, and so you know, it's like eventually to humans keep doing new stuff, but eventually we have to get off the planet to actually do something new. Like we exploit what's here, so we have to go somewhere new. Um, and so like as you—as you wait a long period of time in something like evolution, you're getting a kind of encyclopedia of everything that's possible given the physical structure of the universe. So like you have eyes because there's photons, you know, and you have ears because there's sound waves. And and so it's like—it's almost like this is like our—our structure is a documentation of all the degrees of freedom that exist in the universe as it's constructed. And and it's—it's—it's almost like an encyclopedia in accumulating information. And so with that philosophical viewpoint, you might say that, you know, there is some kind of predestination to it like that, like—well, I mean, we're going to—we're going to see over time as long as it's a divergent process, some exposure of everything that's possible, like some kind of—but it's not necessarily the same configuration. Like there's probably many configurations that can express these different degrees of freedom um in functional ways. Um, but to some extent, those are the fundamental things that exist and can exist, and so we're going to expose them if you wait long enough. Um, so there—there's some degree of predestination there, and even if you do—this reminds me of library learning a lot—but um, another thing I want to say is: I mean, everyone's been—everyone in physics always talks about emergence, right? You have like these simple laws—like underneath the kaleidoscope, it's like just super simple programs and stuff—that doesn't mean that the compositions and like the higher-level emergent phenomena is going to be predictable, right? Even if the simple rules are there. So in physics, I mean, the standard model of physics, it's like relatively simple, right? I don't understand it, but it should be like theoretically super simple. But the fact that biology emerges from that and that chemistry, biology, sociology all emerges from that is, I think, um, it's unpredictable. So the fact that it's like not reducible in like an intuitive sense means that um, just because there is a like underlying simplicity to it doesn't mean that it's always like exploited—exploitable.

You know, maybe if—if—if we think about the current state of machine learning—you know, attempts to develop AI—right, I would like each of you to both, if you would, give a brief—brief recommendations for how you'd like to see—you know, the research paths change—like what would you change about current research paths if you wanted to push us more rapidly towards—AGI—assuming we want to get to AGI—you know, let's—let's—let's assume maybe that's—that's a goal. So like Kenneth, what—starting with you—like what would be your recommendation to the field?

Um, well, we—I think, you know, consistent with this paper, we need to understand underlying representations better. Um, and uh, figure out the degree to which um they actually are impostor representations that don't really capture the things that we expect them to. Um, and then we need to see what the mitigations are. Um, and uh, and there's a lot of possible mitigations, and we don't really know what their implications are. So we need to do a lot of exploring. And then even beyond the paper, and that I think, um, you know, it's—it's really important to look at the question of creativity and open-endedness um and to take it really seriously. Um, like what I find really interesting in the field is the current focus—or at least up to very recently—has been on chains of reasoning—reinforcement learning—improving the ability to reason so that you can solve specific problems. And I just want to point out that doing that is useful. There's no doubt—it's useful—um, and impressive—but it's antithetical to creativity. I mean, once I say that what you need to be good at is if I define where I want you to go and then—then you can get there, then I'm basically training you not to be able to be smart if you don't know where you're going. Like, but that's what creativity is. It's about being able to get somewhere and be intelligent even though you don't know where your destination is.

Um, and people just struggle with this notion. I mean, I think that's why it's not on the radar very much is like people think often like if I—if I say that—let me just—people will just say, uh, you know, that's—if you don't know where you're going, then you can't be intelligent—like you have to be random, then what—what kind of intelligence is that—but I totally disagree. I mean, that's what the most interesting people are doing is that they are saying, actually, based on my life experience, if I take this path, I have a gut instinct, and I can justify that gut instinct. It's not completely implicit—like some of it I can explicitly explain—but what I have a gut instinct about is that it will open up new possibilities. It's not that I know what they are yet, but it opens up a new playground, and I understand this. And by training things to just solve directly a problem, we're training it not to be good at that, which is really funny, um, because that's like the most interesting aspect of human intelligence. Um, and it's really—really underrated that that's like what we're good at. You know, we're good at saying something is interesting. Um, you know, to the point where people think it's almost preposterous that you could even think in a situation like—like I mean, if you put a kid in a playground, that's what they're doing. They don't have a goal for the day. It's like—and we believe that this is not part of human intelligence. And so how are we going to train it to have a chain of thought when it doesn't know where it's going that's still intelligent? Um, a completely different thing. Well, and that's—that's—that's the great irony of—of—of your work and like the title of your book, right? Greatness cannot be planned. So, I mean, I think that's the—that's the core—the core irony. So, Aar, like what would you—what would you recommend to the field? What should we all be doing differently to uh, progress more?

I think uh, one of the high-level things we should be doing is not putting all our eggs in one basket, right? That's like the main point of the open end of this lesson is obviously there should be people scaling up these LLMs to see how far the current paradigm can get us because I'm also curious like to see how far it takes us. Right. So you're applying your principles to the field itself. Yeah. I mean like you can't—I asked Risto this whenever I first like got into AI—Risto Mikulan—and when I got into AI research and I was like, what's your path to AGI? Is it evolution or is it like the current deep learning? And he was like, I don't know, and I don't—I don't want to give you a recommendation because the point is that none of us—we're supposed to all follow what we think is the coolest thing and not just like—just decide like, oh, we need to put 100% of our resource into this one thing. So in that light, I think that there needs to be—I think there—more people should look into, you know, artificial life—Pig Breeder—and the ideas from our paper—and—see because I think it's a very promising direction. And one thing that's like I'm really curious about—even if like the current paradigm is like very cool—you really like it—it's very different than how humans operate and how evolution operates; it's very different—no one will deny that, right?—no one—we don't just train on IID batches of data for like a million steps, right? So no one will deny that. And I think for academia, we really should try to understand and—the evolution and artificial life style of things a lot more. So I think a lot more investments should go there.

Great. Awesome. Well, um, gentlemen, it's been an absolute honor having you on the show. Thank you so much for joining us today. An awesome paper. Thank you. Here. Thanks to both.