Mindscape 280 | François Chollet on Deep Learning and the Meaning of Intelligence

Hello everyone, and welcome to the Mindscape podcast. I'm your host, Sean Carroll. You know that artificial intelligence is in the news. We've talked about AI in various different ways here on the podcast, especially over the last couple of years, where ChatGPT and other large language models have really become an enormous study of interest to many people—for financial reasons, for intellectual reasons. They're becoming everywhere. Right? Google has put them on the first page of search results. Lots of people are using large language models to write texts; uh, you can write programs using large language models; you can write the syllabus for your college course, etc. It's clear that this technology is going to have an enormous impact on how humans behave and live going forward. But there are subtleties.

One of the things that I've talked about is the idea that large language models are amazing because they're able to mimic human speech and behavior. Right? They are able to sound enormously human without actually thinking in the same way that human beings do. Large language models, in some sense, memorize lots of things; they know a lot of facts about the world, and they're super good at interpolating between things that they know. Uh, that includes interpolating different kinds of things that have never been interpolated before, so they can seem creative. They can do things—things that have never been done—based on the training data of things that have been done before. They're less good at going outside of the range of that training data, and one can argue that the processes by which they come up with their outputs are very different than what a human being does in actually thinking, reasoning about the problem presented to it.

And many people, especially people who are experts in AI, understand this attitude perfectly well. It's certainly not new with me; it's—it's well-known to many people—but it is denied by other people who are much more impressed with the progress in large language models and think that we're close to AGI—artificial general intelligence. So I thought it would be fun to talk to someone who is in the front lines of developing deep learning models in AI more generally. So today's guest is François Chollet. He's a relatively young guy, but he—uh, just to give you a sense of his accomplishments—he's a deep learning researcher at Google.

Um, one thing he's done is to develop a software package called Keras—K-E-R-A-S—which is a software library that can be used to interface uh, with deep learning techniques. So you could download it onto your computer and play with Keras and develop your own large language model or modify someone else's large language model if you want to. It's become incredibly popular—3 million something users at last count—so it's had an impact on the field. Uh, François is also the author of a book called *Deep Learning with Python*, and I think there's also a version using R—capital R, the computer language—so you could read that and learn about deep learning yourself. And finally, François has thought deeply about what it means to say that something is intelligent, and in particular, he strongly denies that modern large language models are intelligent in the conventional sense. He says that what they've done is they've memorized a bunch of things effectively, and, like we said, can interpolate between them, and that it gives them a wonderful ability to score well on many current measures of intelligence that we human beings use on each other. Large language models are good at passing tests—right, the bar exam for law school or whatever—large language models are really good at that.

François makes the case that this is not because they're intelligent; it's just because they've learned a lot of things. And to make that clear, he wrote an influential paper called "On the Measure of Intelligence," where he makes the case—he will explain it better than I could—but he makes the case that the whole point of intelligence is to go beyond what you've learned—right, to not merely master a skill—which large language models can do; they can, you know, learn whatever the particular—particular subject matter is and spit it back at you—but to sort of extract—abstract, I should say—from the data that you learn skills that you're not being explicitly taught. So, uh, as François says, he has a three-year-old kid who's very good at generalizing from just a few examples to, you know, build things with Legos that he's never seen before in a way that modern LLMs are not able to do.

So this proposal from François has gone on to become a new competition—the what is it called?—the ARC—ARC stands for Abstraction and Reasoning Corpus—ARC. And the idea is that rather than using questions from typical IQ tests or standardized exams or whatever, they have developed a set of novel logic puzzles. Okay? If you believe that intelligence has something to do with solving logic puzzles, at least here is a set of logic puzzles that are not already out there in the training data many LLMs have already had access to. And guess what? A human being can easily do very well on this ARC test that has been developed—80% success rate, etc.—large language models don't do so well—some of them as low as 0%—but, you know, typically 20, 30%, something like that—evidence for the fact that whatever they're doing, it's not quite intelligence yet, which is not to say we can't get there. So the point of the ARC competition is to incentivize people to go beyond large language models to develop AI systems that truly are intelligent. So it's not just a sort of skeptical attitude; it is an attempt to push us in a better direction.

So we don't know when and if AI is going to become generally intelligent; we know it's not there now, but maybe it'll get there soon. It depends on how clever we human beings are at developing such things. If you visit the show notes page for this episode of the podcast at preposterousuniverse.com/slmp-podcast, we'll give you links to all these things: the paper, the books, the competition, and so forth. Okay, occasional reminder that you can support the Mindscape podcast on Patreon. Go to patreon.com/shmcCarroll and kick in a buck or two for every episode of Mindscape. In return, you get ad-free versions of the podcast, as well as the ability to ask me questions once a month. Very, very worthwhile rewards for such a minor contribution. And with that, let's [Music] Go.

François, welcome to the Mindscape podcast.

Thanks for having me.

So I've talked to people doing AI before on the podcast, and I have this picture in my mind that I just want you to tell me whether I'm on the right track or not. Not that back in the day there were these arguments about symbolic approaches to AI versus connectionist approaches. In the symbolic approaches, you would try to define variables that directly correlated to the world in some way and then hope that the AI would figure out how they all fit together, whereas in the connectionist approaches, you just put a bunch of little processors in there, hooked up in the right way, and—and hope it learns things. And in the early days, the symbolic approach ruled but didn't get very far, and these days we've had amazing progress with deep learning and large language models that are basically in the connectionist tradition. Is that rough picture approximately correct on a very long time scale?

Yeah, that's approximately correct. So, um, symbolic methods—so the big dichotomy here is actually between um, having programmers hard-code a model—symbolic program—uh, of the task that they want to do versus having a system that can actually learn from data how to perform the task. And uh, symbolic approaches, of course, are much more tractable if you don't have a lot of compute, because if—if you only have a very small computer but you have a good brain, you can just figure out the right way to describe a task, and then the computer can perform a task—like playing chess, for instance. Um, however, if you want to make learning work, that's where you need some amount of scale. And as computers got better, then uh, machine learning started getting really popular. And machine learning did not actually start getting popular with uh, so-called connectionist uh, approaches uh, initially. So the one of the first like big—um, after—after neural networks—one of the first big breakthroughs of machine learning were SVMs—that's a—that's a learning algorithm that can do classification, can do regression. After that, uh, Random Forest got very popular in the 2000s—2010, early 2010s—uh, then gradient boosted trees got also very popular. And by the way, so Random Forest and gradient boosted trees are not uh, neural network based; they're not even curve fitting based—um—and after that, you had uh, the great rebirth of neural networks with the rise of deep learning. So starting around um, 2011, 2012, uh, some people started training uh, deep neural nets—specifically deep convnets—so convolutional neural networks, which is a kind of neural net that does very well with images; it—it—it's basically a kind of neural net that knows uh, how to split an image into small patches and uh, look at each patch separately, then merge the information that's seen and—and progresses like this in—in a sort of like modular hierarchical fashion—not—not too—differently from what the visual cortex is doing, by the way—um—and uh, through these uh, new GPU-based components, starting winning machine learning competitions. Um, so AlexNet in 2011 won a couple minor academic competitions uh, with this technique. Then 2012, we had the big breakthrough with the ImageNet large-scale image classification challenge uh, was solved with GPU-trained convnet—um—and then in the following years we had this uh, gradual but very, very fast and sort of like unstoppable rise of deep learning. Like every year there were more people doing deep learning, and deep learning could do more and more things. And one thing that has uh, increased quite dramatically is the scale uh, of these neural nets. Um, so around 2016, 2017, we had the arrival of a new kind of architecture that got very popular, which was the Transformer architecture uh, for sequence processing. Before that, sequence processing was done with recurrent neural networks—specifically the LSTM architecture usually—which dates back from the uh, early 90s, in fact. It's—it's usually, you know, often the case that uh, neural net research uh, is—it's—it's very much grounded in stuff that's from the 80s and 90s—um—but—which you make sound like ancient history, but I was alive then, so not that ancient. You know, I feel like uh, most people doing deep learning today have actually very little knowledge of anything that came before like 2015, to be—to be honest—uh—and—and everyone is pretty much using the Transformer architecture at this time, which was developed in uh, late 2016 and got public in 2017—uh—and it works really well—uh—and it works for sequence data, but pretty much anything can be treated as sequence data, so it actually works for images, it can work for videos, can work for pretty much whatever you want—uh—and uh—finally we had the rise of generative, so even larger scale Transformers trained on uh, as much data as we can—we can grab into them—so trained on the entire internet, in fact. They're not just trained on the entire internet; they're trained on the entire internet plus um, a lot of manually annotated data that's collected specifically for these models. Like currently there are like thousands uh, of people uh, who are employed full-time to create training data for these models—um—and they—they're not very well—usually—I think I think I read that in something you wrote, and it kind of did take me back a little bit. So maybe can you elaborate on this? We'll get back to the architectures and so forth, but so there are people—what are they writing—texts for large language models to be trained on, or are they interacting with the models to correct their mistakes?

So typically the process—it's—it's more—it's more the—the—the second one—they're interacting with the model to correct their mistakes. So they're not necessarily interacting with the model, but basically they are receiving a stream of queries that the model does not seem to be very good at, and the right answers for these queries—right, or the correct—or existing—uh, generated answer—um—and uh, so this is called uh, data annotation or sometimes data rating; uh, it can also, by the way, take the form of actual ratings, meaning that you get a—a choice between multiple generated answers and you pick the best one—um—and every company out there that's training these foundation models is employing typically several thousands uh, of people just doing this full-time. And this is, by the way, this is very much what makes these models useful—is the fact that not only they're trying to predict the next word across pretty much all the text you can find on the internet, but they're also—so trained to sort of like prefer the—the right answer across—yeah—millions of different manually annotated queries.

So as we're recording this in June 2024, uh, many listeners will be familiar with a set of problems that Google was having—putting—having put forward their AI assistant onto search, and sometimes it would give very bad answers. And I—I guess the hope was that—like you say—individual human beings could go in there and just stamp out the bad answers one by one, but that hope seems to uh, be a little bit gloomier than originally intended.

That's right, and it's one of the big challenges and big limitations of LLMs is that you have to apply these pointwise fixes, which are very labor-intensive, right? And they only address one query at a time. It is virtually impossible to fix a general category of issues at once. And the reason why is because these models—they're basically big curves—like they're big differentiable parametric curves that are fit uh, to—to a data distribution. And so you cannot really input into them uh, symbolic programs, for instance, that would uh, be valid for a very large category of problems. Um, you can only input into them data points, and they will fit these individual data points, and they will be able to interpolate across them, so giving them some amount of generalization power—uh, but—not—not—not that much. And so if you want uh, an LLM to perform well, the only option you have is that you need to densely sample the space of queries in which it's going to have to operate. And this is kind of the problem that we saw with the—the weird uh, Google AI answers is that they tended to be unusual queries. And of course, you know, these models—they don't actually understand the queries you're giving them—um—they are just mapping the—the query onto the curve. So you can sort of like picture the curve as—um—a surface—it's a manifold, right? So it's—it's like—you can picture it—I guess in 3D—you can imagine a 2D uh, surface inside a 3D space—uh—and it's exactly what it is—like an applanation, right? Yeah, it's exactly what it is—uh, except uh, in a space that has thousands of dimensions—um—and—um—and basically, you know, in—in that space, different dimensions encode different axes of meaning—um—and—um—they can—they can sort of like interpolate across data points, but they cannot really model, for instance, a situation described in a query, especially not in quantitative terms, and which is why they are not reliable. And my—my advice in general when people start using foundation models is that they're very good at giving you answers that are directionally accurate—that are a step in the right direction—but they're extremely bad at giving you exactly correct answers. So you should pretty much never—uh—ask a foundation model to—especially—especially if—if it's a quantitative problem, by the way—to give you an exact answer and then just blindly use that answer, right? Uh, it's typically better to use it as a sort of like stepping stone to get you something that's in the right direction, and then you refine it yourself, or perhaps you could also automate that and add a sort of symbolic search system uh, to automatically refine the answer. Because if you have a symbolic search system and you have some way of telling whether your answer is correct or not, then you can just, you know, search across a range of answers uh, and verify them, right? So use the LLM to provide you uh, with sort of like an initial—smaller search space—uh—and then use a symbolic system to find the exactly correct answer within that space. But do never—basically blindly trust anything that's written by one of these models.

I have learned that myself. I'm sure that you have also. But so to put it back in the original terms—I'm—I'm getting the impression that rather than thinking of things as symbolic versus connectionist, maybe it's more helpful to think of models where the programmer tries to build in a structure versus models where the model learns a structure just from an enormous amount of data.

That's right. That's right. And um, one thing that's interesting here is that in the first case—uh—there's no intelligence involved. The only intelligence in the picture is the intelligence of the programmer that understands the task, understands the problem, models it in their head, yeah, and then writes down exact instructions—description of the task—a description of the task that is so precise that there is no uncertainty left. And when you actually run the program, it will never have to deal with any kind of novelty—anything that it does not know how to handle—um—because the programmer did a good job; they anticipated everything, right? Every edge case, everything—um—and what—the—the program—you get—people are—people are going to call it AI, but there's actually no intelligence. It's just a crystallized static program. The intelligence here is the mind of the programmer that—that—that developed that program, right? Intelligence is this ability to look at a novel problem—something you've not seen before—and come up with the solution—write the program—right? And um, when you look at learning systems—clearly are capable of learning—capable of learning how to solve problems on their own—or almost so—clearly they must have some intelligence—but uh, the most popular methods for doing this today are just curve fitting. And curve fitting—I mean, clearly it's a form of learning—a curve trained with gradient descent—as nonzero intelligence, right? Uh, because it turns data into solutions at some rate uh, according to some sort of—like conversion ratio—which is not a very good conversion ratio, by the way—is extremely data inefficient—um—it has very, very low intelligence for this reason. A system that is very intelligent would not be limited to these sort of like pointwise uh, mappings like LLMs are—uh—instead, if you wanted to fix um, an issue in an actual intelligent system, you would just explain it—why the answer they gave was wrong—and then they would automatically uh, apply the—applied fix to uh, the entire underlying category of issues, right? Instead, you have to apply these pointwise fixes, right? And the reason why is really because curve fitting is extremely data inefficient, right? It's a very, very low intelligence type of—and from those descriptions—well, I'm sure we'll get to this more later in the podcast—but you can see why it would be very hard for either approach to give rise to true creativity, right? One where the programmer puts in all the structures—kind of limited in that way—curve fitting is kind of limited once you want to wander outside where the data already is.

Yeah. Uh, if you—if you adopt uh, a symbolic approach, um, you're entirely limited by the—the sort of search space that the—the programmer hardcoded into the system. You're—you're limited by what the programmer can anticipate and imagine. And if you employ curve fitting, then you are limited to basically the—the—the convex hull uh, of the latent space representations of your input data points. So basically you're—you're limited to interpolations—you—between data points and your training data, and you cannot really create anything new—uh—anything that you did not expect—if you had seen everything in the training data. And by the way, this is kind of like the reason why foundation models often give you the impression that they're being creative—it's because you haven't seen everything they've been trained on. It's impossible—they've been trained on so much data—so they can surprise you. But if you had seen everything, they would—they would not surprise you. And uh, so that doesn't mean that creativity is something that cannot be achieved by an algorithm; I think it can be—um—but you—you—you have to employ the right set of methods. I think if you look at the history of computer science—um—when we saw real invention—real creativity—initiated by an algorithm—it's been in cases where you had a very uh, open-ended uh, search process uh, operating over a relatively unconstrained search space, right? Uh, because if the search space is fairly unconstrained, then no human can anticipate everything it contains. And the

Search process might find really interesting and useful, uh, and novel points in that space. So, for instance, genetic algorithms, um, if implemented the right way, have the potential, uh, of of demonstrating true creativity and of inventing new things in a way that LLMs cannot. LLMs cannot invent anything because they're limited to interpolations. A genetic algorithm with the right search space and the right, uh, fitness function can actually invent entirely new systems that no human, uh, uh, could anticipate.

Maybe you should explain to the listeners what a genetic algorithm is. Absolutely. So a genetic algorithm, it's basically, uh, a discrete, uh, search process. So it's inspired by biological evolution, right? Uh, you know, in biological evolution individuals have a genome and they pass on half of their genome to, uh, offsprings and to to the offspring and um, and this is basically, uh, um, this is driven by, uh, a natural selection, right? Uh, uh, in order to have offspring, well, you need to survive, you need to reproduce, and so on and so you end up with, uh, individuals that are increasingly good, increasingly fit at surviving and and reproducing, right? And that's, uh, so this sort of like criterion of, uh, of survival reproduction would be called the fitness function. Um, and um, you can try to implement a a computer version of this, right? Where you have, uh, points that are described in some in some way that's going to be the genome, um, and you're going to, uh, apply you're going to, uh, code up some sort of fitness function, a way to evaluate how good a certain genome is, uh, and uh, and you're going to generate a bunch of genomes, you're going to apply a fitness function, select the the best ones, top like 10% or something, and then you're going to, uh, modify them, and that could be random mutations, uh, that could be crossover—you take parts of one genome and cross it over with another—because you're you're not you're not limited by like sexual production, you can actually do whatever you want, you can you can do a crossover between many individuals, for instance. Um, uh, but you you you have basically some sort of discrete mechanism for generating new, uh, combinations or compositions or mutations of existing individuals, and now you have the next generation and you apply the fitness function again, the selection function again, and you repeat. Um, and assuming that your search space, which is basically the space of possible individuals that can be represented using using your your genome, assuming that it's fairly unconstrained, you may end up with some really interesting, uh, findings. Um, the the OG, uh, uh, genetic algorithms, uh, guys, uh, they they came up, for instance, with a very novel design for an antenna, uh, using this technique. Okay, and this this is the kind of design you you could never have obtained, uh, with within an LLM trained on every antenna design out there because it's actually novel. In order to get novelty, you need search. LLMs cannot perform search; they can only perform interpolation.

Good. I did want to, you know, at the risk of, of, uh, scaring some listeners off, I did want to spend just a few minutes digging into how the LLMs work. The LLMs are the things that have gotten so much experience, so much attention these days, and maybe this is the wrong place to begin, but I'm trying to wrap my head around assigning thinking of words as vectors, assigning values to words and saying that they're near to each other or far to each other in a vector space and taking dot products. Can you explain a little bit about how that works?

Sure, sure. So the the big idea behind LLMs and behind, uh, deep learning in general is that, um, the relationship between things can be described in terms of a distance between things, like a literal distance. So you're going to take things, and things could be, you know, uh, pixels or image image patches, or they could be words or tokens. So a token is is is you can think of it as like a word; it could be a subword, yeah, as well. Token basically equals word. And the idea is that you're going to map your things, so your tokens, for instance, into some vector space. So vector space is basically just a geometric space, uh, points of uh, coordinates, and points are things like points are tokens, yeah, right. And, uh, you're going to try to organize these points so that, um, the distance between points represents how, uh, semantically similar they are. All right. Um, and by the way, this is very, uh, this is very similar to Hebbian learning, right? In Hebbian learning, neurons that fire together wire together in the real brain, in in real brains, exactly. And, um, how tightly wired two neurons are could be interpreted as a distance between them, right? So, uh, you you could you could say that it's it's more it's more of a topological distance than a than an actual geometric distance in this case, uh, but the idea is that, uh, if neurons encode concepts, then concepts that tend to cooccur together are going to end up closer in the network, so closer in terms of some some distance function. And it's exactly the same, uh, with Transformers, actually. Um, so the way Transformers work is basically, so you map, uh, these to these tokens to points in the vector space, um, and then, uh, you're going to, uh, compute pairwise distances, and there are cosine distances, basically dot products, uh, between, uh, between words and, uh, between tokens, and, uh, you're going to use that, uh, to, um, figure out a new new coordinates for your points, so incrementally updated, uh, coordinates for your points, and, uh, you're going to do that, uh, by taking into account, uh, the the pairwise dot products, uh, between, uh, tokens, uh, in in in a certain window of text. And, uh, that what what you're effectively doing is that when, uh, tokens already have fairly high dot product, uh, between each other, they're going to, uh, be pulled closer together, uh, yeah. So, um, the the new token representations for the next layer, they're basically obtained by, uh, combining, by interpolating effectively, uh, between existing tokens. So one token's representation is going to become, uh, an interpolation between, um, the the representations of surrounding, uh, tokens, and that's, uh, basically weighted by how related to each other they they already are, are close to each other, they already are in this way. So this basically implements a kind of Hebbian learning, so there is some connection with the way the brain learns. I, uh, but what you what you end up with once you've done this across many layers, uh, in a very high-dimensional space and across a lot of data, what you end up with is a high-dimensional manifold, which is basically just a surface, uh, you know, as I said, you can think of it as as a kind of flag, a 2D napkin in a 3D space. And, um, it's that's exactly what it is because, you know, it must be smooth and it must be continuous because it needs to be differentiable, right? Uh, it needs to be differentiable because the whole process is trained via gradient descent. Gradient descent is basically the only really scalable way, uh, efficient and scalable way that we have to fit curves like this these days, um, and, uh, on the on the on on this manifold, uh, your tokens, so your information, is organized in a very semantically rich fashion, and things, uh, uh, that are semantically similar are going to be embedded very close together, uh, and, uh, different axes, different dimensions along the manifold are going to encode interesting transformations, uh, of the data, transformations that are semantically meaningful, and so on. And, uh, what you end up observing is that the way your tokens are organized on this manifold, uh, ends up encoding a bunch of useful semantic programs, uh, so basically, uh, patterns of data transformation that occurred, uh, frequently in the training data and that the model, uh, found useful to encode, uh, in order to, uh, better better compress, uh, the the semantic relationships between your tokens. And, uh, this compression is necessary because you need to cram, uh, all of these relationships on this manifold, which has it has very high dimensionality; we can cram lots of things into it, but it's still, uh, not infinite, right? You still you still have pretty practical constraints, so you actually need to compress things, and because you need to compress things, you're going to find, uh, these useful, uh, reusable programs, uh, that help compress the data, express it in a in a more concise fashion. And that's really, I think, the most, uh, effective way of thinking about LLMs is that they are big stores of programs, millions of programs, and they're not—when I say program, they're not like Python programs or C++ programs, uh, which are symbolic programs—instead, they are more like vector functions, right? Um, and that means that, uh, you can actually interpolate between different programs. So a vector function is basically just, uh, it's a mapping between a subset of vector space and another subset, right? Um, and it it can cause a useful, interesting transformation, like, for instance, transforming the style of a paragraph from one style to like poetry, right? And it's not obvious that there is there exists a a vector space in which you can embed words in such a way that you could define a vector function that does something like this. It seems extremely hard to imagine, and in fact, before LLMs actually showed that it was possible, I don't think many people would have believed it, but, uh, it works, and that's really the magic of deep learning is that, uh, you express relationships, uh, between things as, uh, a distance function in vector space, uh, and you do it at scale, and magic starts happening. It turns out that you can fit curves, uh, to basically anything if you have large enough space and enough data. I mean, I'll confess I would have been very surprised if you had told me, uh, 20 years ago—anyone very surprised, I don't think anyone expected this to happen—but so for for example, an example that you've used and I've seen elsewhere, thinking of these tokens as elements of a vector space, you can have equations like King minus man plus woman equals Queen.

Yeah, so that's that's an example from Word2Vec. Word2Vec, uh, is only distantly related to LLMs, uh, but I think Word2Vec is sort of like a miniature world, yeah, of the sort of phenomena that you see in LLMs, um, and, uh, uh, in in particular, I think it's Word2Vec is good to illustrate what is a semantically meaningful vector function, uh, so in this case, you know, uh, you have words represented as points in the vector space, and you can actually, uh, add a certain vector to any point to get a new point, which is a new word, of course, because a point equals word, yeah, uh, and, uh, and adding this vector will consistently transform your words in one way, like, for instance, making words plural or making going from a male word to female word, that sort of thing. And you can see how once you once that starts to work, it's almost as if some understanding is creeping into the model, or at least the appearance of understanding.

That's right. So yeah, I guess it kind of depends how you want to define it, but what's what's going on is that, uh, having to organize, uh, tokens, uh, in in a constrained space like this kind of forces you, uh, to arrange them, uh, in such a way that, uh, different dimensions in your space start representing, uh, transformations that that can that that that enable compression of your space. You know what I mean? And you see that scale with LLMs, and because LLMs are extremely nonlinear, uh, the vector transformations that you're going to be looking at are are much more complex, much more powerful, and just adding vectors that can be completely arbitrary, actually completely nonlinear. Um, and LLMs, like you know, they're they're they're functions of millions, uh, of, uh, of very useful vector programs like this that enable, uh, a more concise, uh, representation of this token space. Um, and when you're prompting an LLM with some query, what it's basically doing—what a human would do—is try to understand the words and sort of like picture them in their mind, basically create a sort of like model for what's being said, then you can maybe run some simulation on this model, and so on. So basically, you have this understanding of what is being described and what is being asked, and what the LLM actually does is that it will fetch, um, from from its collection of programs, it fetches either a program it has memorized or maybe an interpolation across different programs it has memorized. And by the way, LLMs are actually pretty bad at compositionality; they're bad at composing different programs. Interpolating between programs, it's—you you can actually chain like many programs like this with LLMs—you're pretty much limited to, uh, patterns that have been exactly memorized by the model in the in in its training data. Um, so it's fetching like a program, and it's reapplying the program, uh, to the input, uh, you're giving, you're giving to the model. And, uh, when it works, it works. So for anything that the model is familiar with, something that has seen thousands of times in its training data, it works, right? So it's great, it's—and and and because it has seen so much data, um, there are millions of possible queries where it will give you, uh, exactly what you want, right? So it can be tremendously useful. Um, but anything that is more unfamiliar, uh, it will not be able to make sense of it; it will it will fetch a program, apply it, it's going to give you the wrong result, and for the LLM, there is absolutely no way of telling, uh, because it's doing the exact same thing in any case, you know, right? Uh, there's no difference for the LLM between generating something that's correct versus generating something that's that's completely off. And so unfamiliarity is one way to trip trip up LLMs, uh, uh, it LLMs really can only give you the right answer for something they've seen before, which is why data annotation, manual data annotation, is so important, right? Um, but it's not the only failure case of LLMs. Um, the you you find also sort of like the opposite failure case where when you have a model that is too familiar with a certain pattern, it will be unable to deviate from it. And, um, a common example is, for instance, you know, the the the the sort of logic puzzle, uh, what's heavier, like one one kilogram of of steel or one kilogram of feathers, for instance. And this this is the logic puzzle that's, uh, uh, occurs tens of thousands of times on the internet, and, uh, for this reason, uh, with the early LLMs, like, for instance, the original GPT-3, uh, if you ask it what's heavier, like one kilogram of steel or two kilograms of feathers, it'd be, oh, they weigh the same; I I know the answer, they weigh the same. So it's not actually trying to read and understand the query; it's just fetching the pattern, right, and reapplying it. And so this has been fixed since, of course, but the way they fixed it, again, it's like it's this pointwise patches; they just explicitly teach the LLM about this new, uh, pattern for solving this particular kind of query, right? Uh, and and if you teach the LLM the right way, then it will start paying attention to the numbers you're providing, right? So that's one example; there there are many other other such examples. And even today, like you take any of these LLMs like Gemini or GPT-4, whatever, you can find, come up with logic puzzles like this where if you provide a small variation, the LLM will break down. Basically, anything that has not been patched by hand will still fail today. Um, and, uh, uh, in in general, this is also the reason why LLMs are, uh, they're very sensitive to the way you phrase things; they're very brittle in that way. And, uh, this is kind of what what gave the the rise to the concept of prompt engineering in general. So prompt engineering is this idea that if you just ask your query the right way—like there's a right way and there's a wrong way—should just ask the right way, you get the right result. Um, and you know, one another way to interpret it is anytime you find a query where you're getting the right answer, it is most likely possible to modify the query a little bit in a way that would be, uh, totally transparent to a human—like it would make total sense to a human—but it will cause the LLM to start failing, right? And this is true for any query; you can always rephrase it in a way that doesn't actually change the query but will make the LLM fail. Specifically, the way the way you find these variations, you just try to make the query slightly more, uh, unfamiliar or unexpected, uh, compared to to what's on the web.

So let me see if I understand, because you mentioned before the idea of the convex hull. So you and I know what that means, but for the listeners out there, should envision a set of points, and we're saying that not only—I I think what's being said is that not only can the LLMs or deep learning models interpolate along the set of points but also sort of the interior that is defined by that set. So if I ask it for a Shakespearean sonnet that explains spontaneous symmetry breaking in particle physics, maybe no one has ever written such a thing before, but it knows a lot about Shakespearean sonnets, it knows a lot about particle physics, the vocabulary words, so it can sort of interpolate its way into giving you a good example.

Yeah, that's right. So, for instance, you could you could ask an LLM to talk like a pirate, but you you could also ask it to talk like Shakespeare, uh, but because you can because these, uh, transformation vector programs, uh, are vector programs, you can actually merge them, you can average them, you can interpolate between them, and that means you can start talking like a Shakespearean pirate, for instance, right? That works, which is something that you cannot do with explicit logic programs, by the way.

Good. Okay, so then the I guess the question is, does the way that the LLM succeeds at sounding so reasonable and smart, um, happen through implicitly making an accurate symbolic model of the world, or is it just a set of correspondences between the frequencies of words, or are those secretly the same thing?

So it's more it's, uh, significantly more complex. The correct answer is basically somewhere in between. In an LLM, you will not find a a symbolic model of the world, but you will you will find a model of word space, a model of semantic space, and that model has some overlap, uh, with the word model that you may have, for instance, but they're different in nature. And the model that, uh, LLMs are working with, uh, is is just not nearly as generalizable as the the one you have. In general, any sort of symbolic model that enables simulation is going to be able to generalize much further away from what it has seen before because it does not just know about, uh, specific situations; it knows about the rules that generated this situation. So you can imagine completely novel situations. The LLM, meanwhile, it's more of a case that it knows about specific situations and can also sort of like average, interpolate across situations, right? Right, but it cannot really move, uh, outside of these interpolations and imagine something that would be impossible if you knew about the rules, uh, generating the situations. And, um, of course, it, you know, the the best way the best way to to really get get develop an intuition about what LLMs do is to extensively play with them and, uh, in in an advisable fashion—like try to make them fail, try to start developing a feel for what, uh, makes them fail. Um, and many people actually never never try that; they just stick as much as possible to things that work, and whenever they find something doesn't work, they blame themselves; they're like, oh, I used the wrong prompt. And, uh, as a result, they tend to have this bias that they're like, hey, LLMs understand everything I'm saying, uh, but of course, of course, this is not this is not quite true. It's, you know, it's very difficult to develop correct intuitions about LLMs because they are so counterintuitive, um, due to their sheer scale—like they have they have seen, they have memorized more text, uh, than than you you read in your entire life by by like four orders of magnitude, you know.

Yeah. Uh, it's it's kind of it's kind of hard to to to imagine that.

Yeah. Okay, so are they intelligent? Not really, but they have nonzero intelligence. The the way we define intelligence is that intelligence, um, you know, most people define intelligence in terms of skills; they're like, uh, if it can do XYZ, it is intelligent, right? And I'm like, yeah, not quite—like this is skill, being skilled at many things is is is is useful, obviously, it's valuable, so LLMs are valuable in that sense, um, but, uh, when you talk about general intelligence, what

Makes it general. Um, is not the fact that you have many XYZ, right? That that that it scales to many tasks. The fact that it should be able to scale to an arbitrary task—like you can come up with a new task and teach it, uh, to your model. If you cannot do that, then the model is not intelligent. So, so intelligence, according to me, is the ability to pick up new skills, to adapt to new situations, to things you've not seen before. So, for instance, uh, going back to this idea of symbolic AI, symbolic AI cannot adapt; it's a it's a static program that does one thing. It cannot adapt to any novelty; it cannot learn anything; it has zero intelligence, like a chess engine has zero intelligence, right? Um, and and um, if you do curve fitting—well, if you just fit your curve and then you you have your static curve and you do static inference with it—uh, you also cannot adapt to any sort of novelty. You can only uh be skillful uh when you are within your data distribution, your train data distribution, uh, because the curve is static, and this is how deep learning works today. You fit a curve, then it's frozen, and you do inference with it, and such a system again has no intelligence.

And you know, lots of people talk about, oh, like LLMs can do in-context learning, but that's actually a total misconception. LLMs do no learning. What they can do is that uh, given a new problem that is slightly novel but still very similar to something they've seen before, they can fetch the correct program or interpolate across different programs that they've learned and uh and solve this uh new uh slightly new task. But that's not learning; that's actually fetching. It's not fetching of an answer; it's fetching of a rule set. So it's it's sort of like one one level higher, uh, which is why it can can of seem like learning; it's not actual learning. So that said, you can actually do active inference within an LM. You can actually make an LM learn genuinely learn new things, and you do do so by actually uh adjusting the curve, uh, to of—and well, when you when you do that, uh, the the main issue you're run into is curve fitting is very data inefficient. Even fine-tuning, doing something like, is very inefficient compared to what humans can do. Humans can actually uh pick up a new task from like a couple demonstration examples. Like I have a I have a 3-year-old at home, and uh it's always fascinating just how quickly it can pick up like very very new skills, yeah, like climbing climbing climbing a wall, for instance, uh, or or just you know, building a car out of Legos. He's seen like five different Lego cars in his life, but he can just imagine his own his own Lego cars and build them from the pieces available. There's no AI system today that can do anything close to this, right? And it's not like he can he can do it because he's seen tens of thousands of Lego cars and tens of thousands of other Lego constructions, and he has access to to unlimited Lego pieces. No, it's it's like it's it's in a handful, right? It's like he's assembled a total of probably fewer than 1,000 Lego bricks in his entire life, but no, he can actually create new things, really complex new things. Um, so LLMs can definitely not do that.

So they have nonzero intelligence because they can actually adapt to some amount of novelty; they can generalize uh beyond the exact train data points they've seen, which is what makes them useful, uh, but they can only generalize close to what they've already seen. If you go a little bit too far away, they they break down, uh, and they can learn; they can actually do active inference, but in a way that's extremely data inefficient. So they have nonzero intelligence, but it's extremely low; it's it's not it's definitely not comparable to like the intelligence of of a three-year-old. My three-year-old is like vastly more intelligent than any LLM out there; it just doesn't compare. And like I feel sometimes that um I feel a deep disconnect with some folks in the AI community that claim that, hey, LLMs today, they're like high schooler level. This is absurd. Have they even even met a human being before? Have they have they ever interacted with an LLM before? Like these are completely absurd claims. Um, anyway, but they're good at certain kinds of test taking, which is what makes people think, well, that's how we measure intelligence. That's right, and this is one of the the cognitive fallacies around the LLMs is that uh the school system loves to test humans uh uh on memorization problems, right? Like school is mostly about memorization. You typically don't even learn rules; you learn facts; you learn points, you know, like point-to-point relating, right? And LLMs are vastly superhuman at memorization; they are memorization machines. They have very very low intelligence, very very low generalization power, but extremely high memorization. And when it comes to showing skill at something at something familiar, um, then you know, you can always trade off uh intelligence for memorization.

Like let's say, for instance, you're you're you're giving your students a physics exam, and uh the concepts are pretty challenging; uh many students probably haven't haven't fully understood them. But what some students could do is just cram a lot of past exams, right? And they may not really understand everything, but they will uh for each problem they will memorize the pattern, and if you just give them the same problem with different numbers, they just fetch the pattern, reiterate. This is exactly what LLMs do, right? And um these students, they can end up scoring very high despite having no understanding of the underlying concepts. Um, and this is true; this is true for human beings where they have a limited memory and a limited amount of time to study, so they can only memorize, you like, 10 exams or something. But what if you have an LLM that can actually memorize 10,000 exams? You know, uh uh it can end up it can end up showing a very strong the the appearance of skill, the appearance of understanding, uh with no actual understanding of concepts. And how do you tell that this is not true understanding? Because after all, you can do your exam, and your exam is what you you're using to to to judge your students. So how do you tell? Well, uh the way to tell is that instead of just giving your students or the LLM uh a problem that's derivative, that's just similar to something that you've given before, you come up with something novel, so something that that's never been asked before. And in order to approach this, you actually need need uh to think from first principles; you actually need to understand the the underlying physics concepts, right? And if you give that to your students that don't understand the material but have studied a lot, uh they will fail; they will score zero, right? An LLM will score zero as well. But then the the the smart the the the smart student uh from the back of the class uh that understood everything but just doesn't care that doesn't care to actually memorize anything, they they will do extremely well, you know, because they're smart. But as a professor, this sounds like hell if I need to come up with novel problems every single time. If if you are looking to test uh understanding and intelligence, then yes, you do. If on the other hand if you're fine with just uh um memorization, then you don't, right? And uh the school system as a whole is fine with memorization; it and sometimes it's because memorization is the goal, but a lot of the time is it's out of laziness; it's using memorization as a proxy for understanding, but memorization is not a good proxy for understanding because you can always uh memorize your way into a high score with no understanding. No argument for me there; it's it's absolutely true, yeah. And and by the way, so on on the just just uh to to to continue this topic a little bit, um on the on this idea that um you if you want to test uh actual intelligence, you need problems that are novel problems where uh the test-taking system or human being cannot have memorized the solution, right?

Um, and um I actually uh released a benchmark of machine intelligence a few years back in in 2019 that's all about this idea. So it's called ARC—ar-AI in in the long form—so it's the Abstraction and Reasoning Corpus for Artificial General Intelligence. And um the idea is that uh um well, deep learning does really well by just memorizing uh data points, but there's very low generalization power. How can you tell that something actually has intelligence? Well, uh you come up with uh puzzles that are all unique, all original, never seen before, not similar to anything you would find on the internet, so not really similar to existing IQ test puzzles, for instance. And uh uh so ARC is basically a collection of such puzzles, and there are public ones, but there are also private ones, which are not more difficult than the public ones, uh but they're hidden. Um, and uh and this is extremely important, of course, because if they were public, then you could just train a model on them, right? And then it would mean nothing anymore. And as that uh deep learning methods and LLMs in particular have scored very poorly on ARC. So we ran uh a competition on the website Kaggle in 2020 on ARC, and this was back when GPT-3 was available; uh GPT-3 got released around around the same time as we released the competition. And so people tried GPT-3, and uh it scored zero, right? And uh the methods that actually worked were discrete program search methods, so not curve fitting. Curve fitting just doesn't work very well for the type of puzzles uh in general. Curve fitting works very poorly to handle any kind of novelty. Um, and so later we also ran uh two years of a new a new edition of the competition; it's called the Athal, um and um you know, it remains extremely challenging; it's like it kind of looks like an IQ test, and it's very easy for humans to do, um but it's extremely difficult for AI to do, and it's very very difficult for LLMs. And we're actually about to launch a reboot of the competition on a larger scale; uh so we are relaunching on Kaggle again, so we're back on Kaggle after four years, uh and we're going to have over $1 million in prizes, and uh the goal is to solve ARC uh to to to pretty much human level, so something like 85%, um and because we know LLMs just don't don't do very well on ARC, the goal here is really to incentivize people to come up with new ideas, uh to look at these tasks, uh recognize just how easy it is for them to solve them and how difficult it is for uh ChatGPT, for instance, to to solve them, and and try to nudge people into into asking themselves, what's going on here? Like why can I do this and and the machine cannot? And try to come up with new ideas, like try to come up with ideas they would not have pursued uh otherwise uh if they if they stayed under the impression that LLMs can do anything; all they need is enough data. That's that's definitely not true; like even after ingesting every IQ test in the world, still not—they cannot do ARC, even though ARC looks exactly like an IQ test. And fundamentally the reason why is because each puzzle in ARC is new; it's something that you cannot have memorized before; it was created for ARC. Um, and uh LLMs have basically no ability to adapt to novelty in this way. And if you want to solve ARC, if you want the million dollars, uh you're going to have to come up with something original, something that's going to be on the path to AGI as opposed to LLMs, which are more often off-ramps on the way to AGI.

And sorry, just as a tiny technical detail, so when one enters the competition, you Francois do not tell their LLMs the questions; they have to sort of let you give the questions without letting the people who wrote the LLMs know what the questions were, right? So the way it works is that you submit a program in the form of a notebook, and you have access to some compute, which is uh 12 hours uh with one P100 GPU and uh one multicore CPU, and uh within 12 hours uh you need to solve 100 hidden tasks. And so you're just submitting the program, okay? So you are never directly seeing the hidden hidden task; it's only your program that you've uploaded that's going to that's going to see them. And then what you get out of that is a score—how many tasks did your program solve—uh and then and then you have to iterate and come up with a better program. How large are these programs? Um, well, um we'll see, but they're competition-constrained, as I mentioned; they can only run for 12 hours, and they only have access to one GPU. Um, so we'll see, but I mean, just as as a complete outsider, when I have an LLM—I kind of, since I don't have an LLM, I think of it as it must have a huge amount of data that it needs to call up to answer these queries. Is that part of what they're sending you—the whole sort of compressed data set—or is it just the weights of different neurons? So if you do want to use LLMs uh in the competition, the way you would do it is you would make your pre-trained LLM part of your program. So uh before submitting your program, you would fine-tune uh your LLM on ARC data, uh and by way, so you're not going to be able to use an LLM API like the ChatGPT API, for instance, because that would require—obviously, it would require kind of showing this third-party service the hidden tasks, which—I mean, again, for the for the non-experts—that means that your competitors are not allowed to call out to the outside world. No, they exactly—you you you actually don't have internet access at all, right? So anything the program needs access to must be part of the program. So if you want to use an LLM, it has to be an open-source LLM, uh and you include it uh in your program, uh so beforehand you would fine-tune it on ARC-like data, presumably, um and then you would you would actually use it as part of your program. And uh so of course it cannot be an LLM that's too large because you just have one P100 GPU. So that said, that's enough for uh if you're using a Float16, that's enough for models that are like 8 billion parameters, which is actually pretty good. Okay.

And going along with this claim that the LLMs are not really intelligent, I've seen related claims—probably from your Twitter account—that they can't reason and they can't plan either. Are these is that a correct characterization? Yeah, that's correct. And uh you know, I could talk about it a little bit, but really I think what what you want is more than just a vague summary. If you want precise scientific references, uh I can I can send you some. So actually, me, let me pull up—there's uh there's this uh professor from uh [redacted] State University has a really good—we can put up links once we publish the episode on PreposterousUniverse.com so people can—we'll get a link to it. Um, do you have any way to send—send you links in here? Uh, there's a chat on the on the right; you can just respond to that. Perfect. So yeah, you can check check out this uh this YouTube video, and the guy also has a bunch of papers, but really, like I could send you a reading list if you want. But if you actually rigorously investigate uh the ability of LLMs to plan or reason, you find that no, they cannot plan or reason. But what they can do is memorize uh patterns, uh memorize programs, and they can reiterate them. And as long as you are uh uh looking at a familiar task where the program is applicable, they will be able to show the appearance of reasoning by fetching the program and applying it. But that's that's kind of different from actual planning and reasoning. And the way you can tell it's different is that if you modify the task a little bit so that the existing program is no longer applicable, the LLM will fail, right? And intelligence would really be the ability to adapt to these changes. So instead of fetching a program, uh uh an interpolated program, it would be the ability to synthesize on the fly a new correct program that matches your novel problem. If you have that, and you can synthesize this program efficiently from just a few examples, uh then you have AGI, then you have general intelligence. And if you have that ability, you should also be able to solve ARC, by the way, because this is what ARC is all about. For each puzzle, you get a couple demonstration examples, and then you get a test example. Um, and if you were able to synthesize on the fly a correct program that matches the demonstration examples, um then then then you would be done. LLMs fail at that because all they can do is fetch. And of course, each puzzle is something they've never seen before, right? And and you know, I feel like I feel like people who um claim that LLMs can reason, they're really stuck at this first stage where they see examples of something that look like reasoning, and they don't try to investigate it. They're like, oh, it's it's working; this is impossible if the LLM was not reasoning, right? But actually what it's doing is just fetching a program, and uh that's just that's just memory. That's just memory. Like an LLM is a program database; that's it. It's an interpretative program database. Intelligence is not being an interpretative program database; intelligence is being the programmer; is having the ability to look at something new and come up with a new program to to to address it.

Well, you just hinted at this a little bit, but I am certainly hearing a lot of people who are nominally experts in the field make noises about artificial general intelligence and how close we are to it, if we're not already there. Yeah, we we we—I mean, the claim that we're already there or like or like LLMs are like high high schooler level in intelligence are absurd. Like I don't I can't even fathom how you can make such claims; it just it it makes zero sense to me. Like I don't I don't even understand how how you can be like so deluded as to as to claim that. Um, but you know, if you want to ask seriously, uh under my definition of of intelligence, which is uh obviously correct—like my opinions are obviously correct, right?—of course, that's why you're on the podcast. The—no, but uh if you want to ask when is AGI coming, it's very difficult to answer because the the situation we're in is that we have no technology today that is on the path to AI. There is nothing that if you just scale it, it gives you intelligence, right? Right. Uh, but that said, uh that does not necessarily mean that AI is very very far away; rather, what it means is that you cannot predict when it will arrive because you need to invent something new. But maybe we'll invent it next year; like maybe the The ARC competition uh will actually uh trigger someone into inventing it, you know. Uh, so maybe it arrives next year; it's possible. It's possible, but it's unpredictable because it doesn't exist yet. And the claims that people are making um are basically that they're founded on the idea that um LLMs are on the path uh to AGI and that you can predict uh how their intelligence will scale uh with with compute and data, uh and uh and the idea is that well, uh GPT-3 was like uh middle schooler level; uh GPT-4 is like high schooler level; GPT-5 is going to be like postdoc level; GPT-6 is going to be super genius, and so on. And um I mean, none of it makes any sense even with with a very loose definition of intelligence.

And do we do we understand what is going on inside the large language models? I mean, how much of a black box are they, or are we still kind of doing the science needed to figure out what is inside the box? Uh, we are still in the process of figuring out how to interpret what they're doing, but there's already a lot of work that has been done uh along the lines of interpreting how LLMs work and visualizing what they're doing. There was a paper from Anthropic uh few days or weeks ago that was actually really insightful on that topic. Okay, so it's that's not like an intractable problem; we will get better; it's not intractable; it's it's an active area of research, and we are making progress. Okay. And by the way, every…

Every time we get new results, uh, they are along the line of showing that LLMs are actually just pattern-matching engines. They are not, not, not intel—they are interpretative databases of programs, again, again. Like the big difference between intelligence and a program database is: like the program database is like GitHub; intelligence is like the programmer. The programmer individually, uh, it knows, uh, dramatically less than what's in the database, but the database cannot adapt. It's only that fixed set of programs. You can maybe recombine some programs, but you cannot just—you have limited ability to recombine programs. Um, the programmer can actually invent anything, adapt to anything, because it has general intelligence, right, right. And that's really, that's really the difference. And people are like, yeah, so if we just scale a GitHub to like, uh, a thousand more programs, then it's going to be AGI. But no, it's just a bigger GitHub; it's just a more general GitHub; it is still not a programmer. There is no level; there's no like amount of stored, memorized programs where you, you develop suddenly the ability to synthesize your own programs on the fly. It's just not how it works. If it worked this way, we already know, because we've already scaled LLMs to literally all the training data that's available out there, which, which by the way is the reason why we have entered the plateau since last year. Uh, it's because we, we, we've been running out of data. And sure, you can scale compute; you can always keep scaling compute, but it's, it's becoming useless because the curve needs to be fit to something; the curve is literally just a representation of a training data set. Uh, if you, if you've run out of data, and how do you improve the model? Well, one way is that you can try to better curate your training data, so you don't increase the scale of the training data, but you increase the quality. That's actually one, uh, one very promising way of, uh, improving LLMs. It's what, it's actually the way LLMs keep improving today. We've already run out of data; to the next stage, that we better curate the data. We are not training the LLMs on more data; we're actually curating it. And I mean, we, we were technically, we're still collecting new data from, from human users, um, so there's a bit of an increase, but on balance, it's actually decreasing. Um, and, um, yeah, and but, um, you're not going to, you're not going to magically find a thousand times more, uh, novel, non-redundant data to train these models on. It just, it just doesn't exist. You're not even going to find 2x, you know, right. Um, and that's, that's the cause of the plateau we've been seeing. And you know, like something like GPT-5 is going to be released, like probably at the end of the year, uh, is going to be a big disappointment, because, because it's, it's not going to be meaningfully better than GPT-4, you know.

It occurs to me slightly belatedly that we should tell people, people what GitHub is, because not all of them will know, right? It's basically just a website that's a, a collection of many open-source programs, uh, put there by organizations, by programmers across the world. And that's not, I mean, so that's the, your analogy for what current generations of large language models are. What we want, in some sense, is something that is more truly creative and has the ability to learn outside the extrapolation.

Yeah, that's right. And even, uh, if you, if you take a, a first-year CS student, uh, their knowledge is extremely limited. They know so little; they've seen so many, so, so few real-world programs, but yet they have a, a much higher ability, uh, to write programs, uh, that are appropriate for a novel problem compared to, uh, a system that has seen every open-source program out there, but that, that is very true to intelligence. Yeah, okay, very good. But so, and I, I'm 100% on your side here. I've tried to convince people that the amazing thing about LLMs is how well they can mimic sounding like human intelligence rather than thinking in the same way that human beings do. Yeah, but, but I think that's actually quite intuitive, like, because you also see it in humans; you also see it in humans that there is this tradeoff between memorization and intelligence, and that with enough memorization you can actually, uh, reproduce the same outcomes as intelligence, uh, and, and the way you can tell apart someone who's operating based on memorization and someone who's actually intelligent, uh, and is operating based on understanding is by presenting them with something new, right? So it's, it's true; it's true for human beings as well. And the reason why our intuitions are off with LLMs is because the scale of memorization is unlike anything that's possible for human beings. Well, and maybe also trying to sort of, you know, um, give some credit to the other side: maybe more problems that we're interested in than we think are solvable by memorizing lots of things rather than by thinking originally and creatively.

Sure. I mean, LLMs, that's that memorization is precisely what makes LLMs useful; is that they've stored lots of patterns for how to perform certain actions, solve certain problems, uh, and they, they can fetch these solutions and replay them, and you may not know about these solutions, so they may actually teach you something new.

Can, uh, well, can an LLM, or could AI in some broader sense, be functioning as a creative scientist? Not an LLM, uh, at least not an LLM in isolation, uh, to actually make the systems capable of invention, capable of developing new theories and so on? Well, either you can, uh, have a human in the loop; the human is actually in charge of the intelligence bits; the LLM is in charge of the memory. So you use LLMs as a sort of like extension of your own memory, sort of like a brain add-on, uh, so that's one way you can, you can create a super scientist that way, by just supercharging an existing scientist, uh, with, with access to all this memorized content, right. Um, and, um, by the way, I'm not, I'm not convinced this is actually super effective, honestly. What, what I've seen is that LLMs are very good at turning, uh, people who are, who have no skill into people who are capable of an average, mediocre outcome. Um, they are extremely bad at helping someone who's already extremely good getting better. It's, it, it basically doesn't work. And, uh, there are many reasons why, but empirically this is, this is what you see, uh, so this is why I don't think LLMs are going to have much impact, uh, in science. The science is not about more mediocre papers; it's actually about the, the top ones; this is what, what's actually conducive to progress. And the, the other way that you could try to make this system capable of novel discoveries is to, um, try to add a search component, like we talked about, uh, genetic algorithms as a way to mine, uh, a search space and find, find unexpected points, unexpected inventions in it. I think you may be able to, uh, create sort of like hybrid LLM plus, uh, symbolic, uh, search systems that would be capable of invention.

I definitely noticed when I ask physics questions of LLMs, if it's a, if it's a fairly straightforward question, they're pretty good, but as soon as it becomes subtle, they are no longer good. I mean that in exactly the places where you don't get a lot of coverage out there in the training data, they can't figure it out. And as you say, like, why would we ever expect them to? Yeah, if they, if they've, uh, seen, uh, many instances of the problem you're asking, they have memorized the solution template, and they can just fetch that solution template, reapply, to give you the right answer, right. Um, if it's something that's slightly different, or that's similar but with maybe one word that actually changes the meaning, something like that, they will still fetch the same template pretty much, but now it's going to be wrong, and they have, they have no, no way of telling, because they don't actually understand, uh, the words that you're, that you're putting in; they don't understand your query; they're just, they're just directly mapping to the solution that they think they know. So there's this famous thing where you ask an LLM a question, it gives you the wrong answer, and then you say, no, that sounds wrong, and it corrects itself. Is that because it actually is correcting itself, or is it just trying another possible answer from its storage of, uh, possibilities?

Uh, it's, uh, adapting, uh, its solution based on, uh, patterns of program modification that it has seen before. So if you propose a pattern, and then you add to it, oh, by the way, this is wrong; here's the correct pattern, and you do this many, many times, the model learns a sort of like modification function that goes from this incorrect solution to a fixed solution. And if you tell it, oh, by the way, there's an error; please give me the right answer, what it's going to do is that it is going to apply, uh, this modification function to, to the previously, to the input it previously produced, and going to give you a new answer. You, and you, you, you, you may be like, hey, so why don't we do it preemptively, uh, but the thing is that in the absence of human feedback, there is no particular reason for the modified program to be more or less correct than the initial program. Like only the human can tell. Yeah, so I think I know what your answer to this is going to be, because we talked about intelligence before, but what about the, off, um, proclaimed dream of letting the AI program a smarter AI, and therefore sort of bootstrapping our way up into greater and greater intelligence?

Well, right now, if you're, if you want to use an LLM to do the programming, uh, it's going to be, uh, constrained by its training data. It can only give you things that are simple interpolations of programs, uh, code snippets it has seen before, which is why LLMs work great as a Stack Overflow replacement, but they do not work great as actual software engineers capable of novel problem-solving. And you know, the, the average senior software engineer, um, is a tremendously capable novel problem-solver, but they're also completely unable to invent an AGI. So you're not going to get an LLM, which has no novel problem-solving ability; you're not going to get it to invent—I mean, you cannot even invent the solution to an arc problem, right, which is, which is pretty trivial; like a four-year-old can do it, um, so no, that's not going to work. Um, but you know, you could ask, hey, uh, why, why, why just use LLMs? Why couldn't we use something else, like genetic program search, since, since I mentioned that genetic algorithms could actually invent new things? Well, um, in practice, I think this is kind of a bad idea. It is viable in theory, because if you think about it, uh, humans were developed by an evolutionary algorithm, right? Uh, intelligence is the answer to a question posed by nature. Um, could we not just get the same answer by asking the question again and just letting a system, you know, run, run its course? In theory, yes; in practice, bad idea, because the scale at which you need to run is excessive. And I'll tell you, um, we already have general intelligence; we are general intelligence. And general intelligence gives you, uh, an extremely effective ability to predict what, what next idea should be tried. Uh, if you try to delegate this sort of like ideation bit to an algorithm, you're wasting resources, right? Uh, because what's going to be, uh, computationally intensive is actually evaluating the solution, trying, trying, trying to implement it, figure out whether when it's actually on the path or not, and so on. Uh, the ideation bit, uh, is not expensive. And so what you're doing is you're effectively outsourcing the things that you're really good at and that cost you very little, uh, to a machine that's really bad at it. And meanwhile, the things that are actually automatable and very expensive, uh, while the machine still has, still, still has to do them, you know. Um, so, uh, it's, it's just an extremely ineffective, uh, uh, idea. Decided that, hey, we can just like brute-force our way to the right AGI architecture; it doesn't work. In fact, it doesn't even work, um, uh, on a much smaller scale. And by the way, another issue that you're going to run with this brute-force search idea is that, um, your search is only going to find you points in your initial search space. You start, as a human programmer, you start by defining the space you want to search over, like all, all possible genomes for your search algorithm, for, for, for genetic search algorithms, for instance. And, um, what if the correct solution was not in your search space? You know, uh, if you don't know where the correct solution is in advance, you have no way to tell. So maybe you're going to be expanding like an extraordinary amount of compute resources to mine a search space that does not even contain the right solution. Um, and this whole idea doesn't even work on, on a much smaller scale, like for instance, um, neural architecture search for a long time was a thing in deep learning. The idea was that, hey, uh, researchers have come up with a number of architectures that perform really well, like there was LSTM, there were Transformers, and so on. Uh, could we not just make a machine that tries a bunch of different architectures, and you should find a better one? It has never worked. There's literally nothing out there that's popular that was developed by an algorithm, despite tremendous amounts of compute dedicated to, to the idea. Um, everything out there, like Transformers for instance, or even the, the more, the more novel and recent architectures like Mamba, XLS-M, and so on, uh, all of these were invented by humans, because humans are really good at inventing, you know, like AI is not idea-constrained today. So trying to outsource ideation is just a bad idea. But it's not because we're magical, right? I mean, it seems like we should, in principle, be able to write computer programs that are as smart as us.

Yeah, in principle, sure; it's just not a straightforward problem; it's just not an easy little RH. The human brain is tremendously complex. Yeah, fair enough. And no one really understands how it works today. So I guess that you're not going to assign a large probability to, um, the existential threat of AI taking over the world. No, so, no, to start is because AGI is not a technology that exists today, and that we have nothing today that would lead to it. We need to invent it; we need new ideas. And this is the entire point of the, the AGI competition is to get people to come up with new ideas, because currently we are stuck, right? We're on an off-ramp, uh, so we need a, a reset. Um, but you know, even if we had a promising avenue to create AI, uh, I think the whole idea that AI is going to end humanity, um, it's, it's based on several deep misconceptions about intelligence. Like intelligence is, is pretty much just a conversion ratio between the information you have to, uh, the ability to operate, uh, in novel situations in the future. Your turning intelligence is you turning your past experience and also the, the knowledge that you're born with, because you know, the day you're born, you're actually not born knowing nothing about the world; you know some things about the world; some things are hardcoded into, into your genes. Um, and so you turn that—it's mostly your experience, but you, you, you turn that into the ability to approach, uh, each new day in your life and actually behave appropriately throughout your day, accomplish your goals and so on. And, um, this, this ability to sort of like chart a path through situation space, uh, does not entail, uh, that the system should have goals of its own or values of its own that you would need to align with human values. It is just, uh, an ability; it's, it's just a path-finding ability. Um, and in order to make something like Skynet or, or, or Terminators, well, you need more than just intelligence, right? You need intelligence plus, uh, goal-setting, autonomous goal-setting. But why, why would you want to give machines, potentially very capable machines, with autonomous goal-setting? Sounds like a bad idea. Uh, and of course, if you have goal-setting, these goals need to be grounded in some value system; you're going to want to give machines their own values, uh, and of course, you're going to want to give machines autonomy, because intelligence does not imply autonomy, by the way, right? So autonomy, in the sense that the ability to perceive the world and act in the world without mediation by humans. There is no machine out there today that, that is unmediated from humans, if only because they need a power supply. There's no machine out there that can just recharge itself and maintain itself, uh, in, in, in perpetuity, right? No machine today has autonomy. So in order to create a danger, you would need to engineer the danger very deliberately, like Skynet, to be honest. You know, the whole thing with Skynet is like, hey, we have this very intelligent thing, and we've given it the ability to make its own autonomous decisions based on its own value system. Hey, let's hook it up to our nuclear arsenal. Uh, it's, it's, it's, it sounds like a bad plan, right? So to make something dangerous, you literally have to create an agent, um, give it, uh, autonomous sensing, give it autonomous acting, give it its own value system, give it, uh, its own autonomous black-box ability to set goals with no human supervision, uh, and then you give it super, super intelligence, right? Well, to be honest, this whole thing already starts being dangerous even before you add intelligence, you know. Uh, and intelligence in itself is just a tool; it's just a way to accomplish goals, uh, uh, if you don't, if you don't hook it up to autonomous goal-setting, then it is pretty much harmless. It's not quite harmless, because, uh, it's going to be in the hands of humans, and humans are dangerous, so it's dangerous in that sense that people are going to potentially use it for, for bad, uh, purposes, but it's not, uh, dangerous in the sense that it would compete with the human species, right? It's no more dangerous than any, any other tool that we have. It's like, it's like, you know, efficient energy is not on its own dangerous; it's just a tool; you can use it, uh, to create clean power, right, or, or you can use it to make a bomb. But if it's going to be threatening, it needs to be deliberately engineered to be threatening, right? And I think, I think AGI is going to be the same. I also think, you know, it's kind of pointless to try to plan for risks in something that is completely, uh, unknown. Like we don't know what AGI really looks like; how are you going to plan for how you're going to handle it? Um, so I think how to handle AGI is, is something that we're going to start making meaningful progress on when we start having it, right? And again, AGI on its own is not a threat; it's just a tool. To make it threatening, you need to engineer, engineer it into, into either something completely autonomous, which sounds really like a bad idea, or, or, or just turn it into a weapon in, in, in the hands of humans.

Well, you've done a very good job of sort of deflating some of the misconceptions about intelligence and large language models and so forth, but I wanted to maybe wind up with giving you a chance to talk about, you know, your day job, because you work on these things; like you, you actually have a lot of, uh, positive things to say, uh, about deep learning models, etc. So, so let's open the door a little bit on what it means to be developing these things. I mean, you have a very successful software package that 3 million people use. Is it, is it something that, should more people out there be developing and training their own large language models?

Absolutely. No, not just large language models, but any sort of deep learning model. I, I think, you know, it would be a sad world, uh, if there were only a fixed set of companies training models and just, just giving those models to, to other people, to, to be consumers, uh, of those models. I think we want, uh, uh, this technology to be a tool in the hands of everyone. I would like every software developer out there to be able to, uh, tackle their own problems using these tools, using deep learning, using large language models, using Kols. Um, and that's, uh, basically the reason why I try to make Kols as accessible as possible, as approachable as possible. So what is, what is Kols? What does that mean? So Kols, Kols is a deep learning library, so it's a software library, uh, for building and training your own deep learning models on your own data. And you, you, it's not necessarily building models from scratch; you can also adapt an existing model, like an existing large language model, for instance. So

Could you take an existing large language model and then feed it all the transcripts of the Mindscape podcast and sort of elevate their importance in the model so that it would, it would mimic some, uh, average scholarly Mindscape, uh, guest kind of point of view?

That's right. So if you want, for instance, to generate new episodes, uh, that's something you can do. You can take, uh, uh, the Gemini 8 billion model, for instance, which is an open source released by Google. It's available in K, uh, you find unit to predict the next word on your transcripts. You can, you can use a technique called LoRA fine tuning, which is basically computer-efficient fine tuning, um, and and now you can generate new, new transcripts. It's probably not going to be, uh, very, very, but it's probably, you know, um, it's going to sound like your podcast. If you, if you start listening, you're probably going to raise eyebrows quite a bit, but at first glance, it's going to sound like your podcast.

Yeah. I mean, have you done this with the equivalent? I know that I, I have not, but people have done like, uh, gen, gen podcasts like this.

Yes, yeah. Do, are you aware of the, um, experiment that was done with the works of Daniel Dennett, the philosopher?

Uh, no. So, um, Eric Schwitzgebel, who's another philosopher who was another guest on the podcast, he and some collaborators trained an LLM on all everything ever written by Daniel Dennett, and then they, uh, asked it some questions. And they asked Dan Dennett these questions. Dan passed away recently, but before that happened, and then they asked some philosophers who are familiar with Dennett's work which of these answers was the real Dennett, and they did better than chance, but in some cases not a lot, depending on the question.

Yeah, honestly, if you're just looking at short text snippets in isolation, it's very hard.

Exactly. And especially, you know, when, when you're reading the output of it, it's meaningful because you're interpreting it, you know.

Yeah, you're giving it more credit maybe than it deserves.

Yes. I mean, LLMs are all about mimicking humans, so they're, they're very good at hacking, uh, your, your theory of mind, at, at, uh, because you have this bias towards, um, interpreting as being like you anything that superficially acts like you.

Right. Yep. The intentional stance, Dennett talked about this actually.

Yes. So I mean, how realistic is it for the typical listener with, you know, a relatively late model MacBook Pro to, um, you know, open up Python on their computer and download some of your libraries and start going to town?

It's very easy. So you're going to want a GPU. So if you have a, a MacBook Pro, uh, uh, one of the recent ones, you actually do have a GPU, and if you're using the tens of pro backend of K, you can actually, uh, do GPU-accelerated computation on your MacBook Pro. So you can do that. You could also use a free GPU notebook service like Colab from Google, for instance. Uh, it's actually extremely easy to just get started and do a LoRA fine-tuning of the Gemini model with scas on your own data. If you already know Python, it's really easy. You, you, you, you, you'll be done that now. And and if you don't, you've written a book.

I have written a book.

That's right, uh, uh, so the first edition was in 2017, then there was a second edition 2021, now I'm actually writing the third edition. Okay. I have a lot more content on, I would guess, LLMs and image generation as well. And is there, I mean, besides the fact that it sounds like a lot of fun and also educational to do this, are there use cases for people training their own LLMs to do their own specific tasks?

Absolutely. If, if you're a business and, and you're having a specific business problem like, hey, I have this spreadsheet with this information, I want to turn that into a set of emails, for instance, uh, you can just, you could prompt the LLM into doing it, maybe, maybe it will work, but if you want better results, you can just actually adapt the LLM to your problem so that it can, uh, not just fetch the right program, but maybe fit the right program from, from the data you provide. In fact, I would say if you're, if as a business you want to make extensive use of LLMs, you should be fine-tuning your own LLMs because this gives you an advantage instead of just reusing the same program database as everyone, which is the, the sort of like public access set. You are starting to develop your own, uh, repository of private programs, train, train on your own data specific to your needs, and that's very powerful.

Is there a, an app out there that will answer my emails for me?

Um, there probably is, you know, there are just so many. Gen, tviv, yeah, at the, I hope that I don't end up starting Skynet by sending an email that was generated by an LLM, but, uh, that there are, we shouldn't leave people with the, uh, impression that there aren't many, many transformative ways that, uh, even LLMs can affect our lives going forward.

Yeah. So, you know, many, many people try to do things with LLMs that it may not necessarily be suited for. My advice is, in general, do not try to delegate any sort of decision-making to LLMs. LLMs are there to give you a shortcut towards the general area that you're looking for, uh, do not delegate your decision, like, do not let the LLM generate your emails for you, but maybe it can help you write emails faster, for instance, or maybe you can fix typos in your emails. Anything that would make me go faster is, is very good.

So I'll, I'll take that as useful advice, France.

Thanks very much for being on the Mindscape podcast.

Thanks for having me. [Music]