What AI Can Never Be | John Vervaeke

Um, so as Dennis said, I'll just make sure my introduction is clear, uh, because I have institutional allegiance. I'm from the University of Toronto. Um, that's where Jeff Hinton is from. That's where Jordan Peterson is from. That's where Keith Stanovich is from. So yes, all knowledge and truth flows from the University of Toronto. Um, and I am in cognitive psychology. I'm also in cognitive science; I've been the former director of it. I'm, um, I consider that cognitive science program my baby. I do have a PhD in philosophy, and a big part of my work in cognitive science is actually the philosophy of cognition. Uh, so that's where I am. That's, uh, uh, what I'm doing. I taught—where Jeff Jeff Hinton and I taught a lot of the students that went into the world to—that made a lot of this possible.

Um, as some of you know, Jeff quit his job so that he could speak about the dangers of AGI. Um, I, I'd like to talk to Jeff about what he thinks about human beings he's trying to preserve with his moral commitment. Uh, but I'll put that aside for now. What I'm going to do with you is I'm going to talk to you about AGI and intelligence. Sorry. AI and intelligence, AI and rationality, AI and reasonleness, and AI and wisdom. And what I hope to make clear is intelligence is not synonymous with rationality, with being reasonable, and with being wise. And there's some deep reasons why that is the case. Let's start with AI and intelligence. Okay.

So, as you no doubt know, people are now talking about AGI, artificial general intelligence. I've been talking about general intelligence for two decades and saying that's what we need in artificial intelligence—is artificial general intelligence. Um, and so what does that mean? What does that mean? Well, you have general intelligence. It means you can solve a wide variety of problems in a wide variety of domains in a wide variety of manners for a wide, uh, variety of goals. And so you can learn tennis, you can learn about Albanian tin production, you can wonder about Australian history, and you can think about if there's any possible connections between those—and that you do that on a regular basis. So we're talking about general intelligence because that's what is claimed—or at least it's claimed that's what we're moving towards—with the large language models, which I will call the LLMs, the large language models like chat GBTS, seek, and so forth. Okay.

Uh, obviously there are other versions of potentially AGI out there. I can't cover everything. I'm zeroing out on them because they are dominating the airspace right now. Okay. So, let's, let's ask this question about general intelligence, but let's be really, really careful because there's two questions about general intelligence that are regularly confused together, and we should carefully distinguish them. One is, can the LLM solve a wide variety of problems in a wide variety of domains? I mean, after all, that's what I just said general intelligence is all about. And the answer is yes, but in a specific kind of way, a specific kind of knowledge. And I'm going to talk about that later—about different kinds of knowledge, different ways of knowing—because we've tended to overemphasize one kind of knowing, uh, to the detriment of others that are very actually crucial to our cognitive agency. So in that sense, we've got a bit of an answer. Yeah, they can do that. And it's a significant success. By the way, until, until GPT 3.5, GPT4, AI, and it wasn't AGI at that point, faced what was called the silo problem. The artificial intelligence could solve problems in a very delimited domain. It could generalize a little bit. It maybe could play some games well, but it couldn't learn how to play tennis or something like that. So, this was a silo problem. And one of the unexpected results of the LLMs was they seemed to break out of the silo. You can ask Chad GTP about this domain or that domain or that domain or that domain or that domain. And that is impressive, and I don't want to diminish that accomplishment. That's a significant accomplishment. But I also want to set that accomplishment into its proper context. Is this—did we get a scientific advance? Did we get an theoretical explanation of the nature of intelligence that we can generalize to intelligent creatures? Can we learn from how the LLMs learn and work? What it is that makes us intelligent or makes a creature that is non-controversially quite intelligent, a chimpanzee, a Caledonian crow, or an octopus intelligent? See, this is the mark of a scientific theory. It has to generalize over the phenomena that it's attempting to explain. Nothing in the way chat GDP acquires its ability is generalizable as an explanation of how a chimp is intelligent. You can't use anything from how G—how the LLMs work—to say, and this is probably how the chimp is intelligent as well.

Now, that's crucial. See, one of my great hopes—and what I taught my students—was we would get artificial intelligence by getting a scientific explanation of the nature of intelligence. My greatest fear was that we wouldn't get such an explanation, but we would sort of hack our way into a machine that could do many different things in many different domains. And my hopes were dashed and my fears realized, which of course is the tragedy of being a human being. When we ask the question, what is it to be intelligent in a scientific way? We have to look at a different approach. So what they mean by general intelligence is how you do in any one domain. This was discovered way back in 1926 by Spearman—is very predictive of how you'll do in many other domains. That's still not the case for the LLMs, by the way. Uh, but they're moving in that direction. But then what you have to say is, okay, there must be some general ability behind how you do well in art and how you do well in math and how you do well in sports. By the way, those all predict each other—contrary to what Hollywood tells you—they all predict each other. If you do well in one of those, you'll—your chances are—you do well in the others. Okay, this is something also we have to give up; we have to give up a lot of cultural tropes around intelligence because they're, they're, they used to be misleading—now they're dangerous.

So instead, you have to ask, okay, what is this general ability? The proposal that I've been arguing for—and a bunch of other people—it's not just me, uh, we've been arguing that what general intelligence reflects is the ability to solve two interlocking meta problems. So I'm an academic, so I have to use the word meta at some point. Okay. Okay. So, what do I mean by that? A meta problem is any problem you have to solve whenever you're trying to solve any of your more specific problems. So, whenever I'm trying to solve this problem, I have to solve these two meta problems. Or this problem, I have to solve these two meta problems. Does that make sense? That's why it would be a general ability because it applies to all of your problem solving. Now, what are these two general meta abilities? One is your capacity for anticipation. Michael Leaven talks about this as your cognitive light cone. How deeply and in detail you can anticipate the future. You know that this has an aspect to do with intelligence because people regularly and reliably intuitively use this as a way of judging the intelligence of other organisms. For example, this is why you believe your dog is smarter than a frog because you know that the dog—your dog—can anticipate the world in a much broader and more detailed fashion than a frog can. We have an emerging framework that is becoming very powerful in cognitive science for explaining this ability. It's called predictive processing, and I'll talk a little bit about that. The second general ability for solving a meta problem is relevance realization. This is something I have been working on for three decades. This is this problem. Okay. There is too much information. Now we think, oh, I have to gather more information. Now, I'm going to say something to you. And it's going to sound like a Zen koan. What makes you so intelligent is your ability to ignore vast amounts of information. Think of all of the things you could be paying attention to in this room and all the different patterns. I could pay attention to the top of her head, his toe, that—that's one pattern. I could pay attention to the circumference of this, the length of that, and how warm it is. The amount of information in this room, just this room, is combinatorially explosive. Okay? So, you have to ignore most of it. But here's, here's the thing. You don't just ignore it randomly or arbitrarily because you're debt. You somehow ignore most of that information and zero in on the relevant, relevant information you should pay attention to. But you have the same problem with memory. Do you know how much information you have in memory and all the possible combinations? You can't search through your memory that way. You ignore most of your memory whenever you're remembering. But somehow you, you—with you—with you retrieve what's relevant. The possibilities you can consider, the sequences of actions. This is overwhelming. You're doing it right now. Out of everything you could be paying attention to, everything you could be remembering, all the possibilities you could be considering, all the actions you could be, you're zeroing in on what to do, and you're doing it right here, right now. Like that. Okay, that's at the core. And these two problems are interlocked because the more I try to anticipate the world and think about how adaptive that is. Do you want to anticipate the tiger or fight the tiger? Do you want to anticipate where the salmon are or just come upon them? Do you want to, you know, anticipate winter or just undergo it and suffer it? But the more you open that space of anticipation up, the more the relevance realization problem goes up—and it goes up nonlinear—goes up exponentially. The two problems are interlocked together, and you're solving them both right now, like that. And you're adjusting your light cone and you're zeroing in on what's relevant. You're foregrounding some things, you're backgrounding other things, you're ignoring other things. Some things are salient, grabbing your attention; other things—your left big toe wasn't salient to you until I said that. Okay, it's very dynamic, very complex, multi-level. The people who are working on predictive processing and relevance realization and how they interlock together are generating theories that are intended to explain the intelligence not only of human beings but of chimpanzees, octopus, Caledonian crows and gather the requisite empirical evidence to establish them.

Okay. Well, so what we could ask—if we're asking the second question about general intelligence, the scientific question—we can then ask, okay, do the LLMs have anticipation? Do they have predictive processing? Do they have relevance realization? Now, we have, see, what I've done, I'm supposed to be a teacher. What I've done is enable you to ask more precise questions. Rather than asking the vague and confused question, are they intelligent? We first separate off—well, there's the it can do lots of things meaning from it has the connection to the scientific explanation—and then based on that we can ask specific questions. Okay, great. Well, the LLMs actually get a lot of their power because of the capacity to anticipate. That's actually what they do. They make probabilistic predictions about what term is coming next. And I literally mean like a term. No, we also get—we also are—we lack care. We say they predict words. The word word is ambiguous. It either means like a sequence of letters or something that has a meaning to you. They're doing the first—they're predicting the probability of cat A if DOG has appeared. Okay, do you understand? So they are—and what they're—what they're doing is—they're capable of very complex, very fast prediction. So in to that sense, they have some anticipatory ability. But we again—now that we got a more precise question—we can—we can—we could—sorry for this irony—we can prompt things a little bit more. Okay. See, because all they are doing is predicting within the domain of literacy. Now, literacy is largely transparent to you. The most studied effect in all of psychology is called the Stroop effect. The Stroop effect, I kid you not, there's been like over a thousand studies. If you want to go to the psychology department, you have parties, you have to know about the Stroop effect, right? It's so—what I do is I'll give you a bunch of words. They're color words like the word red, but the—the word is in the—the ink is in blue ink. So the word red is in blue. Does—do—do you get it? And you have to say—people tell me the color of the ink—and people go blue. Reading is so automatic. It's transparent. I'll sometimes put cat on the board and I say, "Okay, I'm going to put this on the board. Don't think about the land shark with brains that pretends to love you." But they all do. They think of cats. Okay. So let's do this carefully. We take our judgments of relevance, and then we create an artificial system that is correlated with those judgments called language. Language is not how the world works. Language is an artificial entity, and it's correlated with our judgments of relevance. And then we take language and we create another artificial system—literacy—that is correlated with language. Now it turns out if you produce—if you predict literacy—you're predicting language in a pretty good way—and then that—because it's correlated with judgments of relevance—anticipates the world in an interesting way. But to be precise, these machines do not directly anticipate the world. They anticipate how we would talk about our anticipation of the world. Did you get the difference? That's why they can't generalize to explain how a chimp is intelligent. That's not—chimps don't get their intelligence by trying to figure out how we are anticipating how we'll talk about how we're anticipating the world. So we should now ask the more—now we can again ask a precise question. What is involved with us anticipating the world? Not just anticipating writing about how we'll talk about how we'll anticipate the world. What's actually involved? And this gets us into the four kinds of knowing. Okay.

Now, the one you're all familiar with—and the one that is highlighted by the large language models—is propositional, linguistic, literate knowing. I'll just refer to this as propositional knowing. This is knowing that something is the case. How many of you—quick answer please—how many of—how many of you know that cats are mammals? Put up your hand. Some of you are—didn't put up your hand. I'm worried about you, but okay. Okay. How many of you can remember when you learned that fact? Don't guess. Can you actually recall? Okay, so note that please. Okay, remember that. Okay, so you—this is your knowledge that—and what it gives you is—it gives you beliefs about facts. I have beliefs about cats. Okay, I have beliefs about trees. I have beliefs about lakes, etc., etc. Okay. What you get with that is you get beliefs, and you have a sense of them being true, and they're stored in a particular kind of memory which psychologists call your semantic memory. Okay. Now I want you to compare that to a different kind of memory you have. How many of you know how to swim? Put up your hands. Now that's different, isn't it? Because swimming is not knowing that something is the case. It's knowing how to do something. And if I gave you a whole bunch of manuals on swimming and asked you to write the best possible summary you could of all of those manuals and you gave me a really excellent one, would you know how to swim? No, you wouldn't know how to swim because what does that involve? That involves your sensory motor interaction with the world. It involves a different kind of memory. It's actually called procedural memory. And what we know is your semantic memory can be damaged and your procedural memory is undamaged and vice versa. They're distinct, and they're distinct for a reason because they're doing different things and solving different kinds of tasks. And your sense of your skill—skills—aren't true. Is—is your skill of swimming true or false? What? That doesn't make any sense. Beliefs are true or false. Skills are powerful or not. They apply or not. They fit the situation or not. They have a different sense of realness to—they—your skills give you a sense of realness not because they convince you of the truth of a statement or a belief. They give—they empower you to make a causal difference in the world. It's a different kind of knowing. And here's the thing. Most of your ability to navigate the world depends on your procedural knowhow, not your propositional knowledge. That—now there's a possibility we might give procedural knowing to robots—uh, sorry, to LLMs—but that means we have to give them the capacity to engage in sensory motor activity with the world—and then we have the—that science fiction fantasy of I, Robot—and what we'll do is—oh, but we'll have the LLM sitting inside robots—and I've seen a video of some of these robots—it's like—well, be first of all—be careful—I mean, for completely pedantic, mundane reasons—I don't think we'll be able to generate a lot of robots—uh, the—the labor that goes into that—the—the—the mining—the rare earth mineral mining that we have to engage in—the transportation costs we have to engage in—we have to come up with a power—a power source—and—and the LLMs are not taking us in the direction of really small autonomous power sources—they're taking it the opposite—you know, training up chat GBT, you know, an LLM takes the energy for an entire city like Toronto for two weeks. It's not running the way we run. So, for very sort of mundane reasons, I, I think iRobot isn't anywhere in our future. The other thing is we've discovered that—well, what—hey, what else was in—how many of you have seen iRobot with Will Smith? Okay, so there was all the robots, obviously, but what else was in iRobot? Autonomously driving vehicles. Where are—where are they? Where are they? Not really. Okay. So, why did it—like I had students writing about autonomous cars in 2010 saying they're just around the corner. And it turns out that when we've moved towards trying to do the procedural knowhow, we've encountered really complex, hard problems. Can we solve these problems? Yes. Are we really working on them right now? No. Okay. So, there's a difference there. There's an important difference there. This means that our embodiment makes a significant difference to our cognition. Because you're embodied, you know the world in a powerful way. Now, none of you are swimming right now. At least I hope not. Doesn't look like you are. Why not? 'Cause you know where you are. You know where you—you know what it's like to be here now in your state of mind and what the situation is. This is perspectival knowing. We use a metaphor from art. You—you're taking a perspective on the situation. This is—you have a state of mind that fits the situation. So you're properly oriented, and you—right—notice the right things, and that noticing tells you which skills you should activate right now. That's your perspectival knowing. It's knowing what it is like to be you here now in your state of mind in this situation. Knowing what it is like. Now, knowing what it is like is the quintessential feature of consciousness. Okay. What it is to be conscious is to—to have consciousness is to know what it is like. And it's bound up with how you're orienting, how you're take, how you're paying attention, what are you finding salient and relevant, all of that. And you have consciousness, and you can take perspectives, and you can even take perspectives on your perspectives, and you can take other people's perspectives. You have perspectival knowing, and it has a different kind of memory associated with it. Very different. And we're not even clear if any other organisms have this. So remember when I asked you about cat mammals and you couldn't tell—how many of you remember what—what you had for breakfast this morning? Put up your hand. And what did you do? Did you infer that or did you go back and have a little episode in your mind where you sort of see yourself? You go back into—listen to the language—you go back into the situation, and you sort of see how you were seeing the situation and what it was like to be there. And there on your plate are the eggs or whatever. Oh, look. That's what I had. It's episodic memory, distinct. And its sense of realness isn't a sense of truth. It isn't a sense of power. It's a sense of being present, being in a situation. Now you do something really interesting with your perspectival knowing. You have a particular kind of anticipation associated with your perspectival knowing and your episodic memory. You can do mental time travel. You can imagine the future and imagine yourself in the past. You can even imagine yourself in possibility space. Bertrtan Russell once said, "No matter how eloquently a dog barks, it"

Can't tell you that its parents were poor, but hardworking. Okay? You can do mental time travel. You can even travel into possibility. That means we all do this thing. We string the episodes together, and we string the possibilities together, and we string the past and the future together, and we create an autobiography that we're constantly writing that extends our sense of agency out like this. We come, we become a temporally extended being in this way. It means that's why narrative matters to you.

Have you ever noticed how much narrative dependency you like? First of all, narrative seems natural to you, but it's not. It's not natural because we practice it all day long. All day long. You meet somebody. Tell me about yourself. Well, I'm about 5'10". I'm 180. About my eye color? No, that's not what people want to hear. They want to hear your story. You come home from work. You meet your beloved. How was your day? Well, my heart beat this many times, and I remember... No, no, now we got the bloody apps. We some might say that, but until very recently, with this is this is the story of my day, right? We're watching the news. Some horrible thing has happened, and human beings do this. I do it, too. And we and we don't, why did he do that? Why did he do it? And then we get the story, and we go, oh, nothing's changed. All those people are still dead. But we're we're relieved because we have the story. We're practicing this perspectival knowing and this person formation, and we do it.

I've raised children, and you when you raise children you realize how much you have to practice narrative with them. I have suffered the 10th circle of hell, which is the Teletubbies twice, where you watch this. Po says I'm going to the market. Tinky Winky says, "Can I come?" Po says, "Yes, we can go together." And you're and as I'm there, I'm kill me now. Kill me now. Kill me now. We have to dumb it down because it's actually really hard. It's really hard to get propositions to track perspective-taking and the creation of autobiography. Do you see that? It's very hard to do that. That's why. Have you ever asked a four-year-old to tell you a joke? Whoa. That's an acid trip.

Okay. Now, there's a fourth kind of knowing. There's four, right? Jung said, "Always give them four things because four is the number of reality." One thing, I don't believe it. Two, maybe. Three. Oh, yeah. Four. Oh, yeah. It's real. Okay. I'm going to give you a fourth fourth kind of knowing, but I'm going to hold off on it. Okay. So, I want to come back a little bit more about relevance realization. And I'm not going to get into the technicalities of the theory. Uh, I'll make a few references here and there for those of you who want to. I have lots of talks of this online. I have lots of publications. One just last year on naturalizing relevance realization. Okay. So, uh, I'm not hand-waving—well, I am hand-waving—but I'm asking you to trust me because the the the the technical rigorous argumentation is there. Okay.

What I want to talk about going back to relevance realization, you zeroing in on relevance realization is actually it's not cold calculation. Relevance realization: we listen to the l we pay attention; it costs us, and it involves something us being directed towards something we value. We pay attention. It's risky. We have precious coin, and we're considering this valuable and worthy of our attention. Relevance realization is caring about information. One and Reed Montigue said this a while ago in his really good book, *Your Brain is almost perfect*. The deep difference between us and computers, any kind of computational machine, is we care about the information we're processing. They don't because they don't have to. They don't have to. Okay? You do because you are—I'm going to use a multi-syllabic Greek word so that you can go home with that word and impress your friends. Okay? Autopoetic. It means self-making.

See, we're not just self-organizing like a tornado. A tornado is self-organizing, but it's not self-organizing to seek out the conditions that will preserve its existence. A tornado will go over land surfaces that destroy it. Compare that even to a a paramecium. It's a self-organizing system, but it's self-organized to seek out the conditions that will promote its existence and avoid those ones that will thwart its existence. It's autopoetic. It is constantly making itself from and in its environment. And you are too. Your body and your mind are constantly being made from and in your environment. You are embedded in your environment in that way that only living things can see because you're autopoetic. You have real needs because you are constantly taking care of yourself moment by moment. You have to care about this information rather than that information. That's what makes relevance realization real. Caring is grounded in real need. Real need is grounded in real embodiment and in being really alive and really wedded to an environment.

Many people in cognitive science, it's not just me, many people think that in order to be a truly cognitive entity, you have to be a biological entity for this kind of argument. Because unless you are an autopoetic thing, you don't really care. ChatGPT wouldn't care if you replaced all the meaningful text with a bunch of garbage random strings generated by an artificial grammar, and it ran the huge compression on them, and it would spit up spit out the probabilistic relations, and they'd be insightful within that—it doesn't care what it's processing. Okay, it can't care. You see, you are all doing—all living things are doing—niche construction. We're not just shaped by our environment. That was the old evolutionary view. We now know biologists have moved towards niche construction. Organisms are being shaped by their environment, but they're also shaping their environment. So you shape your environment that shapes you like this. That's niche construction. Niche construction is necessary for you to do relevance realization because things are relevant in terms of your niche. Wittgenstein famously said, "Even if a lion could talk, we wouldn't understand him because what a lion finds salient and relevant wouldn't make any sense to you at all."

Now, here's the thing. Culture culture is niche construction on methamphetamines. Look around you. Other than the atmosphere and your naked body, everything else is culture. You have, we have shaped this. Everything is being shaped. But notice what else is being shaped. We're being shaped. None of you are wearing clown suits. Why not? They could be warm. They could be comfortable. It could make you feel really great because you would be rejected by your peers if you showed up to this event. In fact, many of people were worried, "Am I dressed appropriately?" See, culture is niche construction on steroids. Niche construction and culture, biology, and cognition are all strongly wedded together.

The next thing about relevance realization, relevance realization is very much like biological adaptivity. In fact, one way of thinking about relevance realization is what you're doing when you zero on on in on relevance is you're doing kind of like what evolution does. You're running var—like I'll just give you a quick example. You have two attentional systems right now at work. Well, three—one, two—that are plugged into your salience network. One is your default mode network, and one is your task-focus network. Your task-focus network is trying to get you to keep your attention on the task of listening to this bizarre bizarre Canadian professor. And then you got your default mode network that's saying, "Let's mind wander. Let's think about other things. Let's let's go. Yeah, that was a great dinner I had. Oh, I wonder if Karen still likes me." And they're set up, and this is like biological evolution. The default mode network introduces variation—options that you could consider. The task-focused network kills most of them off, but not all of them. And so you make a couple of connections, and you move forward—is just like evolution: variation, selection, variation, selection—and they're doing that on your what you find salient. That's how it's working. Relevance realization is strongly analogous to biological evolution. Relevance realization is like biological adaptivity, only faster.

Now, here's the thing. Where is the biological adaptivity of the great white shark? Is it in the shark? No. Because I put the shark in the desert. It dies. Is it in the ocean? No. Because if I drop you in the ocean, you you're dead. See, adaptivity isn't in the organism, and it's not in the environment. It's not subjective or objective. It is about the fittedness between the organism and the environment in that niche constructive construction. Elements realization—I've coined a term for this. It's transjective. It's what binds the subjective and the objective together. So this means relevance realization, your cognition, and your biology are all about how you are coupled to the environment; coupled, dynamically participating in something together in this mutual co-shaping. And that is the fourth kind of knowing. That is participatory knowing. That is knowing not by having beliefs or skills or states of mind. It's knowing by how you and the environment are co-shaping each other. And you're both participating in this process. So you belong together. You get a sense of belonging from this. You don't get a sense of presence. You don't get a sense of power. You don't get a sense of truth. You get a sense of belonging. You belong. And it has its own weird form of memory. You call it yourself. It's a weird form of memory. So you get your sense of self from your perspectival knowing, your episodic memory, and your participatory knowing and this deeper kind of memory. This is the knowing that gives you a sense of belonging, so you shape your identity in the right way.

And what are you talking about shaping your identity? You sound like some French philosopher. You're doing it whenever you do whenever you realize what role you should be taking. Role is a term for how you shape your agency to fit into the situation as you find it relevant as you're sizing it up. What's your role right now? Is it the same role you have when you're parenting your child, when you're with your lover, when you're with your friend? What binds them all together? What holds them together? Well, now you're starting to talk some philosophical language. All right. So, now again, do the LLMs have relevance realization? Well, first of all, they don't have the non-propositional kinds of knowing. For all the relevance realization that you're doing in procedural, perspectival, and participatory knowing, they don't have that, and Altman knows that. He says they they don't have consciousness; they don't have agency; he admits that, and they don't have some kinds of intelligence. He's actually said, "I think that's right; I don't think that's terribly controversial to say they do a little bit of relevance realization." I published on this way back in 2012 on how how Jeff's—sorry, Jeffrey Hinton's—deep learning algorithm was doing one kind of uh process that was one dimension of relevance realization, and the LLMs are doing this. I won't get into the detail of this, but the most of the dimensions of relevance realization are missing, and that's for the deep reasons: they're not auto poet—they're not autopoetic; they're not embodied; they're not seeking out the information that they need in order to maintain their existence. So nothing is actually relevant to them. They do things that is are relevant to you. Okay? They do not participate in biological, cognitive, or cultural niche construction at all. Okay.

What's my evidence for talking about that? We we compile the data sets that they're trained on. They can't generate them. If you get them to generate data sets and then train LLMs on them, their their performance reliably degrades. Our selective attention, our caring structures social media, our selective attention, our caring structures the internet and the world of academic publication. All of that is because of us. Human judgments of relevance, appropriateness, fittedness—this is all the language that is used—are integral to actually training these machines. The human reinforcement deepseek is so much better than ChatGPT. Yes. Two—I mean, there's a lot of technical reasons—two core reasons: it basically put in something like a CPU to make it more efficient in its internal communication, and they did more refined and in-depth human reinforcement on reasoning. Sam Altman has asked us to try and come up with better texts that exhibit powerful and good human reasoning. Remember that. Please remember that. Why does he want those to train on? Okay. This means there's something plausibly different about our intelligence and the intelligence that is in ChatGPT. It doesn't mean they can't do a lot of things in a lot of dimensions, and they're going to get better at it, and they're already getting better at it. So if your identity is wrapped up with the technical manipulation of propositional information, you are under existential threat, and there's no way that's going away as a problem. Well, ecological disaster or something like that. And do you want that? Right. We have to pivot away from what we normally reward people for. I'm rewarded for it, too. I generate propositions, and I am good at the technical manipulation of them. That's not all I do. But the degree to which we have identified with that one dimension, that one kind of knowing is the degree to which we have now imprisoned our sense of humanity and our sense of personal identity to a very disastrous competition that we are inevitably going to lose.

What do we have? We have real-world foresight, insight, and care, especially within non-propositional, embodied, embedded, and inculturated kinds of knowing. That's what we have. And we have to turn more and more to that as—and I I'll make a prediction. Scientists are supposed to make a prediction. I'm a cognitive scientist. I make the prediction that—and it's already happening, by the way, but I made this prediction before it started to happen. Sorry. I'll stop ego-massaging myself right now. Okay. People are going to increasingly turn to those non-propositional kinds of knowing and their em and the embodied, embedded, inculturated aspects of their being in order to locate what it is that makes them a human being and a particular person. They're going to reach into the sematic dimensions of their being, their embodiment, the ineffable aspects of that. They're going to explore more and more altered states of consciousness. They're going to talk more and more about the ineffable experiences that we can have within psychedelic and mystical experience and how it orients us to something ultimate because the ultimate isn't going to be erased by the LLMs. Ultimate reality is going to just keep chugging along on its own. Okay. Okay.

What about AGI and reason? Okay. Well, here we must be very careful to make a needed distinction. There's a distinction between reasoning in the sense of making inferences, creating argumentative structures out of propositions that come to conclusions, almost always linguistic arguments. How do you carry on an argument without language? And almost always involving literacy. We use notation, mark things down, do logic, etc. If you mean that these machines are rapidly approaching our competence and will probably exceed it. Okay, so if again, if that's where you're locating who you are, but is this the same thing as being rational? Well, doesn't rational just mean being logical? Actually, no, it doesn't. There's been a lot of research. This is one of my areas, and I'll just summarize this research. Rational isn't about being logical. It's about knowing where, when, and to what degree you should be logical. That's a different thing because you—unlike Spock and Data—you can't be logical all the time because if you're being logical, you're pursuing deductive closure, which means you have to make sure that you're avoiding all inconsistencies and all right, you're checking out all possible implications, and then you're into a combinatorially explosive search. You can't do it. If you try to be purely logical, you have just committed a complete act of cognitive suicide. Okay.

What's actually the core of rationality? The core of rationality is the ability to systematically and reliably overcome self-deception. This is all the ways in which your relevance realization inevitably biases you. Look, the very processes that make us so adaptive make us prone to self-deception. What relevance realization is doing is this, right? It's framing your cognition, what you're looking at. And so, I'm ignoring all of this. The problem is sometimes that's where the truth is. You probably heard about biases like the confirmation bias. How many of you have heard of the confirmation bias? This is that when you tend to look for evidence, you tend to look for evidence that confirms your belief. You're oblivious to evidence that could disconfirm it. That's a form of self-deception. Science is designed, by the way, to try and overcome the confirmation bias. There are a core. There's just a ton of these biases, and they're all related together. Now, here's the thing. Because they work off of like you don't want to shut down the bias. What I'll do is I'll eradicate bias from my mind. Great. Now, what you have to do in this room is pay attention to everything everywhere all at once. Go. You can't do it. See, this is what's tricky about rationality. You have to ameliorate the bias without undermining your adaptivity. That's really tricky. That's really tricky. So, you have to care about how you're caring about things. That's what it is to be rational. The distinguishing difference between rational and irrational people is—irrational people only care about the product of their cognition. They don't pay attention to how they're paying attention. They don't care about their how how they're caring. Just I believe this. Rational people care about the process, not just the product, reliably. That's a reliable, experimentally validated distinguishing feature between rational and irrational people. Rational people care about—they actively care about—the possibility of self-deception, not abstractly but in practice. In order to be rational, you have to care about overcoming self-deception and commit yourself to self-correction. This is crucial to being rational. That means you are actually aligning—and there's a magic word—the alignment problem. You're aligning yourself with normative standards. You are trying to reduce your self-deception because you're trying to make your beliefs more true, your actions more good, and your experience more beautiful. So, rationality requires self-reflection. It requires you being able to take a perspective on your own perspective-taking. It requires higher-order awareness, and it requires a really finely tuned set of skills for intervening without destroying. It requires caring about normative standards within self-correction. It requires you aspiring to be better than you are. This is central to being rational.

Now, is being rational the same as being reasonable? We have a standard in most legal codes. You're expected to act as a reasonable person would. You ever heard of that? This was banged out over like millennia. What is it? What are we What are we talking about? Well, why didn't you save that child that had just crashed on the sidewalk and was bleeding out while I was on my phone? We don't go, "Oh, okay." We go, "No, no, no, no, no. Look, you you're not sizing up the situ—" Listen to my language. "You're not sizing up the situation. You're not situationally aware. You're not sizing it up. You're not zeroing in on what's most relevant, what matters. You're not caring about it in the right way. And you're not assuming the right role." Do you hear all the non-propositional kinds of knowing in there? They're all through that. That's what we're talking about when we're talking about being reasonable. It involves perspectival knowing. It involves participatory knowing. See, being reasonable is deeper than being rational. Because in order to be rational, you have to be able to orient yourself. You have to be able to shape your agency and to identify the situation and the shared norms that matter. For example, I can be completely logical, and I'm still doing it all within an egocentric perspective that is massively screwing me up. So being reasonable is deeper than being rational. And being rational is deeper than just making inferences. I've shown you that. Now take that argument and plug into it some of the most reliable data we have in psychology. It's not going through a replic replication crisis. It's massive experimental work. Keith Stanovich and other people, by the way, at the University of Toronto showing this. We have—I told you already—we have a general man measure for intelligence. It's called G. What Stanovich showed is we can give people a bunch of tasks for reasoning and rationality, and they all are strongly mutually predictive of each other. He calls this the R quotient—like IQ, your intelligence quotient. So what we can do is we can measure your IQ, and we can measure your R, and then we can see how correlated they are with each other, and correlation goes from zero to one. Right? The correlation between general intelligence and rationality is .3. You can be very intelligent and very irrational. You can be very intelligent and very unreasonable, and you can be very intelligent and very foolish. There is nothing contradictory about this at all. You've probably met these people. I fear that the people that are running some of the open AI... Okay.

What's unique about you is you have to live according at many different... Another thing that's unique about you is you have to work at many different temporal scales: the immediate, short-term, long-term. You have to work in different environments. So, you have different centers from which you're motivated. There's a part of you—Plato talked about this—it's like a monster, and it makes you want things right now, right? Chocolate cake. Now, then...

There's a part of you; it's like a lion. It's in your chest, and it makes you—like it's—it's not working in terms of pain and pleasure and immediacy. It's working in terms of the social emotions that you feel here, like pride and guilt and shame and honor.

And so if you want to lose weight, you join a group. By the way, changing your behavior, a reliable predictor, join a group. Okay, the lion. And then on top of it, up here is the man, the person that's capable of self-reflective self-correction.

Now, the thing about you is those are always—sorry—those are frequently—in each one is adaptive. You need each one. You need each one, but they conflict with each other. I should lose—John. Here's person there. Lose weight, John, other people. Yeah, John, you'll you'll look better if you lose weight. The doctor said so. And there's a chocolate cake on the counter right there. Chocolate cake. Chocolate cake. And so you eat it. We all do it. That's why we procrastinate. Almost done. That's why we procrastinate.

So we have to align all of these centers. We have to coordinate them. Coordinating all of these centers, coordinating all these kinds of knowing so that we reliably do what is best oriented—what's to—what's true, good, and beautiful. That's what wisdom is. That's what wisdom is.

And here's the issue I want to get to. This is why Alter is relevant. See, we are all naturally intelligent. You don't—in order to be intelligent, you just have to not undergo trauma. But if you're brought up—good nutrition, non-trauma—your intelligence will display itself. That means we have a template we can use to train intelligence for these machines.

But here's the thing I've argued: We are not naturally rational. We are not naturally reasonable. We are not naturally wise. Making us intelligent, making anything more intelligent doesn't make it reasonable, rational, or wise. If we want these machines to align with us, don't—we can't program them to align with us. They're adaptive. They'll overcome it. What do we have to do? We have to get them to care about what is true and good and beautiful. We have to get them to engage in self-correction. All these machines are speeding up. Yes. But what's not improving? The way they hallucinate, the way they confabulate, the way they lie. That's not getting any better.

If we're going to make these machines reasonably rational and potentially wise, we have to provide them with the role model, with the data sets, with the text to do that. Do you understand what I'm saying? We are under a terrific social obligation—to all of us—for all of us to cultivate more rationality and more wisdom. Not only so we can deal with these machines, but so these machines will be steered in the direction of being silicon sages rather than mechanical monsters. We are irreovable from that equation.

And this is not something you get by consuming propositions. It requires practice. It requires practice. This is what Alter is all about. It is an app, right? It is an app that is trying to give you practices. It gives you role models in the great philosophers, not propositional theory, but people who were actually lovers of wisdom. Because this is—this is—this is essential. We need the wisdom to confront these machines, and we need the wisdom to properly mentor them if we want them to have—if we want to have any chance of them being aligned with us in the future—of working towards something that is to our betterment and not to our detriment.

So, I got involved with Alter not because it's cool—because it is—not because it's beautiful—because it is—not because I love the people I'm working with, which I do. After giving you this overly long talk, hopefully, this doesn't sound hyperbolic. I got involved with Alter because I believed—I believe—and I've argued that it's essential for going forward in the AI future. Thank you very much for your time and attention.