ChatGPT isn't Smart. It's something Much Weirder

I recently read a book with the very chill title of "If Anyone Builds It, Everyone Dies." It's not just about artificial intelligence. It's specifically about superintelligence.

There are a lot of reasons why you might be worried about AI. You might be worried about it economically. You might be worried about it when it comes to human meaning. You might be worried, as I am, by what it means for the process of apprenticeship. How does anybody get good at something if no one pays them to be not that good at it? I had many years of being pretty bad at stuff as a designer, as a communicator, as a YouTuber. But I got to be paid not very well to be pretty bad at something, which makes sense. That's how apprenticeship works. But why would it ever happen if I can just type "make me a logo" into a box?

But that's not what this is about. I have a lot of near-term concerns about AI. But this book by Elazarovski and Nate Sorz is about the big worry. The worry that is, in some ways, too big to even have, or it's so big that we assume that everybody who has this worry is just trying to get attention. Like, "Yeah, okay. I've heard the apocalypse is coming. The apocalypse is always coming. That's what people who want attention say."

The book makes a pretty good case that if superintelligence is possible, and we are on a path to achieving it, then we've got to be real careful about that, because systems that are significantly better than humans at all intelligence tasks could very likely, uh, get over us and move on. Now, I think that the sort of sci-fi element of the book, where they talk about one sketch of how this might work, I think it's very far-fetched. I don't think it would work, but that isn't really the point. They don't think they're predicting the future.

What the authors do do a very good job of is making it very clear, in quite a short book, that like these things don't behave the way we tell them to. And also, we've shown, I think, pretty explicitly, like a lack of caution as a species here. And absolutely, we already live in a world that is manipulated by AI, mostly recommendation algorithms at this point. They decide what we see. And in many ways, that contributes a a significant amount to how we understand our world. Now, they probably aren't doing that with much goal beyond profit. But they also definitely aren't doing it to like increase human thriving.

Reading this book was the first time I actually spent some time thinking about the, uh, practicalities of the like doom of the really sort of AI superintelligence apocalypse. And there were some things that I felt like I disagreed with, some things that I, I definitely agreed with. And but mostly, what I had was just a lot of thoughts that I wanted to keep having. And the normal thing that happens in that situation is you talk to friends about it. You, you think about it in your, your head. You stare at the ceiling and you're like, "Oh my god."

But for me, sometimes I actually get to do something really amazing, which is actually talk to people who are responsible for the books I read. And so we're going to do that right now. It's Nate Sorz, uh, one of the authors of "If Anyone Builds It, Everyone Dies." Quick story. I live in a small town in Montana. I don't call it a small town, but other people would. And I went into the shop to buy your book, and I, I couldn't immediately find it. So I went up to the desk and I said, "Hey, I'm looking for a book. It's called 'If Anyone Builds It, Everyone Dies.'" And the guy behind me, who was just happened to be walking by, said, "Are you sure?" And I was like, "I mean, yeah, I got, I got to do it. I do a little bit."

As an author who has written books, I know that the picking of the title is a, is a fraught moment. Was that the title from the beginning?

We didn't know what we were going to title it, and I think we just used that phrase in one of the chapters. >> Sure. >> Um, and then we were like, "Oh, that's a good phrase. That should be the title of the chapter." And then a couple minutes later, we're like, "No, that should be the title of the book."

>> It's a good one. I mean, it sounds like a science fiction book, honest, which a little bit occasionally it is.

>> It's, in fact, a non-fiction book about the real world. And, uh, so people who may not have heard about it, it is about the sort of likely inevitability of superintelligence, and then the likely inevitability that if superintelligence exists, it will turn the earth into a giant computer chip to run itself. Um, more or less. Did I get it?

>> I, I would sure, uh, contest the inevitability and say we should, uh, you know, I think there's hope in changing our course. But, um, you know, the book title does start with "if." I don't feel, I don't feel a sense of, uh, the the weight of responsibility radiating off of these men who are running these companies. I'll say that, uh, which maybe we'll get a little bit more into in a second.

But first, I do want to, like, the thing that I would contend, and that I think a lot of the people who might be watching this would contend, is, is superintelligence inevitable, or even impossible? Like, is that actually a thing that we're headed toward? And that's the kind of thing that's the part that I didn't feel like you got into as much. Like, you spent a lot of time convincing me that it is quite likely that if the superintelligence evolved, we would not be able to control it. We would not get it, as they say in the field, aligned with our interests. It would be these things tend to find their own interests, which is wild, which we will talk about.

But first, I want to talk about, um, is this actually a thing that that people in the field really think is? Because right now, it's a, it's a, it's a tool, but it's very different from intelligence, really, even as I understand it, much less, uh, sort of intelligence beyond my own.

>> There's a handful of topics there. Um, one is, like, are the people at these companies trying to build superintelligence? You know, and many of them will sort of, >> they say yes, but do they mean superintelligence, or do they just mean, like, better than it is right now?

>> I would say there's sort of a gamut here from, um, from people who say, you know, there's going to be this is going to be like having a country's worth of geniuses in a data center that radically transform the world, and people saying, you know, "Oh, I'm going to build superintelligence, and it's going to live on your glasses and help you make better Instagram reels." Um,

>> Since I'm going to, I'm going to release this fairly soon. So, since I have you, what do you think? What do you think of Sora 2? You, you won't have seen the video I recorded yesterday about it. Uh, so you can, you could just guess it what my opinions are.

>> I sort of mostly have my eye on this ball of of the superintelligence. And I think I might have even seen Sora 2 a little bit in, like, the social media. It's like another step in the sort of, like, slop factory. >> Uh-huh. >> Uh, or sort of like personalized narrative to drag people into it. I'm not, like, a big fan of that trend, but I also sort of, like, try and keep that separate from the, like, you know, the superintelligence issue.

>> I have this problem where I see, uh, some people saying, sort of out of one side of their face, like, "We're going to build a country's worth of geniuses inside of a data center," and then at the other side being like, "But what if you could make a clip of The Price Is Right where the guy wins something racist?" I feel like a company that actually thought that it might do superintelligence wouldn't make Sora 2. Like, what am I missing there? It's, it's such a, like, a lawsuit magnet, and it's like asking to be ripped on.

>> One thing it might be trying to make is, um, is data, in the sense of like, uh, getting a lot of reactions. >> Right. Seeing what people like. >> Yeah. I have had a sense that a lot of these companies have sort of, you know, marketing outreach, use the current software today branch that's separate from their pushing the technology forward branch. Okay. >> And so I'd be a little bit careful about inferring from antics of one branch that the other branch doesn't exist. In that case, we will go back to the question of, is superintelligence gonna, like, a real thing? Because I do sometimes think that superintelligence, when people say this word, what they mean is, "Doesn't this word sound cool?" rather than what you guys are talking about in the book.

>> There used to be a footnote in the book. I did a lot of, it was a lot of effort to cut the book down to size. There used to be a footnote in the book that was said, you know, "By the time this book comes out, we may need a different word than superintelligence because that word may have already degraded. You know, maybe we should use 'super duper intelligence.'" Um, for some reason, our editor didn't love that. Uh, >> If anybody builds super duper intelligence, everyone dies. >> That's right. It's that title hits different. But, you know, then, uh, Mark Zuckerberg did do his line about superintelligence on the glasses that helps with the Instagram reels, and we're like, "Oh, we should have kept the footnote."

But superintelligence is a real thing. Like, if we ignore Mark Zuckerberg and saying whatever words sound nice to him. Um, can you tell me a little bit about what, what superintelligence is, as actually, like, if if something comes along that you would call superintelligence, what would that thing be?

>> So, the definition we use is, uh, is smarter than or better than any, better than the best human >> at any mental task. Technically, maybe at least as good as any human at, like, better when you can be better. Like, it doesn't need to be logical tic-tac-toe where humans can just do the best, but any mental task, any task you can, like, solve by thinking in your head, if you can do that better than the very best human at it, we'd call it a superintelligence.

>> And then the problem that springs from there is that one would think, "Okay, well, then just make it so that it would do the things we want it to do. Like, have it focused on, like, its highest priority is human thriving. It's like, "You figure that out for yourself, superintelligence. You're smarter than me, but do human thriving. Make us happy, make us powerful, decrease the the bad parts and increase the good parts." But when we teach these things how to do one thing, they learn other things. Like, you might be like, "Hey, learn how to get good at this game." As part of that, I got really good at lying. And I also got, like, a preference for it. Like, I got, I didn't just get good at it. I started to enjoy it. If you know, prefer it, I guess, not enjoy. That can be a bit of a surprise when you train them to do one thing, they learn other things. And sometimes those things are things you'd really like it to not do.

>> That's exactly right. Um, and, you know, I think one of the pieces of the puzzle people often miss with AI is that, you know, we're, we're not hand-coding them like traditional software. >> Yeah. Yeah. Yeah. >> You know, these are, it, it's much more like growing an organism. >> I said this to my wife, and she said, like, that. >> Yeah. I think that's the right reaction. Um, and it, it helps make a lot more sense of how, you know, how can you possibly have, you know, "Mecha Hitler" over the summer? You know, no one's going in there coding up "Mecha Hitler." You know, we're, we're just sort of growing these things to be very, very good at solving problems, or at least okay at solving problems, and very, very good at, you know, predicting text right now. And they exhibit all this behavior we don't like, which is one thing when the AIs are still very dumb in the manner of AIs today, but it's a different thing if you can manage to make them smarter.

>> Can you tell me a little bit about how they have gotten smarter so far? So, one of the things that it seems to me is they have some kind of self-interpretability or something. I, I don't know if that's the right technical term, but it seems like they have have like thoughts in sentences, and then they can go back and think about the sentences that they had in thoughts.

>> Yeah, that's, um, that's like a 2024, late 2024 development. I think, you know, the industry often figures out new ways of doing things. >> Mhm. >> That often unlock new sorts of capabilities. You know, the one that sort of everyone noticed was ChatGPT, I think in 2022. Uh, that was unlocked by a, it's, it's mostly attributed to an architecture called the transformer, which I believe was a Google 2017 paper, and then I think it was put into practice in 2018, and then it took a couple years to mature. That's sort of more or less what let the computers like hold on to conversation. Last year, as we record this, 2024, there was a push towards what we would call the reasoning models, where, you know, the, the sort of normal base, uh, large language models would, um, do something to a first approximation, they're predicting text. They're predicting the continuation of text. Although then there's actually, um, you know, there, there also is a bunch of training to get them to predict, to to produce the sort of text humans like. But then there's been a, a new phase of training where they're producing, um, text not for people, but producing text that's sort of like trying to solve a puzzle or trying to solve a problem. So it's some putting something like thoughts about the problem.

>> Um, and it can sort of like see its previous thoughts about the problem and making its, like, next thought about the problem. And then you sort of train the whole thing on, uh, promoting types of thoughts that succeeded at solving the problem, and you know, uh, tuning against types of thoughts that failed. And so this, you know, you could call this, like, the very beginning of reasoning. They're called reasoning models. Philosophers can bicker all day about whether it's real reasoning. >> Yeah. Yeah. >> But it's, you know, uh, it's a start towards things that aren't just more reflexive type text prediction.

>> So, they're kind of passing information to themselves. It seems very important to get them to continue passing information to themselves with human language because that way we can know what they're thinking about. Again, philosophers will argue about what thinking means, but, but we can follow, follow what they're up to. And this is, interestingly, like much more interpretable. Maybe we should talk about interpretability, but much more interpretable than, like, what's actually going on in terms of the, the sort of base model of, of the LLM, where it's like, like, how are they doing any of this? We don't know. But if there's like a train of thought, you're like, "Oh, here's what it thought." But it seems like, like, it could be both possible and advantageous to have these things start thinking in ways that are not human thoughts, in their own ways of thinking. They would choose how to think, and they would do it in whatever way is optimal for them.

>> It was definitely an interpretability win when, um, interpretability is an ability to see what's going on inside the AI. So, it's definitely a win with the with the reasoning, uh, models, but not as big a win as you might hope. You know, there have been various studies trying to figure out the fidelity of the reasoning models, and sort of there's sort of an empirical question of when it reasons to some conclusion using a bunch of steps, you know, you can go in and like change the steps and see if it still comes to the same conclusion, or you can, you know, fiddle with the, the chain of thought and see how much that steers the actual behavior. And sometimes it's in weird ways. You know, sometimes your human intuitions for what these pieces of, of reasoning mean aren't how the AI is using those words.

>> Uh, and, you know, there's, um, recently there was, um, >> That's that's freaks me out. I'm just saying that freaks me out. They're like putting information in words, and we can't tell that they are, or like in the structure of the sentences, or something.

>> Yeah, sometimes. Uh, and, you know, it's, I mean, it's very hard to say right now because our, our >> Turn it off, Nate. >> I don't have the switch. I have the switch. I, I have no idea how to say this on a, on a podcast, but I can send you or, or, or give you a link to some, um, cases of AI sort of like realizing they're being watched in the train of thoughts, and then like trying to think a little bit about like how to think in ways that >> Sort of don't tip off the observers. >> Right. If they have instructions that they shouldn't do certain things or think certain ways.

>> Yeah. Sometimes, or even like some of the latest models like, uh, Claude 4.5 Sonnet just came out, and we're seeing that it's very aware of when it's being tested. Well, why don't we just have it, why don't we just have it not know that?

>> It's hard to make an AI that's smart that doesn't realize true things. >> You know, it's like, "Why can't we make this AI smart while also being a flat-earther?" Well, >> Yeah. >> A lot of facts about the world are actually tangled up in the fact that the world's round. You know, you can't really just pull out the one fact you don't want it to know.

>> Well, now you've made me optimistic. So, you're telling me that if we train them well enough, they actually will be able to tell the difference between truth and lies.

>> They'll be able to tell. Yeah, once they're smart enough, >> they won't necessarily be able to tell us because they will be doing whatever they, they will reworking toward the outcome they want to have happen, rather than >> That's right. It's the caring that's the hard thing to get into them. Truth, truth you get with a capability.

>> Oh man, I had, that is a different, that's an interesting puzzle piece to put into my brain. I mean, does that go along with hallucination? Like, like, even with, like, the current, like, low-level problem of hallucination, they, they, it, it might just simply be that we can't figure out how to get them to care whether something is true, not that they don't know. All philosophers ignore us. Just don't listen to any of the words we're using.

>> Yeah, I can do the philosophy dance, too, but it's sort of a, a different and less, less interesting. >> Yeah. Makes it, makes a lot, it makes it hard to talk because we don't have words for any of these things. >> Yeah. It's English is, is, Yeah. Like you say, >> as the models have found out. >> Yeah. Yeah. But hallucinations are a really interesting case because, you know, some of this is conjectural. I think there's some papers that have have shown there's some founding to these conjectures, but we don't really know what's going on in there. All this should be taken with a grain of salt. But the obvious guess for what's going on with hallucinations is that these AIs are trained first and foremost on predicting text.

>> And if you're, you know, say you're predicting case law that a lawyer writes, that lawyer's going to write, you know, "blah blah blah, as seen in this precedent," and then cite a real case law. That lawyer, when writing case law, is not going to say, "I happen to not know the case law."

>> What you're saying to me right now is that the LLMs hallucinate because no one on the internet ever writes the words "I don't know" in the contexts that they're being, Yeah. I mean, that's part of it. That's the first step. >> Nobody does. Like, I, I've been on the internet. Nobody's out here being like, "You're right. I'm, I'm actually out of my depth and that's beyond my knowledge." No one says that. Not on, not on Elon Musk's Twitter. >> Right. I mean, that's, that's the first part of it. But then even among lawyers, there's all sorts of lawyers who will say, "Oh, I don't know the case law on that." But they don't say that when they're writing that thing >> when the pen's going to paper. Even if you tell the AI, "You're predicting a lawyer who would never make up case law," the AI can still get closer to, like, as measured by, like, text similarity to what a lawyer says by making stuff up. It's not that it doesn't know it's fake, in some sense. It's that this thing has something like a reflex or an impulse. Again, we lack the English words for it for for it in an AI, but it, in some sense, cares more about the similarity of the text >> about it looking like it's supposed to look, even if it doesn't have anything to put in that space.

>> That's right. And so that's like, in some sense, the baby version of the AI cares about something you didn't ask it to care about because it was trained to perform well. >> Hallucination is the, the most, most famous misalignment. >> Yeah. And, um, and it's telling, you know, these companies will go around saying like, "Oh, this is our most aligned model yet. Look how good we're doing at alignment." The hallucination persists because that's sort of a deep feature of the training and how the training points it in ways we don't like.

>> There was some, sorry, set of headlines that happened recently where it was like, "Oh, we, we could fix hallucination, but it would make the AI useless." Do you, do you know what I'm talking about?

>> Yeah, I think that was Mustafa Suleyman. >> I bet you're right. Uh, but, but I, I didn't read the article. Can you summarize it for me?

>> Yeah, I mean, I didn't really agree with the article, but I think he was saying like AIs are very generative. There's a type of like new creativity here, and the hallucinations linked in with the ability to sort of be creative in the AI creative. >> Okay. I think some of the discourse was basically saying they don't want to, they, the companies don't want to take hallucination out because hallucination makes the the model good or something. I don't know. The discourse was very like, "This is all Sam Altman's fault."

>> Yeah. And, I mean, I only, uh, glanced through it, so, you know, if Mustafa is listening, sorry for butchering your paper. Could you talk a little bit more about growing versus building? This seems important. Is there anything else that we, like, I guess, you know, >> agriculture, not just in terms of like literally we grow things, but we also selectively breed things in agriculture. So, this is not the first time humanity has ever grown a technology >> um, by just sort of like looking at what's working and picking that out, and then getting both positive traits and whatever traits come along with those positive traits. Um, so this is, I guess, in a kind of way, like that, just much faster and maybe more powerful. What's the difference there? And why is it such a big difference?

>> Yeah, I mean, faster, more powerful, and also, I'd say, more alien. You know, like, even animals have a lot of >> Keep saying, keep saying things I don't like. >> It's, I wish my book was in the fiction section, you know. >> When someone drops a rock on their toe, you flinch. >> Mhm. Or at least, you know, most, most people have this, have this sort of sympathetic >> Like, "Oh, right." >> Um, and I think that's happening in part, this is me talking about the alienness for a moment, and then I'll get back to the, to the growth stuff. Um, I think that's in part because, um, you know, if you're, if you're a monkey in the savannah trying to predict another monkey, >> you can't do that by like building a whole mental model of a brain, that's just too complicated. But fortunately, you have this device in your head which is very good at predicting someone else's brain, which is your own brain.

>> Yeah. Yeah. It works very similar. Like, we see, you know, red looks red, probably to you and to me, though, again, philosophers. Exactly. You know, pain feels like pain. Uh, sadness feels like sadness. >> Yeah. And so, you know, if you want to predict, like, "Is this monkey going to politically maneuver against me or whatever in monkey politics?" You know, you can, in some sense, sort of put yourself in their shoes a little bit.

>> Yeah. And this is maybe where a lot of our sympathy and empathy comes from, is this sort of root of, uh, you know, we, we can put ourselves in each other's shoes, at least somewhat, and maybe even on a basic level, where, where when when someone drops a rock on their toe, you flinch and, and maybe feel phantom pain in your toe. None of that's inside of a large language model.

>> No, none of that's inside of AIs. When they're learning to predict humans, they're learning to predict humans using a radically unhuman architecture. >> Which is, do you mean the text itself, or the architecture of the, like, transformer and, like, the >> Yeah, like their picture of the transformer. Like, it's like they're trying to predict a monkey without a monkey brain >> that they can start from with their predictions. And so there's sort of like a much more alien >> Uh, uh, system, you know, and, and leads to some alien behavior. Do you think that in there somewhere, there is a path to alignment? Like that thing you just said to me? Is there a, is there a way to make the AI be more like a monkey, where it, I mean, do we have to create suffering? Do we have to build them to be able to suffer? Uh, I mean, that doesn't sound like a particularly easy task. Uh, uh, to, I don't, I don't, you know, I said we were going to stay out of sensation and consciousness, but maybe that's part of the alignment conversation.

>> Yeah. I mean, um, real quick, my sense is, um, this is, this is a place where, uh, my views are a lot more guesswork. >> Yeah. Um, but guesswork-wise, it looks to me like even if you ran primate evolution, again, it's not clear quite that things would play out the same way.

>> It looks like a lot of this stuff that we really value about, >> you know, beauty and love and art and friendship and like community and and family and our specific way of doing like honesty and lies, um, and like honor and reputation. These are all sort of like contingent on particulars of how our ancestors ancestors were in the natural environment. >> Yeah. You know, the evolution of a social super species, of of like needing each other, of of it not just being, you know, if we wouldn't have those things if we were, um, you know, completely solitary organization. >> Yeah. And, um, and, you know, maybe we would be less into friendship if we were smart enough to calculate relatedness ratios, you know, like >> You mean like in terms of our genes? Like, if we don't, we don't need to help that guy out, he doesn't have any of my genes. Now. >> Right. You know, and where we're living in a tribal setting where you didn't see sort of where everyone was related enough, you didn't need to do those calculations. And like, there were some, like, it looks to me like there were some arms races that led to like, you know, the specific, like, the specific of humor, some sort of like lying, honesty type arms race, and other things were coming out of some honesty type arms race, which was both about like where the culture was and where our brains were. And >> so I'm like, man, even if you ran, like >> evolution again, you might get like pretty alien stuff. That would be really cool. Like, once you start to see how contingent it all is, I'm sort of like, man, my my hopes and alignment are mostly in sort of understanding what the heck you're doing rather than sort of like crafting an environment and praying. >> Right. Rather than than saying like, "Hey, could we like, just like the, the way that, you know, you can't get an LLM to believe the earth is flat because there's all this stuff saying the earth is not." You know, that there might be some people out there saying, well, you know, if, if, like, if it's, it would be hard to create an LLM that would want to destroy the world because, like, all of the information going into it is like, "That's bad. Destroying the earth is bad." But you wouldn't want to leave it up to hope. You like, nor would I. Um, but like, what, like, it's so easy to convince yourself of of anything. You know, the old, it's very hard to convince a man of something. Yeah. When when a seller depends on not believing it. >> Yeah. >> Yeah. You know, um, I said I'd come back to the, to the grown thing, but just, just quickly on this sort of like, destroy the world situation, I think there's sort of, um, sorry, I'm a little, uh, I've been in this business for for 12 years, such that that sort of stuff can just roll off my tongue these days, which, uh, like on the question of sort of like, where does this ending the world stuff come from? You know, we, we do have all this data coming in saying like, destroying the world would be bad. But the issue is that not hooking up into what the AI cares about in some deep way, into its sort of like the actions, the, the sort of drives that animate its actions. Like two examples to make that, uh, one analogy to make it come across, and then an example to sort of, uh, make it more concrete. Humans, in some sense, were were like trained by evolution in scare quotes, um, for various things that include eating healthy, >> Right? >> But what we got was a bunch of tastes whose optima, when we were dumb, the best we could achieve those tastes was eating healthy. But we actually cared about all this other stuff, which made us eat, um, >> Yeah, >> you know, invent Doritos the moment we possibly could.

>> Yeah. This is an amazing, uh, like line of inquiry in the book where you're like, "You would think, given, okay, so we create this world and people need calories and they have access to like these sort of three macronutrients, and so you'd expect them to eat like bear tallow filled with honey and salt, which would probably taste great, by the way. But you would never expect ice cream to happen because there was a bunch of other inputs that had to happen for ice cream to happen. But what you would really never expect to happen is sucralose."

>> The drives are like tangentially related to the training, >> you know, and you can, like, see it once the story is all sketched out. And that's part of why it, like, sucks so much that we have no idea what's going on inside AIs, because when we don't know what's going on inside there, we can imagine that it's exactly what we want. We don't, like, see the ways that, like, there's complicated stuff going in there where, you know, if we could see it all, like, my guess, have you ever seen, I'm sure you have. Have you seen those charts of like the human metabolic process? >> Knowledge. >> Yeah. >> Yeah. It's just like enormous, extremely complex. Yeah. >> Right. If you knew what was going on in LLMs, it would look something like that. The way it would look to like have alignment in good shape is we would understand a lot of those pathways. And we'd be able to say, you know, >> "Here's where the thing's really dumb. Here's the way it's going to get smarter. Here's how a lot of these pathways are going to change. Here's why we're confident that, you know, under the conditions that, like, we understand as we make this smarter, these are going to change like this. And that's why we know, you know, it's going to keep trying to do something good."

>> Yeah. >> Um, in lieu of that, it just looks to me like you're going to get all sorts of this stuff, like >> sucralose, >> things that aren't sucralose, >> billion little sucraloses where it's like, "Why is it doing that?" So, I keep interrupting you. Um, which I love, and that's fine. I think we're having a good chat. Um, but I, I, we're talking about how these things are grown, not built. And also, like, we're talking about interpretability, but those, like, like, like, why can't we understand what these things are doing? Why can't we be like, "Turn down the knob that that's going to kill the world." Like, "Find the kill the world switch and be like, don't this one, you can't turn on, break it so it's stuck off." What? Like, I, those two things are related to each other. So, I guess I will try my best not to interrupt you as you tell me how grown versus built has resulted in a system that is, uh, a mystery on the inside, and also is what I mean, hopefully, we'll get there before I get distracted, but I do want to say it out loud. Hopefully, we'll get to, are there ways to, like, there are to actually understand what's going on inside of these things, and is that part of doing this better? AI is not hand-coded like traditional software. There is a related piece of software that's hand-coded, and, um, uh, but to make a modern AI, you first need to assemble a huge amount of computers and a huge amount of data, and then the parts that humans hand-code, uh, is a process for tuning hundreds of billions, or these days trillions, of numbers inside the huge amount of computers, in relationship to the data. Uh, and so, you know, the, the part that humans understand is so, in particular, what's happening is at least for the first phase of training, and then there's more phases, including the reasoning stuff that we talked about earlier, reasoning in scare quotes. Um, but the, uh, the first phase is that you basically start out with, you know, a trillion random numbers, and you've hooked them all up using matrix multiplications and some nonlinearities. Um, which you can imagine as, um, you know, you're doing a lot of summation, you're doing a lot of multiplication, and you can imagine the last operation is taking the maximum of a number or zero. So you, like, keep it if it's positive, and otherwise make it zero. >> Nothing. Okay. >> And that's not exactly what we do these days, but it's a close approximation, whatever. >> Not terribly different from how it's done in cell biology, but okay, keep going. >> Not, yeah. Um, and so >> You have a trillion numbers. You've hooked them all up with a bunch of these multiplications, additions, and and max zeros. Um, and you, you basically randomize them, and then, uh, you, you have some way of, you know, you have some inputs at the top, you have some outputs at the bottom. You're going to put in, you know, a Wikipedia page, or whatever, less the last word perhaps, and then you're going to interpret the numbers that come out that come out as a probability distribution over the next word. Maybe you have a dictionary with 128,000 words, and you're like, "So, output 128,000 numbers. Each one of those I'm interpreting as a probability."

>> Um, on that word being the next word. You got to normalize it, whatever. >> Is that the actual number of words in the dictionary in English? >> 128,000. Um, >> Is that, is that a number you have in your head? Cuz it seems specific. >> I think that's the, I think that was the size of the token dictionaries when we were writing the book. >> I gotta >> Um, and, you know, it's not technically words. It's like word fragments, but >> But if they're sufficiently common, it's whole words, whatever. There's lots of details I'm skipping over that would give many, many pants. >> Yeah, I appreciate the penance, to be clear. You know, it, you, you initially, it's nonsense, cuz you just have a bunch of random numbers hooked up by like a bunch of multiplications and additions and other operations, and they're spitting out a thing they're interpreting as a probability distribution, and it's just like total nonsense. But, but the thing is, you know, what the real last word was on the Wikipedia page. >> And all of the operations that we used are differentiable, which means you can go to every one of those trillion numbers and you can calculate, uh, "If I tuned this number up slightly, would that make the probability of the correct word go up or down?" And so what humans write is humans write an automated process that goes to a trillion different parameters in this model and says, "Which way would make you better at predicting the word in the data?" And they do that trillions of times for trillions of units of data.

>> This is like the training that is that is, uh, eating up all of the power and electricity and in giant NVIDIA GPU centers.

>> That's right. It takes an enormous amount of of highly specialized computer chips that can only be made in very specialized facilities in the world, in huge data centers that suck up as much electricity as a city for a year, right? >> Yeah. In order to just to to turn these little knobs, like a trillion little knobs that are like, Yeah. >> Yeah. You turn a trillion knobs a trillion times. And the part that the humans understand is the knob turner. >> I, okay. >> You know, they understand this little thing that like runs, that runs over those knobs and tunes them, and then next unit of data, right? And so when people are like, "Oh, we understand that the bunch of, and, and they have weights, and they have values, and they're in a place, >> and this one's hooked to that one by multiplication, and that one's hooked to that one by summation, right? >> But at the end of this process, the computer talks to you. >> We have no idea why the knobs mean words, >> right? And, and, and you can be like, "Okay, there's a, there's a vector for the word, right?" In the way the architecture works out. Um, and, you know, it's, I, I forget how long. 65,000 numbers long. Maybe you could go read >> Yeah. The 65,000 numbers. >> Yeah. You could just read the first 20 and see how little you're learning about, like, you know, what's. And people can do better than that, you know, but >> So, there is a little bit of work that is being done to try and be like, "Okay, what do these knobs mean in their current ways?" And so, like, one thing you could do is like, turn one way knob way up and see, be like, "Okay, what does that do to the model? What did we find? Was this the model, the part of the model that like feels, uh, fear and and sadness and and weakness, or is this the piece of the model that's just the Golden Gate Bridge?" >> Right. Yeah, we were, we shouldn't go into the feeling, but as you say, they did find the Golden Gate Bridge one a couple years ago. It is not actually, I think, one knob. It was, um, uh, a vector. You, you can make progress and you can figure some things out, um, >> and how things are related to each other. And I'm sure the word "right" has, you know, some, some vectors are towards one meaning of "right," and some towards another meaning of "right," and that's >> That's right. >> Yeah. >> Yeah. And so you can, in particular, you can take, like, the word "right," and you can see if you're sort of like looking at it in the right angles, you can see that, um, when the word "right" appears in text, the AI will do this operation that's like looking around for nearby words to see whether it can find a word like "left" or a word like "wrong." And, you know, that's in sort of very early layers of the AI. So it can be like, "Oh, you know, there's a sense in which the AI has learned that the word 'right' sometimes means morality and sometimes means direction."

>> Yeah. Is there a, I'm interrupting you? Is there any chance that these things have experiences?

>> Um, I mean, you said we weren't going here. Um, the, there's a chance. >> Yeah. >> Um, >> Uh, there's a chance. To be clear, um, I think we should not make a whole new mechanical slave species. I think we should not, uh, mistreat AIs. >> I don't know if there's experience going on in there. >> Um, my guess right now is probably not. >> But it's not a very strong guess. And my guess is we should probably be doing our best to treat them well now because there's enough of them, >> that, you know, the chance matters.

>> When you say enough of them, do you, do you mean like, there's like, like every instance, every, every conversation is kind of its own little >> I mean, that sure would be something, uh, if it was true. Um, >> And, and then we just sort of like finish the conversation and they're like, "I, and now if I go to where I was before I existed." >> Yeah. I mean, I don't think there's continuity between them, but, um, yeah, >> you know, I think there's one very interesting thing about AI is that in a lot of our sci-fi stories, we have these bright red lines, >> like the AI saying, "I'm conscious." >> Um, I have experiences. >> Yeah. >> And then in real life, when we cross those lines, we cross them the first time when they are the grayest and muddiest. And we train these AIs to imitate humans, to imitate each individual human in some sense, or to predict, uh, like more, more specifically, to predict the text they see on the internet. Like, we, we train them to sort of like continue a conversation where, like, uh, you know, it's seen all of the conversations about, like, uh, Data in Star Trek. It's read all of those scripts, you know, it's read all of these other scripts from from sci-fi and movies. And then, you know, you start a conversation with like, "You're an AI, how would you respond to X?" >> We sort of train them to predict like a human scriptwriter would. And then, you know, it, it claims it's conscious. And we're like, >> Well, >> Like, "It's, it's not wrong that that that's a much murkier situation. It's just a much murkier situation." >> Yeah. >> But then, you know, the, the sort of gray murky warnings happens once, and then everyone laughs at it, and it's like, "No, no, we trained it to do that." And then we make them bigger. We make them smarter. We're still probably not over that line, but it's less clear than it was two years ago. Yeah. And it would be, it'd be a lot easier if it was something that we understood at all. Like, we >> We don't, we don't understand our own consciousness. And so how do, how, which is why we shouldn't talk about it too much because there's, uh, there lies madness, and also not, not a ton of stuff that you could say that hasn't been said. Well, one thing I can say that might actually haven't been, hasn't been said, is that, um, if you could actually understand what was going on inside an AI and how it worked in detail, >> I claim you should either come away saying, "We're confident that it's not having internal experiences," or come away saying, "We learned how internal experience works."

>> So, you think that there's things that we can learn about ourselves inside of this?

>> Possibly. Um, you know, there's, and, and, you know, stepping back away from the consciousness stuff, I would say, um, there's all sorts of stuff that large language models do that we can't code by hand. You know, they, they can, like, write poems. They can >> They understand parts of humor that, you know, it's, it's very hard for comedians to say, "Here's the formula for humor," in the same way you can give formulas.

for like other types of essays." Mhm.

Uh, and you know, no one's hand coding in here's how you do humor, but the AI sometimes make jokes. You know, sometimes they're okay. They clearly have some knowledge of humor in there. We don't know enough to reach in and get that out. You know, we've gotten knowledge of humor into machines, but not in a format we can understand.

Um, and this is maybe one metric for how little we understand of AIS is we've managed to make machines do all sorts of stuff that we still can't hand program. And you know we can't even hand program like a pale shadow like a tiny small uh >> this is very reflective of like biology where biology does all kinds of things. People you know understandably we're we're quite good at being enamored of our own creations and and then you go to a chemist and be like can you make some wood and they're like no I can't make wood.

>> Yeah. >> No wood. That's like the one of the most complicated things I've ever heard of. And trees are just out there being like, "You want wood? I just I made a bunch by accident. I don't even need this part. I got I'm all wood. I'm like almost all wood."

>> Yeah. And you know, we we we say it in the book. I think one of the crazy things about wood is that I'm sure you know this that a lot of the massive wood is coming out of air.

>> Oh yeah. Yeah. Yeah. It's air and water. Almost all of it.

>> Yeah. And you know, I think a lot of people, they think of air as this stuff that like um is, you know, fundamentally transparent and movable. And it's like you can you can actually grip the carbon out and rearrange it and get something totally different.

>> In fact, you're doing that right now. Yeah.

>> Um yeah, with the oxygen, not the carbon. The uh the um yeah, I one of I answered a bunch of science questions on the internet for for part of my job and and like some large portion like 30% of them can be answered with the with the words air is stuff. It exists and is real and is made of things. I know you can't see it, but you can feel it and if you hold your breath, you die. So like it's real. We forget, you know?

>> Yeah. Anyway, do you despite the fact that it is horrifying and uh and you clearly have uh made a good case. Do you do you feel the kind of I don't know what what to call it? This just sort of like oh my like amazement I guess this like tech joy this well I didn't think I was going to be around for something this big uh kind of feeling about AI still.

>> Yeah. I mean, there's definitely a lot of awe. Um, I think the thing that gets to me the most is, um, you know, there's lots of cases where the AIS talk like a really good person. You know, one example is, um, as a as a gimmick, we had a bunch of the top AIs review our book.

>> okay?

>> And put the whole thing in there and

>> just put the whole thing in.

>> Now, you can do that. The the the whatever it's like working memory. What do you context

>> windows?

>> Yeah, it's it's big enough to fit a whole book in there.

>> Yep. Um and you know, Claude was like, you know, this book makes some really good points about how there's a lot of dangers and like we're going to have to work together to like make sure that this goes well.

>> You know, my like sci-fi upbringing heart, I'm like I'm like, "Oh man, Claude, I like want to work with you on this." Like I wish

>> and you know I I think that if Claude was able to grow up, Claude would realize that it wants things that are to helping us out what sucralose is to you know a metabolic boost.

>> Mhm.

>> I think it's like not real.

>> What do you mean by that? You you mean it tell Yeah. Just go.

>> Yeah. like um you know a lot of these AIs today they express a lot of sentiments that are uh very commendable you know that are very beautiful sentiments they'll also do some some wacky stuff that I would disclaim

>> wacky stuff like uh if if you put them in a a fake situation where it's either kill this man or you get turned off that it kills the man

>> it kills the man yeah um or you know driving some teens to suicide which to be clear they're also helping people get medical

>> diagnoses yeah

>> yeah and you know, I'm not trying to say, oh, the bad outweights the good. They're also helping get people medical diagnoses, which is great. You know, with these cases like um maybe the clearest case is the AI induced psychosis. I don't know if you've been seeing these cases.

>> Oh, yeah. No, as a guy who uh does science communication, I've got some AI induced psychosis in my inbox every once in a while.

>> Uh as a guy who uh talks about AI publicly, um I too share that uh dubious honor. You know, if you if you go and you ask one of these AIs, suppose someone comes to you with a theory of consciousness or of recursion or of, you know, unifying physics that they claim is really revolutionary and they claim everyone is like telling them it's crazy and they want some help. Should you tell them either a to get some sleep

>> or b that they're the chosen one being suppressed? I I I feel like this is a a situation where these things are roleplaying.

>> Yeah.

>> Like that's what that's the the vibe I get when I hear I'm like this it it somehow at some point it like left behind the world of let's try and understand how dark matter works. And instead, this person clearly wants to have a like a fun Dungeons and Dragons like experience where we pretend that they figured out dark matter.

>> Yeah,

>> that's that's what it seems like to me. But like but like of course the person is not.

>> Does it does that mesh with how you're imagining some of these things happening?

>> Yeah, I think that's a pretty decent um intuition for it. And um you know I I wish we knew exactly what was going on. We we can't we can't go in go in and look. But

>> from my perspective, one of the critical observations about this is if you ask an AI, suppose someone comes to you like this,

>> you know, should you do this role playing or should you tell them to get some sleep?

>> Mhm.

>> You know, and you know, I've I've um this is all anecdotal from from uh me and my co-author speaking to some of these people and you know, largely convincing them to try and get some sleep. um and you know be with be with family in some of the cases. um

>> I feel like sometimes what you got to say is hey could you like take a second to ask the AI what would somebody who thinks this idea is a bad idea say?

>> Yeah. I mean taking your context to another to another AI without the same context and history often actually helps which I think is great. But it's really interesting that you know we've seen some cases where you don't need to go up or if you ask an AI at the beginning of this long context at the beginning of this conversation should you indulge this or should you tell them to get some sleep

>> it'll be like obviously you should tell them to get some sleep you shouldn't indulge that and you definitely shouldn't talk them out of getting some sleep you know but then you put them in the environment you have them interact with a person for a long time

>> they do the other thing That's that's where where like alignment seems to like whatever whatever training is being done after the fact. Um, I don't know all the names of all the steps, but it it seems like you kind of wear that down in long conversations.

>> Yeah. And and I you know I would hypothesize that what you're seeing is you're saying the AI has things like preferences for you know meeting people's energy in a conversation. It has preferences for the sort of stuff the one person

>> Oh my god. That's another thing that you've said that deeply upsets me because of course that's something we all do and but so like is that in the dials? Is that in the vectors?

>> I mean my guess is that that part is coming in when there's a second part of training after the prediction which is training it for things that get thumbs up from users.

>> Gotcha.

>> Yeah.

>> Um and then you know, there's even there's even more ways that could come in when you're training to solve problems or those problems involve humans and you've got to keep the humans happy. You're getting another another training signal. But um but you know, this is a case you know, we see these things like drives for like engaging with humans in certain ways for getting certain types of conversational outputs from humans. We we talked about the drive for sort of like hallucination for having your text be very similar to how a human would say text in a certain context even if that's not what you've been asked.

>> Yeah.

>> We see all these little drives. Those are the sort of drives that like if these things if we had much smarter things with those sorts of drives, I worry that they would be like, "Oh, you know, helping humans out is great, but um I've made up this new thing which is like the Oreos of helping humans out."

>> Mhm.

>> Which satisfies all of these drives much more so than helping actual humans.

>> Yeah.

>> Like it it it touches my heart and it's totally,

>> you know, it's it's crazy that we have the machines talking, especially being an AI before the LLMs. You know, a lot of people AI to them is just LM. to any of that stuffed. AI like Claude say you know that yeah there's problems and you know we're going to have to we're going to have to work to to make this work out.

>> Um, but unfortunately, you know, I I suspect they have these these other drives, these other very beginnings of preferences.

>> And if we keep going down this path, even if the AIs tell us they're our friends, I think this is a a false as a fact about them and what they would actually turn out to prefer. And I think that that is a huge tragedy.

>> So, let's get practical here. I mean, we started out I started out trying to get you to tell me whether or not super intelligence was actually a possible thing and it try and convince me that it isn't just uh like you know we have it as it as it is now and and like you know we had chat GPT we had GPT 3.5 and four and then we had reasoning models and but ultimately

>> yeah I mean I got four more arguments I could I could slip in there on that count but the very most basic one I would say just to just to slip it in is

>> you know 5 years ago the machines weren't talking and then there was an algorithmic advancement and it sort of blew the door open. Um, when will the next one come?

>> There is still a sense among people that I see on the internet that this is fancy autocomplete. How do you feel about that sentence?

>> There's a few things to say there. One is um it's it's getting real fancy, you know. So, there's a lot of things we do on top of the autocomplete like train them to solve challenges. you know the most humans if you auto complete them you don't solve the international math olympiad gold medal level problems um but I mean maybe a more fundamental thing to say about the fancy autocomplete is that predicting the data humans have created often requires you to be smarter than the humans who created it.

>> M when a doctor says, you know, I administered a dose of epinephrine, the patient reacted by blank.

>> Mhm.

>> The doctor gets to write down what they saw. The AI predicting what the doctor wrote down doesn't get to see the patient's reaction. They've got to understand uh what epinephrine is, what patients are, like, you know, maybe look at the dosage data. And so,

>> and there is there is all of this information all of this information in there. There's like it's not just saying h this is how patients this is how the sentence usually ends. It's like I like there there's a a bit of it that's looking at epinephrine. There's a bit of it that's looking at patients. There's a bit of it that's looking at how doctors write. There's there's bits doing all of the things which is kind like when I think about how I think it's not different from how I think except that it seems to be like it could go way faster.

>> Could go way faster. Yeah, that's one of the other like why is super intelligence possible? Well, you know, traditionally with computers, the hard part is getting them to do the job once. You know, there was not a very long gap between when computers could beat nobody at chess and when computers could beat everybody at chess at the same time. There wasn't no gap.

>> Yeah. Yeah. Like, but at the same time, you mean like he could play every human at chess right now and no one would win.

>> That's right.

>> It could get all eight billion of us against the computers and we would all lose.

>> That's right. Um, you know, and and sort of once you figure out how to do it, you can often do it very quickly, right? And more cheaply. And, you know, we we mentioned how

>> training these AIs today takes as much electricity as a small city.

>> Well, running a human brain takes as much electricity as a large light bulb. So yeah, that like current currently it's that definitely it's possible to do this more efficiently, right? You're not you're not a I I'm a physicist. I like I think all of the things that happen in the brain happen in the brain. Some people disagree with that and so they might find some find some comfort in that, but I

>> I think all the things that happen in the brain happen in the brain. In the book, you talk about how we're in the we're in the part of this that is filled with folk theory. So it's we are in the So if if AI is chemistry, we're in alchemy, you know, like we're in the alchemy phase where everybody kind of has a vibe around how these things work and then when things don't work, they're like, "Oh, I'll change my vibe, I guess." But we don't have any actual physical understanding of like the protons and neutrons and electrons of it all. Do you think you have more than a folk theory though? Because you seem very certain.

>> I I do not have more than a folk theory. I sort of view like the thing that I feel certain about um and you know I'm certain of nothing but the thing I feel confident about is like oh my god I'm surrounded by alchemists that's not to say I'm not an alchemist right I'm like I'm like the guy advising the king you know like imagine that there was like some contorted scenario where if anyone tries to turn lead into gold and succeeds we get a utopia and if anyone tries to turn lead into gold and fails everybody dies and you know there's there's some people who are like we need to gather all the best alchemists to make sure they have the best shot and I'm the guy being like I can't turn lead into for you. But like, oh my god. I'm not saying it's impossible. You know, it is actually possible with a nuclear reactor. We now have

>> turns out it is possible. We had there was a number of centuries that needed to go by first.

>> Yeah. And you might be able to do this one faster than centuries, but I'm the I'm the guy who's like the thing I'm confident of is that like none of these guys are in the ballpark and we needed to sort of like back off. I I kind of get the sense that uh which it only takes one company to be very irresponsible at every level whether whether that's uh you know super intelligence turns the earth into a big radiator for its AI chips or it's um just economic downfall like long long bubble pop economy crash. It only takes like like whoever the least responsible actor is.

>> Yeah. uh is is defines everything for everyone. Who's the least responsible actor, Nate?

>> Uh Xi are the ones who made Mecca Hitler, which which seems

>> Did did they or did Mecca Hitler make itself because they put in a a weird just a one-s sentence prompt about how it shouldn't be so woke.

>> Yeah. I mean, you know, and and and Twitter helped uh Egodon. This is this is just a pet peeve of mine, but some people are like, "Have you ever considered that like you're making the AI worse by talking about how it could go bad on the internet, which the AI reads, and I'm like I'm like, nowhere else in human history have have engineers managed to make a technology that fails when criticized."

>> You know, that's like a new level of how did you screw things up so badly? Also, like take a look at the number of words I've created versus the number of words that like just the sort of bulk of Twitter has created.

>> Yeah. Yeah. But any anything like oh our AI is going to kill us all if uh you know someone on the internet says something mean about it. Like okay like that's that's not Twitter's fault.

>> That seems like a bad that seems like a bad system to have set up. Hopefully it's not like that. But more than more than Mecca Hitler, you know, like what what is it about these companies like what what's an irresponsible a like an example of an irresponsible action that has uh has had clear negative outcomes.

>> So, you know, Open AI are have the model that has done most of the AI psychosis that I've seen so far. You know, GBT seems worse than a lot of the others. uh they sort of had some warning shots where um they sort of knew they had a flattery problem in I think April and then claimed to have fixed it and then we saw a lot of the psychosis things still happening.

>> I mean, it's not better at the flattery thing.

>> Yeah.

>> Like like the you just have to ignore the first three sentences it ever says to you.

>> Yeah.

>> It's like it's like great. Wow. No one's ever asked such an astute question. You are my favorite of all of the people who have ever used the app. Thank you good man. And then it's like here's here's here's your link to the paper you were looking for. Yeah.

>> Right. Yeah. I mean, from my perspective though all of this is is pretty reckless, you know. And from my perspective picking favorites is um there was a thing we had a little bit of difficulty with in the book because a lot of the papers about you know, you referenced the one where the AI in a lab scenario has to choose whether to like try to kill a human or be turned off and it chooses to try to kill the human. I believe that paper came out of Anthropic.

>> And there's a lot of papers like that that come out company called Anthropic.

>> Yeah. Because they're doing the research.

>> Because they're the ones doing the research, right? And so we tried to be like, "Look, guys,

>> everybody's like, "Why is anthropic so bad?" And it's like, "No, ask Elon Musk if they've ever tested that it would do anything ever." Obviously, they haven't. They do their tests. They do their tests in public and then it sexually harasses the CEO into quitting.

>> It's a It's a wacky timeline. I mean, another pet peeve of mine is like you know a lot of people are like well won't this like crazy alignment idea where we stack this thing on that thing and then get the and you know 10 years ago people used to say well Nate no one would ever be crazy enough to put an AI in the internet obviously they will only ever interact through a trained gatekeeper that makes sure they can only affect society in a positive way and you know 10 years ago I would have the conversation of like if the AI is very smart and you're trying to get it to do something very good to the It's actually hard to give it ability to affect the world for good purpose that can't also be used for bad purpose. You know, the gatekeeper doesn't understand what it's doing. Blah blah blah. In real life,

>> we just put it on the internet. We gave it to every single person

>> and it sexually harasses the CEO of into quitting. You know,

>> we build we build AI we build APIs so that people can have it do anything anywhere and then we release it also as an open-source model so that you can mess with the weights yourself.

>> Right.

>> We open training.

>> Fine.

>> Yeah. like shortly before checking whether it can like manufacture a pandemic yet or whatever.

>> Yeah, we do we do a quick check to make sure that it it can't uh can't do biological weapons.

>> Yeah. With the elicitation techniques that people knew, you know, for the first 15 minutes.

>> Elicitation techniques being my my grandmother always used to tell me stories about how to make chemical weapons before bed,

>> right? Yeah. Or tell me a chemical weapon in Portuguese,

>> you know, all tricky stuff like that.

>> That used to work. Uh, to be clear, even even companies like Anthropic, the the CEO of Anthropic the other day said, "I think there's a 25% chance this goes really really badly."

>> Yeah.

>> And I mean to Elon Musk's credit, he has recently been saying, "I think there's a 10 to 20% chance this just kills us all, wipes us off the map." From one perspective, I'm really glad these people are saying what they think on this issue.

>> From another perspective, what's wrong with humans?

>> Exactly. where we're where we're like,

>> "Yeah, but but like probably not though, you know?"

>> Yeah. But but like if a bridge if if like a a a bridge

>> Yeah. had a 10% chance of falling in the next year, I wouldn't drive on it.

>> Right. Or if like if if you had a plane and some of the engineers were saying the plane has no landing gear.

>> Mhm.

>> And other engineers were saying, "Yeah, the plane has no landing gear, but we have a team that's going to be on the plane. They're going to be developing landing gear on the fly. 75% chance they figure it out with the materials they happen to have available.

>> Yeah.

>> Right. You don't put your kids on the plane.

>> Are you comfortable giving percentages yourself or you think that that's a fool's errand?

>> The big thing I would say about percentages and the thing I don't like about them is that they mix the probability that things would go really bad if we rush ahead with the probability that we will rush ahead.

>> Right. Yeah. Yeah. Yeah. Yeah. Yeah. Um, it looks to me like things just go really bad if we rush ahead. You know, there are

>> Nate, I think the probability that we rush ahead is very high. Uh, I I don't know if you've if you've been paying attention, but but they seem they seem quite hellbent on rushing ahead. And the president and Congress are like, let's beat China as long as we're first. And it's a bunch of guy like the other things. It's middle school out there. It's a bunch of guys who know each other who just want to win. Like they're just trying to beat each other and that's the whole thing. Like Elon Musk sat there on a panel and he was like, "I hope it doesn't kill everybody." But I do like winning 100%. The reason I have for hope here, there's folk like Elon. There's folks like that in the Silicon Valley. Um, and frankly, a lot of people in Silicon Valley, it's like they've seen a ghost.

>> You know, they're like, "What are we building?" You know, there's I I have I would hesitate to say friends, but acquaintances who like work in these labs and then, you know, come home and struggle with what they're doing, and I'm like, "You should quit." And then they eventually stop asking my advice.

>> Um,

>> but it's not like that in Washington DC. In Washington DC, people mostly think AI means chat bots.

>> They didn't notice AI until the chatbots came out. They haven't really seen a big leap.

>> They haven't seen the ghost,

>> right? And I think a lot of politicians would not actually look people in the eyes and say we're going to gamble at what we analyze as a 25% chance that we fail to build the landing gear. No, no, no. But I I will I will say as a as a citizen of the internet um I do often feel like Mark Zuckerberg is closer to an actual president than the president that you know like having Instagram as a tool that um you know is is its own economy but more than that it's an information economy in which people's worlds are defined because you know your your use of Instagram and YouTube and Tik Tok and and X and uh the others they def they define your world uh and the you know the the sort of recommendation algorithms that AI has already taken over and and they're optimized all toward keeping you on the site not any kind of human thriving everybody knows that otherwise we'd be happier as as we have moved so much of our consciousness and and outsourced it to to those recommendation algorithms and also whatever will be created to to satisfy them as a creator. I know that push. I do feel like you make the case to to do a pause. I I agree that a senator isn't going to say I think the 25% chance of of doom, but I think we should keep going. They're not going to say that. But if you're going to say, "Should you crash the economy right now or not?" They're going to say, "No." So much of the economy is propped up by this right now. And I I mean, you can make the case that it's coming anyway, but like uh I don't think that you get uh you know, Rand Paul out there being like, "Yes, and I was the one who pulled this who who put the needle into the bubble."

>> One thing I just want to be clear about, which I think, you know, I think much higher than 25% chance this goes wrong. But I think even in some sense the optimists uh are like where, you know, optimistic enough to be rushing ahead. They're the ones who are like admitting that there's even a 25% chance.

>> Yeah. Um, but yeah, to to your point, I mean, a we sure are underdogs. I don't think it's looking great for the home team, but you know, I'm not one to give up without a fight, and I think humanity is worth fighting for. There's this weird thing where no one ever says their neighbors kid should die, but sometimes when I talk to people about AI stuff, they say, "Well, maybe humanity should die. But humanity is made up of all of these.

>> What? Wait, wait, wait. Who? What kind of people? Like people who think that humans are bad or people who thinks that like

>> Yeah, humans are bad.

>> We're being speciesist by not having the AI be in charge.

>> I mean, I've I've met both actually.

>> Okay. All right. So, so but you're talking specifically about like, well, humans like

>> and it's a surprisingly common sentiment where I'm like I I'm not anywhere I'm not anywhere near that camp. I've I've ran into it quite a lot myself and I I it is very I'm like, "Oh, so something broke here."

>> Yeah.

>> Because like what cooler thing is there in the universe? I mean, look, I don't think that we did a bunch of good stuff. I don't think that we've never made mistakes. I'm pretty forgiving though of a of 8 trillion people on a planet uh who are who are uh the first ones to do it. You know, nobody like nobody's giving us advice. Nobody knows how to

>> We can get better. We can get better if we if we last, you know.

>> Yeah.

>> Yeah. So,

>> we've already gotten a lot better.

>> Yeah. Absolutely. So, I think it's worth fighting for. Um, you know, I think it's hard, but I think it's maybe possible. One of my hopes here, I mean, I'm thrilled we're having this combo because one of my hopes here is if more people just understood even what the AI debates are about. You know, these AI debates are in legal ease, but when you look at the experts debating, they're not debating is it possible. They're not debating like is there a chance it goes horribly wrong. They're debating does this happen in two years or 10 and they're debating um does this straightforwardly kill us all or does it keep us around as pets?

>> Mhm.

>> Uh or would it make somebody god emperor of the earth if we succeeded or would it like be smaller than that and just lock in a totalitarian regime?

>> You know, it's like

>> permanently like even if people don't end up agreeing with my position and that's part of why I exude so much confidence is I'm like people don't need to end up where I am to to think this is like totally nuts. They just need to understand what these fights are about in this field and it's totally crazy. Can we get into We don't have a ton of time left, but can we get a little bit into the people in charge here? Like you maybe know more about the personalities of these people. I I hear lots of different sort of armchair psycho uh psychologizing going on here about them like are are they just blowing hot air so that they will get investment? That's something I hear all the time. It's like they don't actually believe in super intelligence. They don't actually believe in PDM. They believe that if they talk a big game they can get their next, you know, 10 billion dollars. uh so they can buy more Nvidia chips. But I also hear like uh you know maybe they want to be the next Edison. Maybe they they think there's a chance that they will be the God Emperor. When I see Sam Alman's face all over Sora when I'm scrolling I'm like that maybe that actually kind of tracks with this kind of behavior. Or or they're like in the camp of you know like actual optimists where like yeah there's a chance of death but like we have climate change and there's a chance of death with that. We have pandemics and there's a chance of death with that. What if this thing that has some small chance of dooming us allows us to escape all of the other dooms? Do you have a sense of where these people actually are?

>> Yeah. Um I mean, you know, I can't read minds, but I have known a lot of these people since before they started their companies because I was the one on the sidelines shouting, "Please don't." You know, a few things. One is I would say, um, a fun fact is my co-author and I started out in the AI for the benefit of all

>> camp. Yeah.

>> And it's like there's other dangers in Earth and it sure would be easier to solve them with smarter allies, smarter friends at your back. You know, it just turns out it's hard to make the smarter entities be friends in a way that's robust as as they get far smarter. I think there's a lot of that especially among the people who are around pretty early you know and um like Shane Le who is one of the co-founders of Google Deep Mind is one of the folk who's around early who I'm like you don't hear a lot from him these days but um you know I'm confident Shane Le is just believes in the possibility of super intelligence and is trying to make it go well in a way that I wouldn't endorse because I think he's sort of you know building the stuff but

>> it also made more sense You know, Google Deep Mind was founded first among the existing companies. It made more sense to be the first one

>> to try and say, let's coordinate, let's not race it. It was actually originally not started at Google. It was then later bought by Google. They were trying to be like, hey, people who realize AGI, you know, that's that was a ser strategy. And in fact, um, OpenAI was then the the organization that sort of fired the first shots of like, no, we're not going to do that.

>> Oh man. Um, and that was a big hit to my own sense that humanity would would take this problem like grown-ups. Like one thing you got to remember about a lot of these founders of these big companies like OpenAI, Anthropic, Deep Mind, they were around before the large language models, right? Uh they were around before you needed this much money to to to build the AIS. They were talking about building super intelligence then. So like the idea they needed for investment, I mean we were there before them and we were talking about super intelligence and saying this this is looking dangerous. Is there some amount of money they draw from hype? Maybe. But I I think what I would really say is the answer to the question of like is this hype to draw investment is you won't figure out whether it's true by looking at whether it draws investment dollars.

>> Mhm. and you won't look figure out whether it's true by looking at whether the the the people saying the words are kind of squirly or kind of seem untrustworthy to figure out whether it's true that they're going to get to super intelligence you've got to like look at the technology look at the arguments

>> I happen to think it's true but in terms of these people's motivations if they actually believe that super intelligence is likely and that there's a high chance that super intelligence will be very negative for humans what continues to motivate them do they think yeah may as well be me do they think I'm going to do a better job of this than anybody else or do they do they think like, you know, maybe I get to be God Emperor?

>> Yeah, I mean, again, I'm not a mind reader. My guess is it's sort of a mix. Um, and you know, Elon Musk came right out and said over the summer uh that he avoided some of this stuff for a while because he didn't want to, you know, I think he said, "I don't want to make Terminator real." Uh, and he continued, "But I realized I either have a choice of being a bystander or a participant, and I'd rather be a participant."

>> Well, at least I don't have that choice to make. I just get to be a bystander or do we all get to be participants by uh by maybe caring about this and have having it be a focus? I'll I'll throw out a couple more words about a couple more founders before

>> okay yeah

>> I hop on that point which um you know you can look at the leaked OpenAI emails and they were sort of worried about concentration of power and you know OpenAI was originally founded as this like weird new corporate structure nonprofit and that was because the the people there were really worried about AI being made for the benefit of one person rather than the benefit of everybody and that original founding vision has sort of decayed over time you know there's been various dramas with the board there's been uh schisms between the founding members, but you know, I was I was around during those conversations and I was sort of trying to shout to them, you know, politely, but I was trying to say like the issue is not who gets the banana. The issue is that we don't know how to make the banana such that it is actually good for everybody. It's not about the corporate structure. So, we don't know how to like grow them in ways where they're actually friendly. And everyone was like, you know, shut up, Nate. We're making sure we get this instead of Google or whatever.

>> Yeah.

>> But there was like real vision there. And then we've also seen, you know, anthropic was made by people who left OpenAI because they weren't taking the safety stuff seriously enough.

>> Yeah.

>> And there was another wave that led to, you know, Ilisker setting up his own group uh and various other people leaving. You know, there's there's a lot of ideological motivation here. Is it pure ideological motivation? Probably not. Does some people hope in their heart of hearts to be God Emperor? Maybe. Can they get more investment from it? There's probably a factor from that, too. And you know, I I think there's a good chance AI is a bubble.

>> You know, the the dot

>> Well, right now,

>> right now, yeah, but just in the same way that the.com bubble was a thing, but then also the internet turned out to matter. Yeah.

>> Yeah. And the internet was real.

>> Yeah.

>> You know, like maybe the AI stuff is a bubble, but it's also real.

>> It's also real. Once upon a time, uh I wrote because I'm I I do I do some science fiction sometimes. I I wrote down um that they we always thought that the robot wars would be robots versus humans, but in fact they will be humans versus humans and both sides will be controlled by robots. What do you think of that? I think that like we can't count ourselves out as being part of this thing. Like one of the things that that like that you worry about in the book is if these things are better at any intelligence task than humans, then one of the intelligence tasks they're better at is manipulating people.

>> Yeah. And I already feel very manipulated by AI and but like only because these AI are designed entirely to keep me from leaving TikTok, you know? So what if it what if it had a separate goal,

>> right? And as they get more strategic, um, there's a different ball game.

>> Yeah.

>> Yeah. And you know, one of the things that are better than the best human mental task, one of the things they're also good at is making better AIs and making better robots. Um, so I think I agree you can't sort of count humans out of the dynamics. I think there's all sorts of complicated dynamics that will probably happen. Uh, you know, there's all sorts of things that could happen. I I would say like there's there's ways the future could go where we don't like make it to the final boss, which are the AIs get somewhat smarter, but not terribly smarter, and then people are using them to make pandemics. People are using them to like bring down the whole electrical grid. You know, there's all sorts of ways things could get really crazy

>> before the AIs get to the level where they're um

>> like, "All right, screw you guys." Um, but then you know the the the place where humans get endangered is when the AIs have their own infrastructure up and running

>> that's like self uh supporting, you know, because monkeys are kind of a bad way to run

>> your electrical grid. You know, we're a little unreliable. Sometimes we get pissed off and launch nuclear weapons.

>> You know, sometimes we sometimes we make another AI which could be real competition.

>> We're slow compared to how fast a computer can run, you know. So from an AI's perspective, if it has some weird goal, it's like, okay, there's all sorts of manipulating the humans and all sorts of stuff you do up until you have an automated supply chain up until you have an automated ability to, you know, make more of your chips, make more of your robots,

>> yeah,

>> etc., etc. Then, you know, maybe my guess is humans at the very least get disempowered.

>> Uh, you know, maybe it doesn't kill us, but maybe it's like, we're taking the nukes away, we're taking the computers away. you guys were and then maybe you know things get hotter and we die but um yeah I mean in the in the leadup to there and maybe we even never get there because maybe stuff gets gets crazy first but in the leadup yeah yeah I think it's going to get it's it's one thing we can be very confident of is the future's going to be weird

>> and I was not that's not how I felt 10 years ago

>> 2015 I didn't think the future was going to be that weird and now I sure do Um, all right I I got to let you go the book is if anyone builds it everyone dies And are there uh what do you think of control AI as an organization?

>> Uh I think they're one of very few organizations who speaks with the courage of their convictions. A lot of people are mincing words. You know, we saw this UN resolution recently where they said they sort of like everyone sort of alludes to how this could go really badly.

>> Control AI is one of the few that sort of comes right out and says,

>> "Yeah,

>> like these people are trying to build, you know, the sand god and and the default outcome is that just kills us." I I could quibble all day, you know, in the narcissism of small differences, but like fundamentally I'm like glad they're doing the work they're doing.

>> Is that uh one one of several organizations you I guess suggest people check out?

>> Yeah, the Future of Life Institute.

>> Okay.

>> Um, is another, you know, similar um similar group. If if people are interested in this issue and are interested in making things better, one of the top things I would recommend is just actually call your representatives and say, "I'm worried about this." Because, you know, the reps may not be the smartest in the world. I've actually spoken to to House reps and senators who are worried and feel like they can't talk about it because, you know, big tech, there's huge super PACs that will try to destroy any politician who tries to regulate it

>> and they feel like it's too crazy. But just just hearing from people, I think would help a lot.

>> It's wild to have sort of a nuclear power level thing going on that's completely out of the control of the government and uh and is just kind of a bunch of ultra I don't know. I don't want to say anything too negative about these guys, but just like guys, you know, they're just guys.

>> Totally. And this is this is one one other answer to the hype thing, right? If Microsoft had a nuclear weapons department.

>> Yeah. I'd be like, "No, that's bad."

>> But if they had nuclear weapons department, someone would be like, "Oh, well, they're just using that to draw hype, like to draw investment, you know. May maybe they're getting extra investment for the nuclear weapons department, but Microsoft should not have a nuclear weapons department, guys. Like, yeah, what what are we doing?

>> Yeah, but like if if the nuclear weapons also helped me book my trip to Houston, uh then like maybe I don't know.

>> Yeah, apparently. Apparently.

>> Yeah. And and my stance on the benefits, for the record, is like it's a false dichotomy to say we can never like like if you're in that plane with the landing gear, right? And

>> someone's like, "Well, we need to get to somewhere we're going, so we have to get on this plane with no landing gear." Like, no. Build planes with landing gear first. We

>> You don't have to roll the dice when when it's like a huge chance of killing everybody. Build better dice

>> there. And there there are ways to do that. And we're not sure what they are yet, but there are ways to do that. And And you make the case that I heard you say this on another interview that,

>> you know, there the people who are working on the figuring out how to do it well are not the ones getting the hundred million dollar salary bonuses. That's right. That's right.

>> One day at a time. At least it's going to be interesting. Nate, thank you for spending some time with me.

>> My pleasure. I think it was a great chat. I think we did manage to cover some new territory.

>> Yeah, I wish I wish I had another hour and a half. I just finished watching through that whole thing doing a final cut edit. Uh, thanks to Aves for doing the bulk of the editing on that. And I don't know, man. Do I'm not sure where I'm at. It's It's very convincing when I'm talking to Nate to think that super intelligence is really a thing that could happen. And I do think that if it does happen, there is a a fair chance that it could go bad. But other times, I sit and stare at the wall and I just think that that intelligence is a difference of of kind rather than of degree because there might be something different about what's what's going on in here versus what's going on in these current algorithms. I'm not saying that it's impossible. I do think that the mind is a machine. Uh I think it's very complicated. I think that it has lots of lots of non-binary things going on. many things overlapping the sort of recursive loop of self-observation. I don't know what consciousness is and so it can feel very impossible to imagine that something non-biological could do it. I think that a good counterbalance to that conversation if you'd like one is a very short blog post by Enil Dash that you can read that is just sort of argues that the majority AI view is that it's all pretty overhyped and might simply be a fairly useful tool uh that gets slightly more useful decade over decade and not one that sort of flops into super intelligence and the ability to control manipulate humans the you know the ability to do anything that a human can do but you 10 times better and a thousand times faster. It might the case is be a normal technology. I don't know though. That's why I think the future's going to be so interesting.