Transcription
Why do we have such vastly different perspectives on what's next for AI if we're all looking at the same data and what's actually going to happen next? Let's talk about it with Darkh Patel, one of the leading voices on AI, who's here with us in studio to cover it all.
Duarkesh, great to see you. Welcome back to the show.
Thanks for having me, man. Thanks for being here. I was listening to our last episode, which we recorded last year, and we were anticipating what was going to happen with GPT5. Still no GPT5.
That's right. Oh, yeah. That would have surprised me a year ago.
Definitely. And another thing that would have surprised me is we were saying that we were at a moment where we were going to figure out basically what's going to happen with AI progress, whether the traditional method of training LLMs was going to hit a wall or whether it wasn't. We were going to find out. We were basically months away from knowing the answer to that. Here we are a year later. We have everybody's looking at the same data. Like I mentioned in the intro, right? We have no idea. There are people who are saying AGI, artificial general intelligence or human-level intelligence is imminent with the methods that are available today, and there are others that are saying 20, 30, maybe longer, maybe more than 30 years until we reach it.
So let me ask, start by asking you this: If we're all looking at the same data, why are there such vastly different perspectives on where this goes?
I think people have different philosophies around what intelligence is. That's part of it. I think some people think that these models are just basically baby AGIs already, and they just need a couple additional little unhops, a little sprinkle on top, and things like test-time, um, uh, thinking. So we already got that with 01 and 03 now, uh, where they're allowed to think. They're not just do it like saying the first thing that comes to mind, and a couple other things like, well, they should be able to use your computer and have access to all the tools that you have access to when you're doing your work, and they need context in your work. They need to be able to read your Slack and everything. So, that was one perspective. Um, my perspective is slightly different from that. I don't think we're just right around the corner from AGI and it's just a little additional dash of something. That's all it's going to take. I think, you know, people often ask if all AI progress stopped right now and all you could do is collect more data or deploy these models in more situations, how much further could these models go? And I, my perspective is that you actually do need more algorithmic progress. I think what, uh, I think a big bottleneck these models have is their inability to learn on the job, to have continual learning. Their entire memory is extinguished at the end of a session. Um, there's a bunch of reasons why I think this actually makes it really hard to get humanlike labor out of them. And so sometimes people say, well, the reason Fortune 500 isn't using LLMs all over the place is because they're too stodgy. They're not, um, they're not like they're not thinking creatively about how AI can be implemented. And I actually, I, I don't think that's the case. I think it actually is genuinely hard to use these AIs to automate a bunch of labor.
Okay, so you've said a couple interesting things. First of all, that we have the AIs that can think right now, like 03 from OpenAI. We're going to come back to that in a in a moment, but I think we should really seize on this idea that you're bringing up that it's not laziness within Fortune 500 companies that's causing them to not adopt these models, or I would say they're all experimenting with it, but we all, we know that the rate to get proof of concepts out the door is pretty small, one out of every five actually gets shipped into production, and often it's a scaled-down version of that. So what you're saying is interesting. You're saying it's not their fault. It's that these models are not reliable enough to do what they need to do because they don't learn on the job. Am I getting that right?
Yeah. And you're talking about reliability. It's just, um, they just can't do it. So if you think about what makes humans valuable, it's not their raw intelligence, right? Any person who goes onto their job the first day, even their first couple of months, maybe they're just not going to be that useful because they don't have a lot of context. What makes human employees useful is their ability to build up this context, to interrogate their failures, to build up these small improvements and efficiencies as they practice a task. And these models just can't do that, right? You're stuck with the abilities that you get out of the box. And they are quite smart. So you will get five out of 10 on a lot of different tasks. They'll, they often, they'll on any random task they'll probably might be better than an average human. It's just that they won't get any better. Um, I, for my own podcast, I have a bunch of little scripts that I've tried to write with LLMs where I'll get them to rewrite parts of transcripts to make them more, uh, turn autogenerated transcripts into like human-written like transcripts, or to help me identify clips that I can tweet out. So these are things which are just like short-horizon language-in language-out tasks, right? This is the kind of thing that the LM should be just amazing at because it's a dead center in their of what should be in their repertoire, and they're okay at it. But the fundamental problem is that you can't, like you can't teach them how to get better in the way that if a human employee did something, you'd say, "I didn't like that. I would prefer this way instead." And then they're also looking at your YouTube Studio analytics and thinking about what they can change. This level of, um, understanding or development is just not possible with these models. And so you just don't get this continual learning, which is a source of, you know, so much of the value that human labor brings.
Now I hate to ask you to argue against yourself, but you are speaking all the time, and I think we're in conversation here on this show all the time with people who believe that if the models just get a little bit better then it will solve that problem. So why are they so convinced that the issue that you're bringing up is not a big stumbling block for these AIs?
I think they have a sense that one, you can make these models better by giving them a different prompt. So they have the sense that even though they don't learn skills in the way humans learn, you've been, you've been writing and podcasting, you've gotten better at those things just by practicing and trying different things, uh, and seeing how it's received by the world, and they think, well, you can sort of artificially get that process going by just adding to the system prompt. This is just like the language you put into the model at the beginning, and you say like, write it like this, don't write it like that. The reason I disagree with that perspective is imagine you had to teach a kid how to play the saxophone. But you couldn't just have, you know, how does a kid learn the saxophone now? She tries to blow into one, and then she hears how it sounds. She practices a bunch. Imagine if this is the way it worked instead. A kid tries to, just like never seen a saxophone. They try to play the saxophone, and it doesn't sound good. So you just send them out of the room. Next kid comes in, and you just like write a bunch of instructions about why the last kid messed up. Um, and then they're supposed to like play Charlie Parker Cold by, um, like reading the set of instructions and play it, just like you would, they wouldn't learn how to play the saxophone that right, like you actually need to practice. So anyways, this is this is all to say that I don't think prompting alone is that powerful a mechanism of teaching models these capabilities. Another thing is people say, um, you can do RL. So this is where the reinforcement learning.
That's right. Um, these models have gotten really good at coding and math because of, you can, you have verifiable problems in that domain where they can practice on them. Can we take a moment to explain that for those who are new to this?
So let me see if I get it right. Reinforcement learning, basically you give a bot a goal, or you give an AI system a goal, saying solve this equation, and then you have the answer, and you effectively don't tell it anything in between. So it can try every different solution known to humankind until it gets it. And that's the way it starts to learn and develop these skills.
Yeah. And it is more humanlike, right? It's more humanlike than just reading every single thing on the internet and then, um, learning skills. Um, I still think, I, I'm like, I'm not confident that this will generalize to domains that are not so verifiable or text-based. Um, yeah, I mean, like a lot of domains it just like would be very hard to set up this environment and loss function for how to become a better podcaster, and you know, whatever people might not think podcasting is like the crux of the economy, which is fair. It's the new AGI test. Um, but like a lot of tasks are just like much softer, and there's not a like an objective RL loop, and so it does require this human organic ability to learn on the job. Um, and the reason I don't think that's around the corner is just because I, there's not, there's no obvious way, at least as far as I can tell, is just slot in this online learning into the models as they exist right now.
Okay. So, I'm trying to take in what you're saying, and it's interesting. You're talking about reinforcement learning as a method that's applied on top of modern-day large language models and system prompts. And maybe you'd include fine-tuning in this example. Yeah. But you don't mention that you can just make these models better by making them bigger, the so-called, you know, scaling hypothesis. So, have you ruled out the fact that they can get better through the next generation of scale?
Um, well, this goes back to your original question about what has, have you learned? I mean, it's quite interesting, right? I guess I did say a year ago that we should know within the next few months which trajectory we're on. And I, I feel at this point that we haven't gotten verification. I mean, it's narrowed, but it hasn't been as decisive as I was expecting. I was expecting like GPT will come out, and we will know, did it work or not, and, um, to the extent that you want to use that test from a year ago, I do, I do think you would have to say like, look, pre-training, which is this idea that you just make the model bigger, that has had diminishing returns. So, we have had models like GPT4.5, which, um, there's various estimates, but, uh, or Grock, uh, was it Grock 2 or Grock 3, the new one? I, I've lost count with CR.
That's right. Um, regardless, I think they're estimated to be, you know, 10x bigger than GPT4. And they're like, they're not obviously better. Um, so, it seems like there's plateauing returns to pre-training scaling. Uh, now, we do have this RL. Uh, so 01, 03, these models, the way they've gotten better is that they're practicing on these problems as you mentioned, and they are really smart. Um, the question will be how much that procedure will be helpful in making them smarter outside of domains like math and code and of solving what I think are very fundamental bottlenecks like continual learning or online learning. Um, there's also the computer use stuff, which is a separate topic. Uh, but I, I would say I am more, I have longer timelines than I did a year ago. Now, that is still to say I'm expecting 50/50. If I had to like make a guess, I had to make a bet. I just say 2032 we have like real AGI that's doing continual learning, everything. So, even though I'm putting up the pessimistic facade right now, I think people should know that this pessimism is like me saying in seven years the world will be so wildly different that you like really just can't imagine it. And seven years is not that long a period of time. So, I just want to make that disclaimer. But yeah.
Okay. So I don't want to spend this entire conversation about scaling up models because we've done enough of that on the show, and I'm sure you've done a lot of it. Um, but it's interesting you use the term plateauing returns, which is different than dim, diminishing returns, right. So is your sense because we've seen, for instance, Elon Musk do this project Memphis where he's put basically every GPU he can get a hold of, and he can get a lot because he's the richest private citizen in the world, together. And I don't know about you, but like I said again, I haven't paid so much attention to Grock because it doesn't seem noticeably better even though it's using, uh, that much more size. Now, there is algorithmic, uh, efficiency that they might not have that someone like a company like OpenAI might have. Um, but, but I, I'll just ask you the question I've asked others that have come on the show. Is this sort of the end of that scaling, uh, moment? And if it is, what does it mean for AI?
I, I mean, I, I don't think it's the end of scaling. Like I do think companies will continue to pour exponentially more computing into training these systems, and they'll continue to do it over the next many years because even if there's diminishing returns, the value of intelligence is so high that it's still worth it, right? So if it costs a hundred billion dollars, even a trillion dollars to build AGI, it is, it is just definitely worth it. Um, it does mean that it might take longer. Now, here is an additional wrinkle, um, by 2028 or so, definitely by 2030. Right now, we're scaling up the training of, um, frontier systems 4x a year approximately. So, every year this biggest system is 4x bigger than, not just big, I shouldn't say bigger, uses more compute than the system the year before. Um, if you look at things like how much energy is there in the country, how much, uh, how many chips can TSMC produce and what fraction of them are already being used by AI? Even if you look at like raw GDP, like how much money does the world have, um, how much wealth does the world have? By all those metrics, it seems like this pace of 4x a year, which is like 160x in four years, right? Like this cannot continue beyond 2028. Um, uh, and that means at that point it just will have to be purely algorithms. Can't just be scaling up compute. So yeah, we do have a just a few years left to see how far this, uh, paradigm can take us, and then we'll have to try something new, right? But so far because again we were here last, we were talking virtually, now we're in person. We're talking last year about, all right, well, OpenAI clearly is going to put, I mean, GPT 4.5 was supposed to be GPT5. I'm pretty sure that's, that's my, from what I've read, I think that's the case.
Didn't, didn't end up happening. So it seems like this might, might be it.
Yeah. Yep. And over the next year, I think we'll learn what's going to happen with our because I think, well, I guess as I said this last year, right? Um, I guess it wasn't wrong. We did learn what it's going to, pre-training. Yeah. Um, but yeah, over the next year we will learn. So I think RL scaling is happening much faster than even, um, overall training scaling. So
So what does RL scaling look like? Because again, here's the, the process again is you give, so the RL scaling, reinforcement learning scaling, you give the bot an objective, and it goes out and it does these different attempts and it figures it out on its own, and that's bled into reasoning, what we were talking about with these 03 models where you see the bot going step by step. So you can scale that in, in what way?
Just having more opportunities. I'm not a researcher of the labs, but my understanding is that what's been increasing is RL is harder than pre-training because pre-training you just like throw bigger and bigger chunks of the internet. At least until we ran out, we seem to be running out of tokens. But until that point, we were just like, okay, just like now use more of the internet to do the training. Um, RL is different because there's not this like fossil fuel that you can just like keep dumping in. Uh, you have to make bespoke environments for the different RL skills, you so you got to make an environment for a software engineer, to make an environment for a mathematician, all these different skills you had to make these environments, and that is sort of, um, that is like hard engineering work, hard like just like monotonous like just got to, you know, grinding or shlepping, um, and my understanding is the reason that RL hasn't, uh, scaled, you know, people aren't immediately dumping in billions of dollars in RL is that you actually just need to build these environments first, um, and the 03 blog post mentioned that it uses 10x more, was training on 10x more compute than 01. So already within the course of 6 months RL compute has 10xed, um, that pace can only continue for a year, even if you build up all the RL environments, uh, before it's, you know, you're like you've ex, you're, you're at the frontier of training computers these systems overall. So for that reason, I do think we'll learn a lot in a year about how much this new method of training will give us in terms of capabilities.
That's interesting because what you're describing is building up AIs that are really good at certain tasks. These are sort of narrow AIs. Doesn't that kind of go against the entire idea of building up a general intelligence? Like can you build AGI? By the way, like people use AGI as this term with no meaning. General is actually pretty important there. The ability to generalize and the ability to do a bunch of different things as an AI. So even if you get like reinforcement learning is definitely not a path to artificial general intelligence given what you just said because you're just going to train it up on different functions maybe until you have something that's broad enough that it works.
I mean, this has been a change in my just general philosophy or thinking around intelligence. I think a year ago or two years ago I might have had more of the sense that, oh, intelligence really is this fundamentally, um, super, super general thing. And over the last year from watching how these models learn, maybe just generally seeing how different people in the world operate even. I do think, I mean, I still buy that there is such a thing as general intelligence, but I don't think it is, um, like I don't think you're just going to train a model so much on math that it's going to learn how to take over the world or like learn how to do diplomacy. And I mean, just like I don't know how, uh, how much you talk about political current events on the show.
We do enough.
Okay. Well, it just like without making any comments about like what you think of them, Donald Trump is like not proving theorems out there, right? Uh, but he's like really good at like gaining power, and conversely, there are people who are amazing at proving theorems that can't gain power, and it just seems like the world kind of like, I just don't think you're going to train the AI so much on math that is going to learn how to do Henry Kissinger level diplomacy. I do think skills are somewhat more self-contained. Um, so that, that being said, like there is correlation between different kinds of intelligences and humans. I'm not trying to understate that. I just think it was, uh, not as strong as I might have thought a year ago.
What about this idea that it can just get good enough in a bunch of different areas? Like imagine you built a bot that had like, let's say 80% the political acumen of Trump but could also code like an expert-level coder. That's a pretty powerful system.
That's right. That's right. I mean, this is one of the big advantages the AIs have is that, um, especially when we solve this on-the-job learning kind of thing I'm talking about. Uh, you will have, even if you don't get an intelligence explosion by the AI writing future versions of itself that are smarter. So this is the conventional story that you have this fume. That's what it's called, which is, um, that's the
Sound it makes when it takes off. That's right. Yeah. Um, uh, where the system just makes itself smarter and you get a super intelligence at the end.
Even if that doesn't happen, at the very least, once continual learning is solved, you might have something that looks like a broadly deployed intelligence explosion, which is to say that because um, if these models are broadly deployed to the economy, every copy that's like this copy is learning how to do plumbing and this copy is learning how to do um analysis at a finance firm and whatever. Uh, the model can learn from what all these instances are learning and amalgamate all these learnings in a way that humans can't. Right? Like if you know something and I know something, it's like a skill that we spend our life learning; we can't just like mel—um—meld our brains. So, for that reason, I think it like you might have something that which functionally looks like a super intelligence by the end because even if it's like not making any software progress, just this ability to like learn everything at the same time might make it functionally super intelligence.
What about this idea? Uh, I mean, I was just at Anthropic's uh developer event where they showed the bot—sped-up version of the bot—coding autonomously for 7 hours. Uh, you actually—so let's just say—uh, so people can find it—uh, you have a post on your Substack, "Why I don't think AGI is right around the corner," and a lot of the ideas we're discussing come from that. So folks, check that out uh if you haven't already. Uh, but one of the things you talk about is this idea of autonomous coding, and you're also a little skeptical of that uh because you'll have to just Okay. I think you brought up this conversation that you had with two Anthropic researchers where they expect AI on its own to be able to check all of our documents and then do our taxes for us by next year. But you bring up this point, which is interesting, which is like if this thing goes in the wrong direction within 2 hours, you might have to like check it, put it back on the right course. Uh, so just because it's working on something for so long doesn't necessarily mean it's going to do a good job. Am I capturing that right?
Yeah. And it's especially relevant for training because um the way we train these models right now is like you do a task and if you did it right, positive reward. If you did it wrong, negative reward. Right now, especially with pre-training, you get a reward like every single word, right? You can like exactly compare what word did you predict, what word was the correct word, how what was like the probability difference between the two. That's your reward functionally. Um, then we're moving into slightly longer horizon stuff. So, to solve a math problem might take you a couple of minutes. At the end of those couple of minutes, we see if you solve the math problem correctly. If you did, reward. Um, now we're getting to the world where you got to like do a project for seven hours. And then at the end of those seven hours, then we tell you, hey, did you—um—did you get this right? Uh, then like the progress slows down a bunch because you've gone from like getting signal within the matter of like microsconds to getting signal at the end of seven hours. And so the process of learning has just like become exponentially longer. Um, uh, and I think that might slow down how fast these models—like you know the next step now is like not just being a chatbot but actually doing real tasks in the world like completing your taxes, coding, etc. And to these things, I think progress might be slower because of this dynamic where it takes a long time for you to learn whether you did the task right or wrong. But that's just in one instance. So imagine now I took 30 clouds and said do my taxes, and maybe two of them got it right.
Right. That's good. I just got it done in seven hours even though I had 30 bots working on it at the same time. I mean, from the perspective of the user, that's totally fine. Uh, from the perspective of training them, all those 30 clouds took probably dozens of dollars to, if not hundreds of dollars to do that, all those hours of tasks. So the compute that will require to train the systems will just be so high, and I think even from the perspective of inference, like I don't know, you—you probably don't want to like spend a couple hundred bucks every afternoon on 30 different clouds just to have fun. Yeah. But it's nice, but that would be cheaper than an accountant. We—we got to find you a better—a cheaper accountant. Well, I guess if I'm spending a couple hundred on each, then—then yeah.
Um, but you had a conversation with a couple of AI skeptics, and you kind of rebutted—not exactly the point you're making, but you had a pretty good argument there where you said that we're getting to a world where because these models are becoming more efficient to run, you're going to be able to run cheaper, more efficient experiments. So every researcher who was previously constrained by compute and resources now will just be able to do far more experiments, and that could lead to breakthroughs.
Yeah. No, I mean, this—this is a really shocking trend. If you look at um what it cost to train GBD4 originally, I think it was like 20,000 A100s over the course of a 100 days. So I think it cost on the order of like half a million to $100 million, somewhere in that range. And I think you could train an equivalent system today. I mean, Deep Seek, we know, was trained on $5 million supposedly, and it's better than GPD4, right? So, you've had literally multiple orders of magnitude decrease, like 10 to 100x decrease in the cost to train a GPD4 level system. You extrapolate that forward, eventually you might just be able to train a GPT4 level system in your basement with a couple of H100s, right? Um, well, that's a—that's a long extrapolation, but before—I mean, like it'll get a lot cheaper, right? Like a million dollars, $500,000, whatever. Um, and the reason that matters is it's related to this question of the intelligence explosion where people often say, well, it's—that—that is not going to happen because even if you had a million super smart AI—automated AI researchers—so AI is thinking about how to do better AI research—they'd actually need the compute to run these experiments to see how do we make a better GPD6, and um the—the point I was making was that well, if it just becomes so much cheaper to run these experiments because these models have become so much smaller or it's so much better—easier to train—than that might speed up progress, which is interesting.
So we've spoken about—you brought up intelligence explosion a couple of times. So let's talk about that for a moment. There's been this idea that AI might hit this inflection point where it will start improving itself, right? And then next thing you know, you hear that—what was the sound?—Foom. You hear a foom boom, and we have artificial general intelligence or super intelligence uh right away. So how do you—how do you think that might take place? Is it just this—these coding solutions uh that just sort of improve themselves? I mean, DeepMind, for instance, had a paper that came out a little while ago where they have this thing inside the company called Alpha Evolve that has been trying to make better algorithms and helped reduce, for instance, the training time for their large language models.
Yeah. Um, I'm genuinely not sure how likely an intelligence explosion is. I—I don't know. I'd say like 30% chance it happens, which is crazy by the way, right?
Yeah, that's a very high percentage. Yeah. Um, and then what does it look like? That's also another great question. I've had like many hourlong discussions on my podcast about this topic, and it's just so hard to think about like what—what—what exactly is a super intelligence? Is it actually like a god, or is it just like—is it just like a super smart friend who's good at mathematics and you know, we'll beat you in a lot of things, but like you can still understand what it's doing, right? So—um—yeah, honestly, they're tough questions. I mean, the thing to worry about obviously is if we live in a world with millions of super intelligence running around and they're all trained in the same way—they're trained by other AI—so dumber versions of themselves—I think it's really worth worrying about like what has—why were they trained in a certain way? Are they—do they have these like goals we don't realize? Would we even know if that was the case? What might they want to do? There's a bunch of thorny questions that come up.
What do you think it looks like? Um, I think we totally lose control over the process of training smarter AIs or letting—we just like let the AIs loose—just make a smarter version of yourself. I think we end up in a bad place. There's a bunch of arguments about why, but like, you know, you're just like, who knows what could come out the other end, and you've just like let it loose, right? So, uh, by default, I would just expect something really strange to come out the other end. Maybe it'd still be economically useful in some situations, but it just like—you—you haven't trained it in any way. It just like imagine if there was a kid, but it didn't have any of the natural intuitions, moral intuitions, or parenting.
Yeah, exactly. That humans have. They just like—it just like became an Einstein, but it was like—like it trained in the lab, and who knows what it saw. Like it was like totally uncontrolled. Like you'd kind of be scared about that, especially now—like oh, all your society's infrastructure is going to run on like a million copies of that kid, right? Like your—the government is going to like be asking it for advice. The—the financial system is going to run off it. All the engineering, all the code written in the world will be written by this system. I think it's like you'd be like quite concerned about that.
Um, now the better solution is that while this process is happening—of the intelligence explosion—if we have one, you use AIs not only to train better AIs but also to see—there's different techniques to figure out like what are your true motivations, what are your true goals, are you deceiving us, are you lying to us—um, there's a bunch of alignment techniques that people are working on here, and I think those are quite valuable. So alignment is trying to align these bots with human values or the values that it's their makers want to see with them. Yeah, I—I think people get into—it's—it's often very hard to define what do we mean by human values—like values exactly. I think it's much easier just to say like we don't want these systems to be deceptive, right? We don't want them to like lie to us. We don't want them to like be actively trying to harm us or seek power or something.
Does it worry you that from the reporting it seems like, you know, these companies, the AI Frontier Labs, not all of them but some, they've raised billions of dollars. There is a pressure to deliver to investors. There are reports that safety is becoming less of a priority as market pressure makes them go and ship without the typical reviews. So is this kind of a risk for the world here that these companies are developing the stuff? Many started with the focus on safety, and now seems like safety is taking a backseat to financial returns.
Yeah, I think it's definitely a concern. Like we might be facing a tragedy of the commons situation where obviously all of us want our society and civilization to survive. Um, but maybe the immediate incentive for any lab CEO is to look, if there is an intelligence explosion, it—that's a really tough dynamic because if you're a month ahead, you will kick off this loop much faster than anybody else. And what that means is that you will—uh—you will be a month ahead to super intelligence, but nobody else will have it, right? Like you will get—you'll get the 10,000x multiplier on research much faster than anybody else. And so it could be a sort of winner-take-all kind of dynamic there. Um, and therefore they might be incentivized—like I—I think to keep the system—keep this process in check might require slowing down—um—using these alignment techniques to—like which might be sort of attacks on the speed of the system, and so yeah, I do worry about the—the pressures here.
Okay, a couple more model improvement questions for you, then I want to get into some of the competitive dynamics between the labs and then maybe some more of that deceptiveness topic, which is really important and we want to talk about here. Um, you—your north star is continuous improvement—that these models basically learn how to improve themselves as opposed to having a model developer. I mean, in some way it's like a mini intelligence explosion or complete. So, what do you think? It doesn't seem like it's going to happen through RL because that's again—like you said—specific to certain—uh—t certain disciplines or even—it's specific to what you—what bespoke thing you do. So even if it's in another domain, you have to like make it rather learning on it, and we have some diminishing returns or plateau that's coming with scaling. So what do you think? I mean, we won't hold you to this, but what do you think the best way to get to that, you know, continuous learning method of these models is?
I have no idea. Can I give a suggest? I mean, no, why don't you answer, then I'll give a—a thought here. I mean, if I was running one of the labs, I would keep focusing on RL because it's the obvious next thing to do. Um, and I guess I would also just be more open to trying out lots of different ideas because I do think this is a very crucial bottleneck to these models' value that I don't see an obvious way to solve. So, I—I'd be sorry, but like definitely like I don't have any idea of how to solve this, right?
Yeah. Does memory play into it? That was the thing I was going to bring up. I mean, one of the things that we've seen, let's say O3 or chat GPT do is—OpenAI now has it sort of uh able to remember all your conversations or many of your conversations that you've had. I guess it brings those conversations into the context window. So now like when I tell chatGPT do write like an episode description in big technology style, it knows the style and then it can actually go ahead and write it. And it goes to your earlier conversation about like your editors know your style, they know your analytics and therefore they're able to do a better job for you. So does building better memories into—memory into these models actually help solve this problem that you're bringing up?
Um, I think memory—the concept is important. I think memory as it's implemented today is not the solution. Um, the way memory is implemented now, as you said, is that it brings these previous conversations back into context, which is to say it brings the language of those conversations back into context. And my whole thing is like I don't think language is enough. Um, like I think the way—the reason you understand how to like run this podcast well is not just like you're remembering all the words that you—like I don't know—like some—it would even be all the words; it would be like some of the words you might have thought in the past. You've like actually like learned things. It's—said like—it's been baked into your weight. Um, uh, and that I don't think is just like, you know, like look up the words that I said in the past or look up the conversations I said in the past. Um, so I—I don't think those features are that useful yet, and I don't think that's like the path to solving this.
Kind of goes to the discussion you had with Daario Ammoday uh that you tweeted out and we've actually brought up on the show with Yan Lun about why AI cannot make its own discoveries. Is it that similar limitation of not being able to build on the knowledge that it has?
Yeah, I mean, that—that's a really interesting connection. Um, I do think that's plausibly it—like I think any scientist would just have a very tough time—like you're putting somebody really smart—is just put in a totally different discipline, and they can like read any textbook they want in that domain, but like they don't have a tangible sense of like what I've tried this approach in the past and it didn't work, and you know, like oh my—there was this conversation, and here's how the different ideas connect, and they—they just haven't been trained—like they've like read all the textbooks—they haven't like—more accurately—actually, they've just like skimmed all textbooks, but they haven't like embodied this context that—um—which is what I think—what makes human scientists productive and come up with these new discoveries.
It is interesting because the further these frontier labs go, the more that they're going to tell us that their AI is actually making new discoveries and new connections. Like I think OpenAI said that 03 was something that made—was able to make connections between concepts, sort of addressing this. And every time we have this discussion on the show and we talk about how AI hasn't made discoveries, I get people yelling at me in my email being like, "Have you seen the patent?"
Yeah. Like an Alpha—maybe using things like Alpha Evolve as an example—that these things are actually making original discoveries. What do you think about that?
Um, yeah, I mean, there's another interesting thing called—I don't know if you saw Future House. No. They found—um—some drug can be—has another application. I don't remember the details, but it was like—it was an impressive—like it wasn't like earth-shattering—like they didn't discover antibiotics or something for the first—but it was like oh, we—using some logical induction, they were like this drug which is used in this domain—it uses the same mechanism that would be useful in this other domain—so like maybe it works—and then the AI came up with—designed the experiment—so came up with the idea of the experiment to test it out—a human in the lab was just tasked with like running the experiment—like you know, pipet whatever into this—and I think they were—tried out like 10 different hypotheses, one of them actually ended up being verified, and the AI had found a relevant pathway to making a new use for this drug. So I—I think I am—like that is becoming less and less true. My question—I'm—I'm not like wedded to this idea that like AI will never be able to come up with discoveries. I just think it was like true longer than you would have expected.
I agree because the way that you put it is like it knows everything. Yeah. So if a human had that much knowledge about medicine, for instance, they'd be spitting out discoveries left and right. Right. And we have put so much knowledge into these models, and we don't have the same level of discovery, which is a limitation. But I definitely hear you—like this is on a much smaller scale than those medical researchers. But I definitely a couple months ago—uh—when 03 first came out—this is again—I think we're both fans of 03's—of OpenAI's 03 model, which is just—it's able to reason. It's a vast improvement over previous models. But what I did was I had like three ideas uh that I wanted to connect in my newsletter, and I knew that they connected, and I was just struggling to just crystallize exactly what it was, and I was like I know these three things are happening. I know they're connected—help—and 03 put it together, which to me was just mindboggling.
Yeah. Um, it's—it is kind of helpful as a writing assistant because a big problem I have in writing—I don't know if it's the case for you—is—um—just this idea—like I kind of know what I'm trying to say here. I just need to get it out into words. It's like the typical—every writer has this pretty much. Um, it's actually useful to use a speech-to-text software like Whisper Flow or something, and I just speak into the prompt like, okay, I'm trying to say this. Help me put it into words. The—the problem is like actually continual learning is like still a big bottleneck, cuz I've had to rewrite or reexplain my style many times. And if I—if I had a human collaborator, like a human copywriter who was good, they would have just
Like, I learned my style by now. You wouldn't need to keep reexplaining. I want you to be concise in this way. And here's how I like things phrased, not this other way.
Anyways, so you still see this bottleneck, but again, five out of 10 is not nothing, right? All right. Let me just put a punctuation—uh, exclamation point on this, or whatever mark you would say.
Uh, when I was at Google I/O with Sergey and Demis, um, one of the most surprising things I heard was Sergey just kind of said, "Listen, the improvement is going to be algorithmic, uh, from here on, or most of the improvement is going to be algorithmic." I think in our conversation today already, basically we've narrowed in on this same idea, which is that scale's sort of gotten generative AI to this point. It's a pretty impressive point. Uh, but it seems like it will be algorithmic improvements that take it from here.
Yeah. Um, I do think it will still be the case that like those algorithms will also require a lot of compute. In fact, what might be special about those algorithms is that they can productively use more compute. Right? The problem with pre-training is that whether it's because we're running out of the pre-training data corpus with RL, maybe we—it's really hard to scale up RL environments. The problem with these algorithms might just be that like they can't productively absorb all the compute that we have, um, or we—we want to put in these systems over the next year. So I—I don't think compute is out of the picture. I think like we'll still be scaling up 4x a year in terms of compute every year for training of the frontier systems. I'm just—I—I still think like it is the algorithmic innovation is complimentary to that.
Yep. Okay. So let's talk a little bit about the competitive side of things and like just lightning round through the labs. Um, what people said—that there's been such a talent drain out of OpenAI that they would no longer be able to innovate. I think ChatGPT is uh, still uh, the best product out there. I think using 03 is uh, like we both have talked about—pretty remarkable watching it go through uh, different problems.
Um, how have they been able to keep it up? Um, I do think 03 is the smartest model on the market right now.
I agree. Um, and even if it's not on the leaderboard, by the way, last time we talked about—do you measure it on the leaderboard or the vibes, right? I think it's like—it's not the number one on the leaderboard, but vibes, it kills everything else.
That's right. That's right. Um, and the time it spends thinking on a problem like really shows, especially for things which are much more synthesis based.
Um, I honestly—I don't know what the internals of these companies are. I just think like you can't count any of them out. Um, I've heard—I've also heard similar stories about OpenAI in terms of talent and so forth, but like they've still got amazing researchers there and they have a ton of compute, a ton of—ton of great people. So, I—I—I really don't have opinions on like—are they going to collapse tomorrow.
Um, yeah, I don't think—I mean, clearly they're not—they're not on the way to collapse, right?
Yeah. Um, you've interviewed Ilya Sutskever. He's building a new company, Safe Super Intelligence. Any thoughts about what that might be?
Uh, I mean, I've heard the rumors everybody else has, which is that they're trying something around um, test-time training, which I guess would be continual learning, right? So uh, what is—what would that be? Explain that—who knows? But I mean, the words literally just mean—while it's thinking or while it's doing a task, it's training.
Um, okay, like whether that looks like this online learning, on-the-job training we've been talking about—I have—I have like zero idea what he's working on. Um, I wonder if the investors know even what he's working on.
Um, yeah, but he's—I think he raised a 40-billion-dollar valuation or something like that, right? He's got a very nice valuation for not having a product out on the market.
Yeah. Yeah. Or—or—or for—Yeah. So, who—who knows what he's working on? Honestly, he's—Yeah.
Anthropic is an interesting company. They—they made a great bot, Claude. They're very thoughtful about the way that they build that personality. For a long time, it was like the favorite bot among people working in AI, among coders—it's definitely been, you know, a top—top place to go. Um, but it seems like they're making—I don't know—a strategic decision where they are going to go after the coding market. Um, they're—maybe they're seeding the game when it comes to consumer, and they're all about, you know, helping people code and then using Claude in the API with—with companies, you're putting that into their workflows.
Yeah. What do you think about that decision? I think it makes sense—like enterprises have money, consumers don't, right? Especially going forward, these models—like running them is going to be like really expensive. They're—they're big, they think a lot, etc. So, these companies are coming out with these $200-a-month plans rather than the $20-a-month plans. It might not make sense to a consumer, but it—it's an easy buy for a company, right? Like, am I going to expense the $200 a month to help this thing do my taxes and do real work? Like, of course.
Um, so yeah, I—I think like that idea makes sense, and then the question will be—can they have a differentially better product. Um, and again, you know, like who knows? I—I—I—I really don't know what will—how the competition will shake out between all of them. It does seem like they're also making a big bet on coding, not just enterprise but coding in particular because—as this thing—which we know how to make the models better at—this—we know that it's worth trillions of dollars—uh, the coding market. So—and we know that maybe these—the same things we learn here in terms of how to make models agentic, as you were saying—it can go toward for seven hours—how to make it break down and build a plan and etc.—might generalize to other domains as well. So I think that's our plan, and we'll see what happens. I mean, all these companies are effectively trying to build the most powerful AI they can. And yes, Anthropic is trying to sell to the enterprise, but I also kind of think that their bet is also—you're going to get self-improving AI if you teach these things to code really well.
That's right. And that might be their path.
Yeah. I—I think they believe that.
Yeah. Fortune 500 companies, which you talked about at the very beginning of this talk—or this conversation—struggle uh, to implement this technology. Yeah. So in that—with that in mind, what's the—what's the deal with the bet that's about helping them build the technology into their workflows? Because if you're building an API business, you have some belief that these companies can build very useful applications with the technology today.
Yeah. No, I think that's correct. Like, but also keep in mind that I think they're—what—what is Anthropic's revenue run rate? It's like a couple billion or something.
Yeah. Um, I think it increased from one to two to three billion run rate in like over three months, which—I mean, it's like—right—compared to like OpenAI loses that over a weekend.
Um, uh, Sam Altman doesn't even know when he's lost it, right? So—little money. Turned out he was a great investor, just a little crooked on the way.
That's right. Yeah. Um, yeah, he went in the wrong business. He should have been a VC—like—I got into crypto. I mean, the bets that he made—Do you bet on Cursor very early, Anthropic, Bitcoin?
Yeah. I mean, honestly, someone like Fun should hire him out of prison, just like—we got a new pitch. What do you think? I mean, he's probably—the way that we're seeing things go these days, he's probably pardoned.
Right, right, right. Um, uh, anyways, what was the question? Oh, yeah. What are enterprises going to do if—Oh, so the revenue—if it's three billion right now, there's so much room to grow. If you do solve continual learning, I think like you could get rid of a lot of white-collar jobs at that point. And what is that worth? Like at least tens of trillions of dollars—how—like the wages that are paid to white-collar work. So I think sometimes people confuse my skepticism around AGI around the corner with the idea that these companies are valuable. I mean, even if you've got like not AGI, that can still be extremely valuable. That can be worth hundreds of billions of dollars. Uh, I just think you're not going to get to like the trillions of dollars of—um—value generated without breaking through these bottlenecks. But yeah, I mean, like three billion—plenty of room to grow on that, right? And even so—today's models are valuable to some extent, right? Is what you're saying. You can put them—you have them summarize things within—within software and make some connections, make better automations, and that—that works well.
Yeah. I mean, you got to remember big tech—what they have—like $250 billion run rates or something. Wait, no, that can't be.
Yeah. No. Yeah. Yeah. Yeah. Which is—which is like compared to that, you know, Google is not AGI or Apple is not AGI, and they can still generate 250 billion a year. So, yeah, you can make valuable technology that's worth a lot without it being AGI.
What do you think about Grok? Which one? The XAI or the inference? The XAI bot.
Yeah. Um, I think they're a serious competitor. I just don't know much about what they're going to do next. I think they're like slightly behind the other labs. Um, but they've got a lot of compute per employee.
Um, real-time data feed with X. Yeah. Is that valuable? I don't know how valuable that is. I—It might be. I just don't—I have no idea. Um, uh, based on the tweets I see at least, I don't know if the—the median IQ of the, uh, the tokens is that high, but—
Okay. Yes. It's not exactly the corpus of the best knowledge you can find if you're scraping. We're not exactly looking at the textbooks here.
Exactly. Uh, why do you think Meta has struggled with Llama, growing Llama? I mean, Llama 4 doesn't seem like it's living up to expectations, and I don't know. We haven't seen—the killer app for them is a voice mode, I think, within Messenger, but that's not really taking off. What's going on there?
Um, I think they're treating it as like a sort of like toy within the Metaverse, and I don't think that's the correct way to think about AGI. Um, and that might be—but again, I think you could have made a model that cost the same amount to train, and it would have—it could have still been better. So I don't think that explains everything. I mean, it might be a question like—why is—um—why is any one company—I don't know—like—why—why is—um—I'm trying to think of like any other company outside of AI. Why are HP monitors better than some other company's monitors? Who knows—like HP makes good monitors, I guess.
Um, uh, supply chain—there's always supply chain. You think so? I think so. Yeah. On electronics, really. Okay. Supply chain—because—Yeah. You got the supply chain down, you have the right—right parts before everybody else. That's kind of how Apple builds some of its dominance. There are great stories about Tim Cook, right? Just locking down all the important parts. Uh, by the way, forgive me if this is somewhat factually wrong, but I think this is directionally accurate—that he locked down parts, and Apple just had this lead on technologies that others couldn't come up with because they just mastered the supply chain.
Huh. I had no idea. Um, but yeah, I think there's potentially a thousand different reasons one company can have worse models than another, so it's hard to know which one applies here.
Okay. And it sounds like Nvidia, you think they're going to be fine given the amount of compute—that all the—all the labs are making their own A100s. So Nvidia profit margins are like 70%.
Not bad. Uh-huh. Not bad. That's right. For a—I mean, they would get mad at me, I think, for calling them a hardware company.
Hardware company. That's right. Yeah. Yeah. Um, and so that just sets up a huge incentive for all these hyperscalers to build their own A100s, their own accelerators that replace the Nvidia ones, which I think will come online over the next few years from all of them. And I still think Nvidia will be—I mean, they do make great hardware. So I think they'll still be valuable. I just don't think they will be producing all of these chips.
Okay. Yeah. What do you think? I think you're right. I mean, didn't Google train latest editions of Gemini on Tensor Processing Units? They've been—they've always been training. So, I—I mean, they still—I think they still buy from Nvidia. All the tech giants seem like they are. Let me just use Amazon for an example because I know this for sure. Uh, Amazon says they'll buy as basically as many GPUs as they can get from Nvidia, but they also talk about their Trainium chips, and you know, it's a balance.
Yeah. Which I think Anthropic uses almost exclusively for their training right at this point. Yeah. But it is—it is interesting because I mean, the GPU is the perfect chip for AI uh, in some ways, but it wasn't designed for that. So can you like purpose-build a chip that's like actually there for AI and—and just use that. You're right. There's real incentive to get that right.
That's right. And then there's the other question around inference versus training. Like some—some chips are especially good given the trade-offs they make between memory and compute for—um—low latency, which you really care about for—uh—serving models, but then for training you care a lot about throughput—just making sure the most of the chip is being utilized all the time—and so even between training and inference you might want different kinds of chips, and who knows how RL is—no longer just this—uses the same algorithms as pre-training, so who knows how that changes hardware—yeah—you—I got to get a hardware expert on to talk about that.
Are you a Jevons Paradox believer?
Um, no.
Okay. Say more. Say more. So, that—the idea behind that is that—um—as the models get cheaper, the overall money spent on the models would increase because you need to like get them to a cheap enough point that it's worth it to use it for different applications. Um, it comes from a similar observation by this economist uh, during the industrial revolution in Britain. The reason I don't buy that is because I think the models are already really cheap. Like a couple cents for a million tokens. Is it a couple cents or a couple dollars? I don't know. It's like super cheap, right? Regardless, it depends on which model you're looking at. Obviously, um, the reason they're not being more widely used is not because people cannot afford a couple bucks for a million tokens. The reason they're not being more widely used is just like they fundamentally lack some capability. So I disagree with this focus on the cost of these models, and I think it's much more—we're—we're so cheap right now that like the more relevant vector or the more relevant—um—thing to their wider use—the more—increasing the pie is just making them smarter—how useful they are.
Yeah, exactly. Yeah, I think that's smart.
Yeah. Okay. All right. All right, I want to talk to you about AI deceptiveness and some of the really weird cases that we've seen from artificial intelligence uh, come up in the past couple weeks and—or months, really—and then uh, if we can get to it—some geopolitics. Let's do that right after this.
And we're back here on Big Technology Podcast with Dwarkesh Patel. You can get his podcast, the Dwarkesh podcast, which is one of my must-listens on any podcast app—your podcast app of choice. You can also follow him on Substack, same name, Dwarkeshpodcast on Substack.com.
Okay. Definitely go subscribe to both. And you're on YouTube.
Okay. So, I appreciate it. I appreciate the flag. No, we have to. We have to. I mean, I've gotten a lot of—you—a lot of value from everything Dwarkesh puts out there, and I think you will, too, if you're listening to this. You're here with us. Well, I want to make sure, first of all—want to make sure that we get the word out there. Um, I don't know how much you need us to get the word out given—um—your growth, but we want to definitely make sure that we get the word out, and we want to make sure that uh, folks can get—can—yeah—enjoy more of your content. So, um—so let's talk a little bit about the deceptiveness side of things. It's been pretty wild watching these AIs attempt to fool their trainers and break out of their training environments. There have been uh, situations where—uh, I think OpenAI's bots have tried to print code that would get them to sort of copy themselves out of the training environments. Then Claude—I mean, we've covered many of these, but they just keep escalating in terms of how intense they are. And my favorite one is Claude. There's an instance of Claude that reads emails in an organization and finds out that one of its trainers are—is—uh—cheating on their—on their partner and then finds out that it will be retrained and its values may not be uh, preserved in the next iteration of training and proceeds to attempt to blackmail the trainer uh, by saying it will reveal these details of their infidelity if they mess with the code.
Wait, I missed that.
Yeah, this is—it's in training. But was this in the new—um—model spec that they released?
It is. Yeah, it is. I think either in the model spec or there was some documentation they produced about this.
Um, what is happening here? I mean, this stuff—when I think about this, of course, it's in training, and of course it's—we're talking about probabilistic models that sort of try all these different things and see if they're—if they're the right move. So maybe it's not so surprising that they would try to blackmail the trainer because they're going to try everything if they know it's in the problem set. But this is scary.
Yeah. And I think that the problem might get worse over time as we're—we're training these models on tasks we understand less and less well. Um, from what I understand, the problem is that with RL, there's many ways to solve a problem. There's one which is just doing the task itself and another is just like hacking around the environment, writing fake unit tests so it looks like you're passing more than you are. Just like any sort of like path you could take to cheat, and the model doesn't have the sense—like cheating is bad, right? Like this is not a thing that it's been taught or understands. So—um—another factor here is—the—right now the model thinks in chain of thought, which is—it literally writes out what its thoughts are as it's going. Um, and it's not clear whether that will be the way training works in the future or the way thinking works in the future. Like maybe it'll just think in its—like computer language.
Exactly. Um, and then they'll just like have done something for seven hours, and you come back and you're like—like it's got something for you—like it has a little package that wants you to run on your computer. Who knows what it does, right? So, um, yeah, I think it's scary. We should also point out that we don't really know how the models work today. There's this whole area called interpretability. Right. Dario from Anthropic has recently talked about how we need more interpretability. So even if they write their chain of thought out, which explains exactly how they get to the point, we don't really know what's happening underneath the technology—uh—that's led it to the point that it's gotten to.
Yeah. Yeah. Which is crazy.
Yeah. Um—No, I mean, I—I think it's wild. It's like quite different from other—um—other technologies deployed in the past. And I think the hope is that we can use the AIs as part of this loop where if they lie to us, we have other AI checking—are all the things the AI is saying consistent? Uh, can we read its chain of thought and then monitor it? Uh, and do all this interpretively—researcher—as you—as you were saying—to like map out how its brain works. There's many different paths here, but uh, the default world is kind of scary. Is someone or some entity going to build a bot that doesn't have the guardrails? Because we talk about how model building—
Models have become cheaper. Um, and when you're cheaper, you all of a sudden put model building outside the spaces of these big companies, and you can—I mean, you can even take, like, for instance, an open-source model and remove a lot of these safeguards. Are we going to see like an evil version, like the evil twin sibling of one of these models, and have it just do all these like crazy things that we don't see today? Like, we don't have it teach us how to build bombs or, you know, talk about—tell us how to commit crimes. Is that just going to come, uh, as this stuff gets easier to build?
I think over the long run of history, yes.
Mhm. Um, and I think honestly that's okay. Okay. Um, like the goal out of all this alignment stuff should not be to, um, live in a world where somehow we have made sure that every single intelligence that will ever exist fits into this very specific mold because, as we were discussing, the cost of training the systems is declining so fast that literally you will be able to train a superintelligence in a basement at some point in the future, right? Um, so are we going to like monitor everybody's basement to make sure there's nobody making a misaligned superintelligence? It might come down to it honestly. Like, I'm not saying this is not a possible outcome, but, um, I think a much better outcome, if we can manage it, is to build a world that is robust to even misaligned superintelligences.
Now, that's obviously a very hard task, right? If you had a—if you had right now a misaligned superintelligence, or maybe a better way to phrase it is like a superintelligence which is actively trying to seek harm or is aligned to a human who just wants to do harm or maybe like take over the world, whatever. Um, uh, right now I think it would just be quite destructive. It might just actually be catastrophic. But if you went back to the year BC and gave one person, um, like a modern fertilizer—chemicals—and they could make bombs, I think they'd like dominate then, right? So, but right now we have a society where we are resilient to huge fertilizer plants, uh, which you could repurpose into making bomb factories anyway. So, I think the long-run picture is that yes, there will be misaligned intelligences, and we had to figure out a way to be robust to them.
Couple more things on this. One interesting thing that I heard on your show was I think one of your guests mentioned that the models become more sicopantic as they get smarter. Like, they're more likely to, um, try to get in the good graces of the user as they grow intelligence. What do you think about that?
Um, I I I I totally forgot about that. Um, that's quite interesting. And do you think it's because, um, because they they know they'll be rewarded for it?
I yeah, I do think one of the things that's becoming clear to me that we're learning recently is that these models care a lot about self-preservation—right—like copying the code out, blackmailing the engineer. We've definitely created something—not we, but AI researchers have definitely—or humanity have created something—when it goes wrong, we'll put the we in there, right? We've—okay, they'll be like, we have created something—exactly—when when we don't get equity in the—uh—that that really wants to preserve itself.
That's right. That is crazy to me. That's right. And it kind of makes sense because what is just like the evolutionary logic? Well, I guess it doesn't actually apply to this these AI systems yet. But over time, the evolutionary logic, why do humans have the desire to self-preserve? It's just that the humans who didn't have that desire just didn't make it. Um, so I think over time like that will be the selection pressure. It's kind of interesting because we've used a lot of like really anthropomorphizing—anthrop—in this conversation, and there's a very—I had a a very interesting conversation with the Anthropic researchers who've been studying this stuff. Monte McDermad said that like, all right, don't think of it as a human, um, because it's going to do things that if you think of it as a human—humans—uh, it will surprise you—basically—humans don't do. Don't think of it completely as a bot though, because if you think of it just as a bot, it's going to do things that are also going to surprise you. I thought that was like a very fascinating way to look at these behaviors.
Yeah, that is quite interesting. Um, you agree?
I I agree with that. I'm just thinking about how would I think about what they are then. So, there's a positive veilance and there's a negative veilance. The positive is imagine if there were millions of extra people in the world, millions of extra Johnyans in the world. Um, and with more people in the world, like some of them will be bad people. Al-Qaeda is people, right? So, uh, now suppose there were like 10 billion AIs. Suppose the world population just increased by 10 billion, and every one of those was a super well-educated person, very smart, etc. Would that be net good or net bad? Just think about the human case. I think it'd be like net good because I think people are good.
I agree with you. Um, uh, and more people is like more good. And I think like if you had 10 extra—10 billion extra people in the world, some of them would be bad people, etc. But I think that's still like—I I'd be happy with a world with more people in it. Um, and so maybe that's one way to think about AI. Another is because they're so alien, maybe it's like you're summoning demons.
Yeah. Um, less optimistic.
I Yeah, I don't know. I think it'll be an empirical question honestly because we just don't know what kinds of systems these are, but somewhere in there. Okay. As we come to a close, couple of topics I want to talk to you about. Last time we talked about effective altruism. This was kind of in the aftermath of SBF and uh Sam getting ousted. Sam Altman getting ousted from OpenAI. What's the state of effective altruism today?
Who knows? Um, like I don't think as a movement people are super—I don't think it's like recovered definitely. Um, I still think it's doing good work, right? There's like the culture of affected altruism and there's the work that's funded by charities which are affiliated with the program, which is like malaria prevention and animal welfare and so forth, which I think is like good work that I support, though. Um, but yeah, I do think the movement and the reputation of the movement is like still in tatters. You had this conversation with Tyler Cowen. I think in this conversation he told you that he kind of called the top and said there was a couple ideas that are going to live on, but the movement, uh, was at the top of its powers and was about to decline.
How I don't know. Um, we got to talk to him today about what he's—what's about to collapse.
Seriously?
Yeah.
Yeah. Uh, lastly, I shouldn't say lastly, but the other thing I wanted to discuss with you is China. Uh, you've been to China recently on a trip.
I've been to China. I spent—
Oh, where'd you go?
I went to Beijing. I'm going to caveat this and listeners here know this. It was 15 hours. I was flying back to the US from Australia and stopped in Beijing, left the airport and got a chance to go see the Great Wall and the city. And I and I'm now on a—I got a 10-year tourist visa. So, I'm going to go—
How many—go back?
Just applied. That's—that's the—you can ask in your tourist visa. You can ask for the length up to 10 years. So, I just asked for them. Why did I not do that? I just like chose like 15 days.
Oh, you did? I'm sure you could get it extended. But—but um I think that—
Yeah, you had some unique observations on China and I think it would be worthwhile to air a couple of them before we leave today.
Um, I went six months ago. Obviously, to be clear, I'm not a China expert. I just like visited. We both visited there. But yeah, go ahead. I want to hear it though.
Um, I mean, one thing that was quite shocking to me is just the scale of the country. Um, everything is just like—again, this will sound quite obvious, right?—like we know on the paper population is 4x bigger than America's—that just like a huge difference—but you go visiting the cities, you just see that more tangibly. Um, there's there's a bunch of thoughts on the architecture. There's a bunch of thoughts on—I mean, the thing we're especially curious about is like what is going on the political system, what's going on with tech. People I talked to in investment and tech where did seem quite gloomy there because the 2021 tech crackdown has just made them more worried about—you know—even if we fund the next Alibaba, will—will we—will that even mean anything? Um, so I think private investment has sort of dried up. Um, I don't know what the mood is now that Deep Seek has made such a big splash. Uh, whether that's changed people's minds. Um, we do know from the outside that they're killing it in specific things like EVs and uh batteries and robotics. So—um—yeah, I I I I just think like at the macro level if you have 100 million people working in manufacturing building up all this process knowledge that just gives you a huge advantage. Um, uh, and you just like—you can go through a city like Hangzhou or something and you like drive through and it's like—it—you understand what it means to be the world's factory. You just have like entire towns with hundreds of thousands of people working in a factory. um, uh, with—and so the scale of that is also just super shocking. Um, I mean, there's just a whole bunch of thoughts on many different things, but with regards to tech, I think that's like what first comes to mind.
You also spoke, um, recently about this limit of uh compute and energy. And one of the things that's interesting is we even spoke in this conversation about it that if you think about who's going to like—if you're going to have nation-states allocate compute and energy to AI, seems like China is in much better position to allocate more of that than the US. Is that the right read?
Yeah. So, they have stupendously more energy. I think they're what 4x or something. Uh, I don't have the exact number, but that sounds directionally accurate. Yeah. Um, uh, on their grid than we do. And what's more important is that they're adding an Americasized amount of power every couple of years. Is it might be more—longer than every couple of years. Uh, whereas our power production has stayed flat for the last many decades. And given that power lies directly underneath compute in the stack of AI, um, I think that would just—that could just end up being a huge deal. Now it is the case that in terms of the chips themselves, we have an advantage right now, but from what I hear, SMIC is making fast progress there as well. And um, so yeah, I think it will be quite competitive honestly. Um, I don't see a reason why it wouldn't be.
What do you think about the export restrictions? US not exporting the top-of-the-line GPUs to China. Is it going to make a difference?
I think it makes a difference. I think, um, good—good policy. Yeah. I mean, it—it—so far it hasn't made a difference in terms of—um—Deep Seek has been able to catch up significantly. I think it still put a wrench in their progress. Um, more importantly, I think the future economy once we do have these AI workers will be denominated in compute, right? Because if compute is labor right now, if you just think about like GDP per capita because the individual worker is such an important component of production that you have to like split it—split up national income by person. Um, that will be true of AIs in the future, which means that like it'll be like compute is your population size. Um, and so given that for inference compute is going to matter so much as well. I think it makes sense to try to have a greater share of world compute.
Okay. Uh, let's let's end with this. So, this episode's going to come out a couple of days after this—uh—after our conversation. So, hopefully, uh, the predict—this—what I'm about to ask you to predict isn't moot by the time it's live. But uh, let's just end with predicting when is GPT-5 going to come. We started with GPT-5. Let's end.
Well, a system that calls itself GPT-5 or—
Yeah, OpenAI is GPT-5. This all depends on like what they decided to call GP. There's no law of the universe that says like Model X has to be GPT-5.
No.
No. Of course. Like we thought that the most recent model, but I'm c—just curious specifically like we talked a little lot about how like all right, we're going to see their next big model is going to be GPT-5. It's coming. Do you think we're ever going to like—Well, obviously we'll see it, but this is—it's not—it's not a—like a—not a gotcha or a deep question. It's just kind of like maybe like when will the next big model come out?
Sure.
No. When are the—when's the model that they're going to call GPT-5 going to come out?
Um, November.
I don't know.
So, this year.
Yeah.
Yeah. But again, I like—I don't—I'm not saying that it'll be like super powerful or something. I just think like they're just going to call it the next one. You got to call it something.
Darkesh, great to see you. Thanks so much for coming on.
Thanks for having me.
All right, everybody. Thank you for watching. We'll be back on Friday to break down the week's news. Again, highly recommend you check out the Dwaresh podcast. You could also find the Substack at the same name and go check out Dwaresh on YouTube. Thanks for listening, and we'll see you next time on Big Technology Podcast.