Transcription
The sort of scary open secrets in the AI industry right now is that right now that is kind of just a hope. It's not something that we can be at all confident in and in fact there's lots of evidence and arguments that we're not on track to achieve that.
Current AIs for example will often lie to people or they will like be tell them to do something and they go do something else and then pretend that they did it. So it's an inherently difficult problem to make something that's super intelligent and also has the values and virtues that you want it to have and it doesn't seem like we're on track to solve that problem. Also it seems like the sort of problem that you could think you solved when you haven't actually solved it. That's a big reason why this is scary.
So for all those reasons it's possible that we'll end up essentially creating a new species that ends up ruling the world and then maybe we go the way of other extinct species in the past that were outcompeted by humans. That's one possibility. There's many more. I wouldn't say human extinction exactly. I would say something like 70% chance that this goes horribly wrong like human extinction but that's just one of several possibilities.
That's Daniel Kokotajlo. He used to sit inside OpenAI forecasting where this was all headed. He quit, walked away from $2 million just so he could tell you what he saw. He's not selling you a product. Here's what he's warning you about.
You lost $2 million for not signing an anti-disparagement clause which would mean you could speak you couldn't criticize the company.
Yes. Well so I got to keep the money.
Oh you got to keep the money.
What happened was after I had left, said my goodbyes etc. I got the exit paperwork and it included this clause that said you basically have to agree not to criticize the company again. And also a clause saying you can't tell anyone about this. And so I thought that was kind of rich coming from a non-profit that's supposed to be you know for the benefit of all humanity. So I didn't sign it and if you don't sign you don't get to keep your equity. So your compensation you know what they pay you is a bunch of money and then also a bunch of stock basically. But then they had this stuff in the contract that they get to yank back your stock if you don't sign this thing. And my wife and I you know we're upset about this. We talked about it for like a month or two, consulted some lawyers and then ultimately decided to just refuse to sign.
Which would you would have lost $2 million.
That's right. Which was like 80% of our net worth at the time. Fortunately, it didn't go the way we expected. It blew up basically on the internet. Like when people heard that we had done this and that we said no, it became like this huge scandal employees at the company started like asking questions in Slack and like asking leadership like wait, what? Like why are you going to take away our equity? What is this? You know, cuz a lot of people hadn't really noticed this before. It had been whispered about, but it hadn't been sort of like a thing that most employees knew about. And so they backtracked and they said never mind, never mind. We'll change the paperwork. You can keep the equity. It's fine.
And Sam Altman came out and said he was embarrassed that he didn't realize this was going to
Yeah, he had no idea apparently.
You don't believe him.
I think he probably knew. And if he didn't know, then people close to him probably did such as his head lawyer.
Why did you decide not to take the $2 million?
I mean
Most people would have I think.
It's true. Most people would have and most people did. And you know, money is nice, but like it's not the only thing, you know? Sometimes it's good to take a stand on principle. I also became a bit more disillusioned with the AI industry. So OpenAI, Anthropic, and DeepMind all had these sort of founding narratives of like yes, these risks are real, but we've thought about them and we're going to try to handle them responsibly and that's why it's important for us to keep doing what we're doing. And I increasingly came to think that these were rationalizations to justify what they were doing rather than sort of like deeply guiding their actual behavior and that when push comes to shove, they'll follow their incentives rather than do what's actually good.
So think about that. A man walks away from 80% of his net worth because a company built around the words for the benefit of all humanity tried to buy his silence on the way out. He only got the money back because the internet found out and it turned into a scandal. If they hadn't been caught, he loses $2 million for refusing to shut up about what he watched happen inside that building. That tells you everything about how much these companies actually believe their own mission statements. Listen to this.
So right now they're focusing on automating coding. They're taking their AIs, they're making them bigger, they're training them for longer, and they're especially focusing the training on getting them to be good at autonomously writing and editing code, because that will help the companies go faster, right? If they can automate the code, then they can do their own work better and faster, and accelerate progress. The next step, which they've already begun, is to look at the rest of the research process as well, coming up with ideas, analyzing experiments, communicating those results, all the other part of the research process. They're trying to figure out how to train AIs to be good at those as well, so that they can have AIs do the entire [music] thing autonomously.
When you say do the entire thing, what do you mean do the entire thing?
So, like Anthropic and OpenAI in particular are trying to automate themselves. Like they're trying to make it the case that they don't really need human employees anymore. They just have a giant army of AIs that's churning away, doing all this autonomous research to make better AIs, to train the new AIs, put them in charge, so they can make even better AIs, and so forth. And of course, not just >> [music] >> happening internally, but also like interfacing with the world, right? Like going out and talking to people, collecting the data, setting up the training environments, >> [music] >> doing the business deals, and so forth. Like they're trying to automate all of that. The reason why they're doing this is because they're trying to get to a position where they have AIs that are superhuman at everything, superintelligence, and they're trying to get there before their competitors do. Needless to say, this is incredibly dangerous, I would say, you know? And in addition to being dangerous, it's a power grab, right? Like if they actually succeed at this, then they'll be sitting on top of this army of superhuman AIs, that will give them immense leverage over all sorts of other actors in the economy. [music] In so far as they can work out something with the presidents and, you know, integrate it into the military or whatever, then that would give the US immense [music] hard power over all of the countries, right? Obviously, nobody knows exactly when this is happening.
A friend of mine who knows some of these people sat me down once upon a time in London. He's actually said this a few times to me, but I remember one particular conversation where he says that some of these AI CEOs predict the probability of extinction at being I think he said 7%. I don't know why I have that number in my head but I remember being less than 10% and the point he was making to me was that even if it was 1% like if there was 100 buttons on this table now >> Yeah. >> and one of [music] them would end the world. Would I dare press any of them you know?
No.
I wouldn't press any of them but he made the case to me that these AICO's are very smart and they understand super intelligence and that they think actually if there was 100 buttons on this table right now maybe 10 of them could end the world. I've heard you say I think it was on the Daily Show the interview you did you said that you think there's a 70% chance of human extinction due to AI.
I wouldn't say human extinction exactly. I would say something like 70% chance that this goes horribly wrong like human extinction but that's just one of several possibilities. But yeah basically like for example possibly the AI's take over and then don't actually kill everyone you know? Maybe they do something else. Like just cuz they've taken over doesn't mean they're just definitely going to kill us right? They might but they could do something else. So that's why I don't usually say like 70% chance of like actual human extinction but 70% chance of like something like AI's taking over. Some sort of very big catastrophe like that.
It could lead to human
I guess I got two points there which is you've been around these CEO's I mean you've worked for Sam Altman at Open AI before you quit. Do you think that they think there's a chance of human extinction?
Yes. The plan these companies are running right now is not to build helpful tools for regular people first. It's to automate their own workforce hand the keys to an army of AI researchers and let that army build a smarter version of itself over and over until humans aren't needed in the loop anymore and the man who spent years forecasting this for Open AI is telling you straight up 70% odds this ends in something close to human extinction. That number should not be something we're this calm about. Let's continue.
If AI does in fact get incredibly powerful that's going to change the balance of power between nations. That's going to disrupt a lot of things. That puts us at increased risk of crisis more generally. Another one what about those jobs? You you you're going to lose your taxi job but not just a taxi driver everybody. There There be a few exceptions like people whose jobs for legal reasons are only allowed to be done by humans, but for the most part, everybody should be afraid that their jobs are going to be lost, even if we manage to avoid all the other problems.
What skills should people {slash} students focus on over the next 10 years?
Like imagine if you were someone living in Mexico in like 1500, and then you hear that like the conquistadors are coming. You could be asking yourself like, okay, well, what sort of job should I be switching to to like survive this transition? But like you have a lot more to worry about besides that. But yes, I think that I would say that like if we manage to avoid the loss of control problem, and we end up with humans still in charge of the AIs, and humans can like say what the AIs goals and values are supposed to be, even as they become much smarter than humans and AIs as they run the whole [music] economy, then probably there will be regulation that protects some areas, and you can try to guess at what those areas might be. Maybe stuff that's more like judges, potentially.
What about podcasters? Be honest.
Probably not podcasters. Stuff like, you know, being a nanny, maybe, right? Like I think that even if there's a robot nanny that's like really, really good, I think a bunch of people might prefer to have an actual human because they might be creeped out by the idea of a really good robot nanny. So you can sort of reason like that. There's also like stuff that might be legally protected, like maybe judges, for example, like are going to be legally required to be humans and not robots.
Some people say though there's going to be so many jobs created that we can't foresee right now, like there was in the Industrial Revolution or the Internet boom or whatever.
The problem with that is that past technological advancements have been more narrow. They've like automated some things, but not everything. But we are talking about a hypothetical future situation in which everything gets automated. So [music] there isn't any new job that you could do that the AI couldn't also do, except if it's like protected by regulation or something. That's also a thing. For example, right now there's this sort of like cycle where the AI has learned to do a certain thing, like write copy, or draft code, or like [music] debug something. And then humans who used to do that thing switch to managing AIs, or switch to doing the other stuff that the AIs can't do. And that's why there's been this dynamic historically of new jobs opening up and people flooding to them. But if it gets to the point where the AIs can do everything that humans can do and better and faster and cheaper, then whatever that new job is that you might have switched to, like the AIs can switch to that, too. And they'll already be be better at it than you.
Guys, he's comparing losing your job to the conquistadors showing up at your door. That's not a throwaway line. That's how he actually sees it. And the honest answer he gives about which job survive is basically nothing. Maybe a judge, maybe a nanny people don't trust a robot with. Everyone else, including the podcasters interviewing him, gets swept up in the same wave. This isn't some far-off industry getting disrupted. It's every job at the same time once the automation actually kicks in. Now, listen to this.
Jeff children?
Yeah, we have two children. It's kind of sad. Like, I think that one way or another this will probably all be over by the time they're old enough to join the workforce. So, I don't think they'll ever join the workforce.
When you say this will be all over by the time they join the What do you mean by this will be all over?
So, [music] these milestones that I described, like AIs automating their research, AIs getting superintelligent, AIs [music] then exploding onto the economy, taking the jobs, building robot factories to build more robots to build more factories, etc. GDP starting to go vertical. That sort of thing is what I mean. Like, all of those events transpiring. Maybe there's like, you know, 10, 20% chance or something that hits a wall and and none of this comes to pass, even if you don't do anything.
I noticed that when I asked you if you had kids, your demeanor changed quite considerably.
Well, it's Yeah.
[music]
It's like you dropped into a different state. Obviously, that's been central to the rumination that you've been experiencing.
Well, it is a sad topic, right? Like, when I had kids Like, the reason to have kids is in large part about the future, you know? Like, it's not just like a cuddly thing to have with you in the moment. It's cuz you have all these hopes and dreams about how they'll grow up and how they'll go and do their own thing and be their own person and stuff. And [music] because of what's happening with AI, I think a lot of those dreams are in jeopardy.
Presumably, you still would have had kids.
I've actually flip-flopped on this occasionally. Yeah. Ba- Basically, the top-line answer is I'm not sure. The My first child, we had her when we were She was born 2019. Yeah. So, this is before my timeline shortened a lot. So, at this point I was interested in AI, I was tracking the field, I was making forecasts, but I didn't like actually expect it to happen soon, you know? And then this caused like when I did start thinking like, "Oh my gosh, it's going to be happening like real soon, like by 2030, you know?" That caused some reconsidering, and so I basically told my wife like, "Let's not have any more kids. It's too uncertain, you know?" But that turned out to be really hard because especially for my wife, like we already had one kid and like no siblings. So, eventually I sort of gave in and was like, "Okay, well, you know what? We already have one. It's going to be all right. Like maybe the future will be good, and even if it's not like well, we're all in the same boat together."
It's quite chilling what you're saying. It's chilling because you know more than me. And if you're at home saying to your wife, "Listen, maybe we should pause on having more children and building a family because of what's going on with AI."
To be clear,
Is it
Yes, I mean yes, it's very concerning. I am chilled. This is bad. This is what I've been saying. I hope things go well. I think things might go well. I think that there's a lot we can do to like steer things in a better direction.
So, this guy who forecasts this industry for a living looked at his own daughter and quietly accepted that she probably won't grow up into a normal working life. He's not being dramatic for the camera. This is someone who told his wife to stop having kids because the future felt too unstable to bring more people into it. When the person who studies the timelines for a living starts talking like this about his own family, that's the moment you stop treating this as some distant sci-fi conversation.
Why should the average person care?
High-level thing is absolutely everything is going to change for the whole world and including therefore for them and their families. Could change for the better, could change for the worse depending on the details of how it's done. So, for example, everyone could die. This is the classic loss of control scenario or one version of it. If we do build these super intelligences and we use them to automate all the jobs and [music] we put them in the military and we, you know, have them giving advice to politicians and so forth, they will eventually have accumulated enough real-world power that they don't need humans anymore. And they're smarter than us, they're more strategic, etc. At that point we sort of have to hope that they are virtuous, that they have, you know, the goals that we wanted them to have, the values that we wanted them to have, etc. And the sort of scary open secrets in the AI industry right now is that right now that is kind of just a hope. It's not something that we can be at all confident in, and in fact there's lots of evidence and arguments that it we're not on track to achieve that.
what, must be almost coming up to 15 years thinking about this stuff. If this here was a button, and if you press that button, your plan S would occur, and it would shut down every data center that is currently training a frontier AI model for good. There would never be any other AI labs working on these problems. Would you press that button?
I was about to slam it until you said for good.
Oh, [laughter] okay.
Like I think if it was a sort of temporary shutdown, I would totally slam that button. Because we are not ready to do this, you know? [music] Like civilization is not ready to have these companies automate themselves and then get smarter and smarter and then have the superintelligence. Like no, there's a bunch of reasons why that's really dangerous.
Now, the entire plan for how humanity survives this comes down to a hope, a hope that a system smarter than every human on Earth happens to end up with the values we wanted it to have, even though nobody can actually check. And when he's handed a button that would shut the whole thing down right now temporarily, he says he'd slam it without hesitation. He only hesitates when it's permanent because he still believes there's good on the other side of this if it's done right. But by his own estimate, we're not doing it right. We're racing three companies, a handful of CEOs who don't trust each other, building something none of them fully understand, betting the whole species on the hope that it turns out fine. If this made you uneasy, subscribe, because whatever happens next is happening faster than any of us are being told.