Everywhere you turn right now, it's about AI. Every headline, every news segment, it feels like the story of the decade, and nobody can escape it. So, of course, you'd feel threatened. Is your job safe? Will your industry even exist? What does this mean for your kids, your future, your place in the world? Everyone's asking the same questions.

But the louder the noise gets, the harder it is to know who to trust. Is this actually real, or is it one of the greatest hype machines of all time? That's why this is one of the most important conversations you'll ever watch. Because if there's one voice to trust right now, it's Tristan Harris. You know him as the central figure in *The Social Dilemma*. The Netflix documentary seen by over a 100 million people that changed how the world thinks about social media. Before that, he was inside Google raising the alarm on how technology hijacks our attention and rewires our brains. And today, as founder of the Center for Humane Technology, he's become the most trusted voice on how tech is reshaping our future. He's not just another commentator. Rather, he's the one person who can show us what's really coming, what must be prevented, and what it all means for us.

If you're new to Mighty Pursuit, we exist to help you make sense of the whole person, mind, body, and spirit. Not as vague wellness clichés, but as deeply interconnected parts of your everyday experience. We believe putting the whole person back together is the key to living well. And in today's episode, we zoom into our service category, looking at how AI could spiral into a humanitarian crisis and what justice demands of us to protect humanity. But it doesn't stop there. AI also raises massive questions about our higher purpose and about the future of what work will look like for all of us.

This is a rare opportunity to hear from a voice unlike any other. Tristan Harris has sat inside the rooms with some of the most powerful people on the planet. He's briefed Congress. He shaped the policy conversations that define regulation. He's seen firsthand the potency of this technology. And now he's here to show us what's really coming and what's at stake for all of us.

So, I want to take us back to 2020 when *The Social Dilemma* came out. You were at the center of it. And for a moment it felt like the world finally woke up to how social media was kind of rewiring our brains and impacting our culture. So over the course of the last 5 years, over 100 million people tuned into that, which makes it one of the most successful in Netflix history. So my question is twofold. Looking back now, 5 years later, how much of an impact do you think the documentary really had? Did it shift the trajectory, or is it already too late? And the second part of that is, how does that speak into what's happening right now with AI?

Yeah. Well, obviously, since *The Social Dilemma* came out, we've totally resolved all these problems. We replaced the division algorithms of social media with ones that rewarded unlikely consensus. And so now, instead of scrolling and seeing examples that made you pessimistic, anxious, and depressed about everything every day, now you're scrolling and whenever it's about politics, you see examples of, uh, unlikely consensus and solutions to our problems. So it actually changes the psychology of the world. We, um, you know, we actually instilled a simple rule, uh, in the technology industry, which is, you're only allowed to ship products that your own children use for eight hours a day. That cleaned up about 90% of the problems. Uh, schools and curriculums across the country, you know, made *The Social Dilemma* and a humility curriculum for technology required. So we basically, in order to graduate from engineering or computer science, you had to study all the places that we'd screwed up technology in the past, whether it was microplastics and forever chemicals, or the thalidomide, or, um, you know, leaded gasoline that dropped, you know, collective IQ by a billion points. So now everybody who is building technology suddenly is now massively aware of all the problems that can emerge, which created a whole new humble culture around delivering technology to the world. Um, there was a trillion-dollar lawsuit for the damage that social media caused. Uh, so, you know, just like big tobacco, we realized, Michael Milken, the great capitalist, said social media has created about a trillion dollars of damage to social capital from social trust, relationships, developmental problems in children, attention spans, critical thinking, all the mental health care costs. And once we realized that that was causing trillions of dollars of damage, that led to the biggest lawsuit in history, which then had these companies pay out billions of dollars to re-up and fund investigative journalism and and local news around the world. So we had democracies going from backsliding around the world to forward-sliding all around the world. I'm just obviously being facetious, but this is an example of if we had woken up and really taken these problems seriously, we would have done things like that. We would have changed the culture that developed technology, the culture that receives technology, having a tech-critical culture. We would have had, uh, different laws that bind, uh, these bad races that we're in. Uh, so with social media, that race was the race for engagement and attention, which created the race to the bottom of the brain stem for who can go after the youngest users earliest and then hijack their limbic system with dopamine and all of that. Um, there is a way to completely reverse almost everything that has happened, but it would take a very deep, comprehensive approach that sees the incentives, the culture, the economics, the law, the venture capital, and all these forces that funded this societal catastrophe.

So, I think, you know, later we're going to get into AI, but I've learned to speak about social media in this kind of alternate positive history because otherwise, it's easy to get depressed at how little has changed. But I will say that I do think the film clarified the problem to billions of people. And I think that now it's not just the 150 million who saw the film. I think it's like billions of people now recognize and acknowledge that we made a huge mistake with how we rolled out this technology. And I deeply hope that that mistake is recognized when we talk about how do we make different choices with AI this time.

Yeah. I mean, it's interesting because that document came out. Facebook came out in 2004. MySpace, 2002. So you already had like 16 years of basically social media history by the time that documentary came out. I mean, ChatGPT has started to become widely used over the past two years. So we're kind of at the early innings. Well, and just to name it, it took 2 years for Instagram to get to 100 million users. It took ChatGPT something like 2 months to get to 100 million users. We are rolling this technology out faster than we have rolled out any other technology in history. What does it say about our culture or our leaders or our politics that there could be such awareness of the reality of what's going on and nothing, I won't say nothing, but there could be very little movement after that awareness comes into full spectrum. What does it say about our culture and our leaders that we've done so little, uh, with the with the knowledge that we have?

Yeah. I mean, it's all about whether or not we have rules that govern the road, rules of the road. If we don't have any rules, then the companies just follow the default incentives that are in front of them. So if ChatGPT launches and gets 100 million users in a very short period, if I'm Anthropic or Google or, you know, Microsoft Copilot, do you think I'm just going to hold back the AI capabilities that I have, or am I going to deploy my AI even though I think it's not actually safe for people yet? If they're deploying it, then I have to, and I have to also get 100 million users. Otherwise, Wall Street's going to punish my stock price, and I won't be able to get the same capital expenditures and keep racing for for AI. So, obviously, there's a cultural problem in us not collectively responding and uproar when this happened. Uh, I would say, but the deeper problem that the culture has to be in service of is, are we changing the incentives for something different to happen? By incentives, I mean the business models, the profit motive, the market share dominance. Um, because I, I hope in a moment we get to deconstruct like, what is the actual thing that is driving this race? Um, because by default, you would think that with a power, with a technology this powerful of AI, uh, we would be releasing it more carefully than we've rolled out any other technology in history. It's more powerful than all the other technologies. It's important to say why, by the way. Intelligence is the foundation of all science and tech development. So like, if you, if you make an advance, um, in rocketry, the the technology of rocketry, that doesn't advance the technology of biology. If you make an advance in the technology of medicine or energy, that doesn't advance rocketry. But if you make an advance in intelligence, which is all of the things that give rise to science and technology development, the, you know, writing equations down, inventing new materials, um, doing a search for new kinds of chemistry. If you automate intelligence, you just accelerated all scientific and technology development, meaning you get better rocketry, you get better medicine, you get better biology. Not necessarily for good, by the way. You also get better weapons, you get better bioweapons, you get better, uh, dangerous chemistry. Uh, so AI is unique in its power and, as I said, in a recent TED talk, it dwarfs the power of all other technologies combined because AI is so omni. It's not just dual use. A hammer can be used for good or for evil, but AI actually accelerates AI. Like that's important to get, right? So if I invent intelligence, what got me the AI? Well, I had to write some code. Well, I can use the AI that I created to look at the code that I wrote and make it 20 times more or like 20% more efficient. I can, how did I make AI? Well, I used GPU chips from Nvidia that are, you know, really hard to make and design. Well, I can use AI to say, hey, can you automate the chip design and make more efficient chips? Yep, you can do that, too. Can I use AI to generate more training data to accelerate the process of training AI? Yes, I can do that, too. So, AI applies to like everything. What's that? It applies to everything. It applies to everything. And so it's really important for people to get that AI is distinct from all other kinds of technology. Um, a simple line is, nukes don't invent better nuclear weapons, but AI invents better AI.

Which is wild to think about because if you consider the impact that social media has had on our culture and our world over the past 20 years, I mean, it's absolutely massive. The way it's shaped public discourse, the way it's shaped our brains. I mean, even the way it shapes how business is done now. I, I was commenting with my friends a couple weeks ago, it was like, you know, you used to have magazines, you used to just like hop on TV like once at night, uh, maybe 30 years ago, but now everything is found out through social media. And then you zoom out and it's like, wow, AI is so substantially bigger than that.

Yeah. If you were to, if you were to think about that and dwell on it a little bit deeper, what does AI threaten to like destabilize about our our culture and population at this current pace?

So AI affects literally everything. And just to name that, um, I, I had a big dinner, I host, we hosted last night and had a bunch of really incredible people at the table. And I started by saying, the problem with talking about AI is that all conversations almost end up dysfunctional by default because it touches jobs. It touches bioweapons. It touches, uh, politics and whether democracy can exist. It touches truth and whether people will know what's true because of deepfakes. It touches fraud. It touches, uh, cyber weapons. It touches warfare, autonomous weapons. And I'm not saying this to try to overwhelm people. I'm just naming that there's sort of a, a space of impact. And there's many different domains of impact, more domains than I think any other kind of technology. So, um, I think social media, as massive and significant and sort of world system overturning as it was, which it really did do, I think if you really appraise of the consequences in a full way, AI is going to be far bigger than that. Um, and if you think about how much social media changed just by affecting something core, which was our attention. If you just imagine the sort of zoom out, you see the ant colony of humanity, just 8 billion ants doing their thing. And you sort of put social media into that equation. And it's like all of the ants just completely changed everything that they were doing, everything that was top of mind in the ants, all the decision-making of the ants, the relationships of the ants, which ants they followed, a whole new political structures and power structures where we're electing people based on the the popularity on social media, which tends to sort for the dark triad traits of narcissism, uh, and and the ability to sort of just charisma and social and the ability to just get attention and popularity. So it changed power structures, it changed relationships, it changed childhood development, it changed democracy, it changed the flows of information, it changed what we trust. It, it made it so that most people get their information from influencers rather than from things that are coming more through slower processes that came from institutions. I'm not trying to romanticize the institutions and say that they were perfect, but we just did this sort of Indiana Jones swap with the thing that was conferring power and authority from the slow-moving, merit-based system of institutions for whatever their flaws, to now suddenly it's like, if I'm really good at playing the likes and followers game and grabbing attention, I'm suddenly I've got millions of followers and I'm influencing what people think. And you're the expert. And you're the expert. And so that's not to say that there are not amazing experts who were better experts than people in the institutions that have come up through that system, and we should celebrate that effect of social media. But if you zoom out and look at it more systematically, what are most people seeing and consuming most of the time, and it's not more wisdom and more expertise, it's just more popularity guiding the sort of, you know, joystick of where all this goes. And this was foreseeable. Um, I'm giving a lecture this week in in New York at the, um, Alfred Korzybski, uh, memorial lecture. He was the founder of general semantics. There's a great thinker by the name of Neil Postman who studied media. He wrote a book in 1983, I think, called *Amusing Ourselves to Death*, and he worried about technology and media and television in particular, that television was not about informing people. It was about entertaining people. But to understand that, you had to look at the medium of television. What does the medium of television reward? And if you listen, if your listeners know the phrase, "The medium is the message." The message is like this distracting piece of meat while the burglar, this is his line from the book, the burglar kind of takes your attention away with with the structure of the medium. Um, so all that is to say, I think it's very easy to underestimate the impact of something. And I say this as someone whose friends started Instagram. Like, I introduced the co-founder of Instagram to his wife at my, you know, apartment, uh, years ago in San Francisco. I went to school together at Stanford. I know these people who built these products. They're incredible, lovely human beings, and I still consider him a person that I love and a friend, even though we haven't talked actually the last probably 8 years because of the work that I've been doing. Um, and it, it shows you that it's not about the the goodness of the people, it's about the incentives that that guide the technology. And, and even he and I think many people who were building these technologies were, I think frankly, in denial of the consequences of what this would do. That's what I experienced is you talked about some of these problems, and the answer was, well, this is just a moral panic. This is just reflexive fear of new technology, which is what we always say, you know, every time there's a new technology, there is reflexive fear of it, and we should acknowledge that the moral panics do happen. But looking at it really comprehensively, did social media really deserve that level of attention and critical thinking? And I think it actually deserved even more than I think we're still lagging behind fully appraising of that the full, uh, balance sheet of costs to how society works and whether we're going to have a sustainable society.

Yeah, I think the incentives piece is so important to understand. And part of that is a motivating factor for for our company itself. It's it's the question of why do we not build things in the vision that you just articulated upfront with like the alternative history with social media? Why do we not build things with, oh, is this going to be healthy for people? Is this going to be healthy for our relationships? Is this going to be safe? Is our food supply safe? Is this going to create more thriving in people's bodies and and longevity and and all these other things? Or there seems to be a incentive that just goes down the route of greed or or money as a default or or growth or, I don't know, success. There's so many different things that it seems like people are automatically wired to or bent to. How do you make sense of that? Why we don't prioritize the health of a population first?

Well, first, it's important to say it's not that it never happens, but it happens increasingly rarely. I think there was an incident in the 1980s where, um, Tylenol had a had a problem where I think someone had basically figured out how to put poison inside of the Tylenol. I can't remember how this this all went down, but they had to invent the tamperproof, the childproof, like, uh, uh, you know, the thing where you had to open the the bar differently. Um, so basically, when this when this poison was discovered, um, I, I need to make sure I get the history right. People can look it up. But instead of lying and saying there was no problem, everybody should keep buying Tylenol, um, the CEO of Tylenol admitted the problem. For a moment, their stock price did sink massively, but then they actually came out on the other side and said, "This is what we did to solve this problem." And in the long run, they actually did, uh, their stock price did come back, and they did get the trust of customers. If you compare that to today, when you're talking about incentives, it's like, can Facebook afford to make a change where they just say, "We're not going to go after 13-year-old users anymore. We're just gonna only go after 18 years old and up because that's what's good for people. That's what's safer for people." Um, well, first of all, if they can't stop TikTok from going after 13-year-olds or 10-year-olds or 9-year-olds, then Facebook's market cap and stock price is based on competing with this other company for a finite supply of human attention. So, they have to go after the young users, uh, in so far as the other companies are. Now, they could say, alternatively, this is wrong. We should not put young people's brains into a product that was totally designed in a bad way for them. But if they said that, they'd almost be admitting that they have a product that's not designed for people and that's good for people in the first place. But they could take a leadership position and say, "At least this isn't good for kids." And we think it's better for the whole world to not have this for kids than to have everybody racing to get kids addicted early and then screw up their brains and critical thinking and their continuity of attention and make it so they can't hold a thought and give everybody ADHD and freneticism. It's really sad when you understand the incentives. And I'm not saying this, by the way. You might hear this as some kind of like doomer conversation, and I don't at all.

Okay, good. Cuz I, I want people to hear that it's like, all of this is coming from the love of the world that we actually want to exist. And I think there's a temptation to often steer away from what we don't want to look at because it's uncomfortable. But it's like, if you look away from your shadow, that doesn't mean that your shadow goes away. Like your shadow still, you know, that which you look away from still happens. And I feel like with technology, with social media, and now with AI, it is really forcing us to confront our existing misaligned relationship with technology that we haven't necessarily been great stewards of our technological power. And if we look at that, if AI is sort of inviting us to look at that, then it gives us a chance to say, what does it look like to deploy technology in a wise, careful way? Because that's what it's inviting us into. Because if we don't take that invitation and we just put a booster rocket on the back of our misaligned technology rollout process, we're going to get exponentially more of these crazy consequences that are the unintended consequences that we didn't foresee.

Yeah. I mean, if you look at the Forbes like billionaires list, which is which is really interesting, versus previous generations and previous centuries of of when you look at wealth, I mean, it's it's absolutely absurd when you consider what you just said about the ants and the attention and everything else, how incredibly rich people became and and even the the presence of trillion-dollar companies. I mean, these companies are so much, they're literally the most powerful people now in the entire world. Literally, you have like basically five people who could with with a snap of their finger could destabilize like our world. And that's how powerful these technologies are. And and that's why it's confusing for me. If so, with your example with Facebook, it's like, okay, we're just not going to target 13-year-olds. Are we going to be maybe we'll be like a $700 billion company instead of a $1.5 trillion company? Right? It, it's like it seems sort of like trivial. Like, okay, still, do you really need more money? Do you really need to have a 1, you know, a $2 trillion market cap instead of a $700 million market cap?

Exactly. Billion-dollar market cap. Yeah. Yeah. So, and then when you think about AI, it's interesting because if like Chat, if if OpenAI had like a stock right now and it continued on the trajectory of where we're at right now, like is that going to be like a $10 trillion company potentially? Like, is Tesla with the robots and everything else, is that going to be a $15 trillion? I mean, we're talking about levels of wealth that are insane.

Yeah. Like literally insane. It's important for people to get that with AI, we actually are talking about like, I mean, Elon Musk in his, I think it was the Tesla board meeting, said that the Optimus Prime robot, the Optimus robot that he's building at Tesla, which is meant to replace all physical labor in the economy, he said the pro, that product alone would create a $25 trillion market cap opportunity for Tesla, which is which is half, I think, the global labor economy, I believe. So you have to really read between the lines like, what is he saying when he says that? He's saying, I'm going to own the economy. Um, people really need to get this is not some kind of, um, conspiracy theory. What is the mission statement of OpenAI? Do you do you know what it is? No. It's basically, our mission is to build artificial general intelligence and be able to automate everything that a human can do. Meaning, what that means is that you will replace the employee you're currently paying to edit this podcast or to do a marketing analysis or the legal analysis for you. Every person in your company, they want you to be able to automate that person's job because then instead of the company paying that person, they pay the AI company. So it's just imagine all these companies that are currently paying, you know, their hundreds of employees and all the dollars are flowing down to them, and then suddenly this new country of geniuses in a data center pops up onto the world map. That's Dario Amodei's face. He's the CEO of Anthropic. So he says like, AI is like a country of geniuses in a data center. So imagine literally a world map and a new country popped up the size of China, but it doesn't have a physical geography and it has a population of 100 million Nobel Prize-level capable digital workers, but they don't eat, they don't sleep, they don't demand healthcare, they don't complain, they don't whistleblow, they work 24/7 for less than minimum wage. So if I, it's funny because we talk about the problem, you know, it's a political topic, but immigration, and that's this big threat for some people in this country. Well, as Yuval Noah Harari, the author of *Sapiens*, says, AI will be like a flood of digital immigrants that are much more capable of taking all the jobs than the small amount of jobs that have been taken by immigrants right now. Mhm. And what that means though, is that all of the money in our economy will start to flow to that country of geniuses in a data center. But it's not a country. It's not a nation-state. It is a set of companies, really, like five companies. You have like DeepMind, you have Google, you have OpenAI, you have Anthropic, and they're building this really sophisticated artificial general intelligence. That's what they're aiming for. And they think they're going to have that within, you know, somewhere between one and the next, people think by the end of this decade, really. Um, which is crazy. And you can debate the timelines and say maybe it's 2 years more than that. But the point is that is an unbelievable amount of change, and it is what their mission statement is gunning for.

Do you believe that? Do you believe that artificial general intelligence actually is possible in the, like, that timeframe?

Um, it is very hard to know timelines. What I'll say is that even in the year 2021, just go back to 2021. If you told someone that we're going to have an AI that can pass the bar exam, the legal exam, or pass the MCAT exam, or be the top 200th best programmer in the world, and we're going to have those AI capabilities in the next like 3 years. I would have thought that you were crazy. Like I would have, does that make sense for a second? I'm just like, so like, there you are in 2021. What have we got going on? So President Biden had just been elected. Social media was kind of the thing. AI wasn't a thing, really. I mean, AI was just narrowly used in all these different applications for like, you know, automating job application selection or automating a loan to someone at a bank. Um, and that had its own problems. But we didn't have AI that could generate entire applications worth of code or invent a cyber weapon or generate, you know, 10-minute long videos or pass the MCATs or the bar exam. And that changed in like 2 to 3 years. GPT-3 came out like 2 years later than that. So just imagine that if in 2021 you could sit in your own mammalian meat suit and look at AI and say, there's nothing that could cause that to happen in 2 years. Here we are in 2025 saying, "Oh, well, there's nothing that could, you know, what's next?" Right? And I feel like it's very easy to have, I think, a, um, a skeptical or dismissive attitude. And with with AI, you want to be right. You want to act before it all happens. You don't want to wait until afterwards. Think about, you asked me at the beginning of this conversation, um, why hasn't more happened with social media since *The Social Dilemma*? And the real answer is that we waited until after it had become fully entangled with everything in society. Like your identity is your Instagram account. People actually message each other with Instagram. So for me to like not use Instagram is to shut down my phone. I can't shut down my phone. We waited too late for us to, um, uh, regulate or set guardrails on social media. And you only get one period before entanglement of a new technology.

Why would, if you're an engineer at Meta or Google or whatever, why would you create this technology if you know it's going to replace you? And it is replacing you because like, I think Microsoft over recently had released 30% of their code is being written by AI now. So, and Dario from Anthropic said just last week, I think in DC, that 70 to 90% of their code is being written by their AI system now. Well, why would, why would you do that as an engineer? Create something that would just replace you and then you're fired?

So, there's a lot of things we're diving into here. This is all good. Um, so first of all, they're not yet actually letting go of that many employees, even though they're writing way more code way faster. Um, there still is this open question for humans needing be able needing to be able to review the code that's written by AIs, but eventually the AIs will get so good that they won't really need the human review. Cuz all the time that humans are reviewing the code right now and then going back and forth with the AI, well, that's going into the training data for the next AI. So each time people who think that, oh, the AI is not that good and I have to still look at it and then course correct it, well, guess what? Every time you're doing that, you're training the AI company's AI so that the next one incorporates all of your human oversight. So it gets better and better and better. And so people need to realize that if they had, if they sit here smugly saying like, "Well, I'm going to keep my job." Like all of the ways in which we are currently overseeing the AIs are going to be used as training data to to make that kind of collapse. That's one. You ask the question, why are they even trying to automate programming in the first place when the engineer's job is programming? Well, what's important to know is that the AI companies, why aren't they racing to solve climate change? They said that AI is here to give us cures for cancer and solve climate change. Well, if that's true, why aren't the AI companies just only racing to solve climate change?

Yeah. You say, what's that? You, yeah. No, I agree. Yeah. Yeah. Right. Because it seems like, okay, well, if you're going to solve those problems, why don't you make that the first thing you solve? And the answer is that their real goal is not to try to address those problems now, but to race as fast as possible to the to getting artificial general intelligence or even superintelligence, which is superintelligence is defined as an AI that is more powerful and more intelligent than all of humanity combined. The reason that they're racing to that is that that thing will be able to address all these other problems, they think, and they'll help them own the world, frankly, and dominate the world. And the way you get to these more advanced AI systems is you automate AI research. So just take a second. Here you are at OpenAI. They've got hundreds of employees. Some percentage of them are doing research on AI, right? Yeah. They open up Twitter, they read the research papers, they like think about them, they generate a new hypothesis, they write a new experiment, they write it up in English, then they write the code for it, and then they run the experiment and then they see if they get a result. And if they get a good result, they incorporate that, and the AI gets better. So that's AI research. But for each of those steps, you can have the AI do that work for you. So you have the AI that reads the research papers, summarizes them, what's what's five new insights or hypotheses that we could form to run experiments? Then the AI thinks about those experiments. Not thinks in a human or consciousness way, but it generates the text for what the experiment might be. Then it generates the code for that experiment. Then it runs the code. Then it writes up the results. And soon you go from hundreds of employees doing AI research at OpenAI to automated AI researchers. And I can snap my fingers and go from one digital AI researcher to fork it and have a hundred million digital AI researchers going at superhuman speed. So this is what's called an intelligence explosion. And if you read or your listeners know the paper *AI 2027*, that paper was about why the companies are actually in a race to automate AI research. That's the prize because if I automate AI research, I can just get way ahead of everybody else. And so because this race is so intense and the race is the dominating characteristic of why they're deploying the technology, they're really in a race to do this intelligence explosion, but that's incredibly unsafe. And people need to understand why.

And maybe we can, we can get into that. If you were to help people understand what AI is currently, as what it means in large language models versus AGI versus superintelligence, and the graph of that, how would you differentiate those three terms so people understand?

Sure. Um, so first of all, AI has been with us for a long time. Um, people have been using AI. When you, you know, look at your Face ID on your phone, it's using some AI to figure that out. You take a photo on your phone, it uses AI to clean up the photo and clean up the pixels. When you, um, you know, highlight some text and say, speak it out loud, it's using AI to generate a voice. We've had text-to-speech and these kinds of things for a long time. When you use social media and it figures out which post, which video to show you next in your newsfeed, it's using AI to make a prediction of what video could I show you next that's most likely to keep you watching. And it makes predictions with high accuracy. And that's why people keep doomscrolling, cuz it's really good at knowing what's going to keep your attention. All of these though were narrow AI systems. It's like very narrow, right? You know, you bought this product on Amazon. Maybe the AI can say, here's five other products that I can recommend to you. That's a very simple narrow AI. That's different than what they're really aiming for now, which is general intelligence. So artificial general intelligence. General meaning, do all the things that a human mind can do. So think about everything that a human can do. You can plan. You can develop a strategy. You can think of an idea. You can ask questions about your own ideas and be recursively in a loop of like looking, looking through your own thoughts. You can make predictions. You can, um, empathize with the person. You can read signals. You can pick up patterns. Um, I know it sounds abstract, but there's a lot of things that a human brain can do and a lot of things that a human mind can do. Um, and AGI is the goal of replicating that entire suite of things because that is unique in that I can replace a human being. I can replace a legal analyst at my company. I can replace a person who's doing copywriting of writing copy for the title of this video. I can replace someone who would have I had to pay to illustrate some graphic for this video. And when I have general intelligence, that gives me something that we said before, suddenly makes companies want to pay AI companies to do their work rather than pay their own employees. So that's why AGI is the, the prize. Yeah. Um, and if I have AGI, again, I can, it's not just that I have this like one digital brain. It's I can snap my fingers and use those huge data centers that are bigger than the the size of Manhattan. They're like 8 miles long or whatever, um, that are filled with these Nvidia chips. Just imagine, you know, data centers that are almost the size of Manhattan filled with these NVIDIA GPUs. Where are they? They're all over the country. They're all over the world, mostly in the US right now, remote places, places where there's nuclear power, places where they build coal plants, wind turbines. I think, um, I mean, sorry, methane, uh, turbines powered by methane. Uh, Elon Musk's one near Memphis, Tennessee, um, has a bunch of methane-producing turbines, uh, next to it. I think that's right. Um, and, um, you can snap your fingers and you can go from this one digital brain to now I have a 100 million of them doing work simultaneously. They're writing code. They're performing analysis. They're agents that are running around the internet, booking flights, paying people for things, writing messages to people on Craigslist, paying people in crypto. Like this is a crazy, it's a crazy transformation of our economy being automated by AI. And that's why AI, AGI is so sought after. But then what you want is there's no reason why intelligence should be capped. People believe they think there's no reason why, you know, human intelligence is the smartest kind of intelligence that we could have. You could have systems that are smarter than human. Think of the the chess-playing AI that IBM made in 2004 that beat Gary Kasparov at chess. That's superintelligent in a narrow domain of chess. But now you have AIs that are going to be superintelligent than all human programmers combined. Well, they could, or all human biologists combined. They could control humans then essentially, if they wanted to. Yes. Because there, the way that people in the AI risk space refer to it is, can you have something that is less intelligent ever controlled something that was 10 times more intelligent than it? No. Let me give you a metaphor because I know this might sound sci-fi to listeners. Let's say you're sitting around, it's a bunch of chimpanzees. The chimpanzees are hanging out and they're grooming each other and taking the fleas off each other's backs and they're drink, you know, eating some bananas and they're like, you know, I have an idea. I think we should make these super smart chimpanzees, uh, that are 10 times smarter than the chips than us. And one of them says, "Yeah, that sounds amazing. Like maybe they'll be able to do all this crazy stuff for us and they'll be able to, you know, benefit chimpanzees and they'll automate and create this world of abundance. We'll never have to go search for bananas again." And then the other one says, like, "No, this sounds really dangerous. Like, well, if we create this smarter than intelligent version of chimpanzees, like, are we going to be able to control it?" And the other responds like, "Well, what's the worst they're going to do? Like, eat all the bananas?" So, just imagine, where do chimpanzees exist in our world now? Or just imagine one sitting in the Manhattan Zoo. Yeah. In a zoo or jungle. In a zoo or a jungle. And mostly we've probably extincted a bunch of the species and made it harder for them to probably thrive in in certain places, and we've taken over the world. Could the chimpanzees even envision gunpowder or airplanes or nuclear weapons or quantum physics or Einstein or computation or iPhones or 5G? They can't envision. They couldn't literally conceptualize what a smarter version of them could create. So, just to ground people in how insane this is, we are the chimpanzees now, and we're sitting around a table doing human things, mostly scrolling on social media and doing nonsense, and we're like about to make this way smarter thing. We are making this smarter thing, and it's already outcompeting us at a number of different domains. I believe it's already beating military generals at war game strategy and planning. And as soon as it outcompetes us, there's going to be a a race for who's willing to automate and have AIs do military planning and strategy versus having humans do military planning and strategy. So you keep zooming out, and it's like every place where there's a human making decisions, as soon as AI is better or more effective, we're doing it faster. There's going to be the mass temptation to swap it out because then, as a company, I start outcompeting the other companies, or as a nation-state, I start outcompeting the other nation-states. So it really is this race to kind of automate, but then we are increasingly handing over decision-making to machines that are not infallible, that make mistakes. The same ones that, you know, generate six fingers on the hand that hallucinate an answer to an image query also hallucinate answers to your questions. Is, well, what happens when they're hallucinating in the middle of a battle plan and they're automating some kind of war in Taiwan now?

So people really can grasp what you're saying. A lot of people are interacting with ChatGPT on a personal basis, and to them, that's what AI has become, or like Google Gemini, or, or whatever they're using right now, Claude. If you were to help them understand the different forms of AI that would be employed or are currently being used right now, what is the full scope of that so people could understand how this would even come to be?

Okay, so when most people hear us talk, they're probably thinking, "Okay, well, hold on. I don't understand what Tristan's saying because I'm sitting here. I've got a blinking cursor of ChatGPT, my baby's burping in the background, and I just ask, 'What do I do if my baby's burping?'" And it just spits out this amazing answer that actually helps me with my my life. That's most people's experience watching this right now. And so when we say that AI carries these like crazier risks like loss of control or unpredictability, we're not talking about the blinking cursor of ChatGPT suddenly deciding behind that blinking cursor to go rogue and do something on its own. What we're talking about is behind that blinking cursor. So imagine like, you, you know, you go behind that sheet of glass that's your computer screen and you go zoom way out to the data center that's this big building with millions of these, you know, GPU chips from Nvidia, hundreds of thousands of them, that are basically collectively operating like a digital brain. So right now, that digital brain responds to you typing a question and hitting return. It pings the digital brain, and the digital brain comes back with an answer. But what I'm talking about is a world where you create agents, meaning like an automated AI that's going to run off and do work for a week and it's going to come back to you after a week. So that digital brain, instead of waiting for your blinking cursor, hitting return, it's just hitting return to its own blinking cursor and its own thoughts, running off and doing a job for a week. You know, "Generate me a brand new TikTok app that outcompetes TikTok, that's more viral, that, um, has automated generated content and that optimizes it perfectly for this audience of people in this country." Boom. It just goes off, writes the code, uploads it to the app store, creates a bank account. Like, we're, we're going to live more in that world. So, I know this sounds crazy to people, and I almost want to stop and just say like, um, not, I mean, I just, for myself, even, it's like, this is really crazy. It's insane. It's insane. Yeah. Like, it's actually just important to say like, we need to stop pretending that this is just okay or normal. This is not normal, and it is not okay. And any regular person who cares about, you know, looking their family members or the people that they love in the eye, or doing the favorite things that they love doing in this world with, as, you know, in some way that or life and connection can continue, like this race is threatening all of that. And it's, you know, again, it's, it's, I think to establish the next thing, we should just briefly give evidence. Um, we have all of these sci-fi tropes of *2001: A Space Odyssey*, you know, and there's the big HAL 9000, uh, Terminator, and they've desensitized people to the idea that like AI would go rogue because we've only seen it in a movie. Like the idea that it would like deceive us or have its own strategy or be self-interested has always felt like science fiction to people. And I want

To say for your audience, like as someone who was worried about social media for many years, I used to be upset at people in the AI community worried about AI risk who were like, "AI is going to go rogue. It's going to scheme against us." I was like, "What are you talking about?"

Because I was actually saying that I felt like there's a lot of AI safety people from this community called Effective Altruism. Yeah. And I thought that they were overlooking this runaway narrow AI that exists right now called social media, which is an AI pointed at your brain, just calculating what to show people, that was wrecking society.

Algorithms. Algorithms. Yeah. And I was like, "This is the one. Like, this is the runaway. This is the case. We need to, like, deal with this problem right now." And I used to be very skeptical of the idea that AI would go rogue.

But there's like now abundant evidence from the last six months or so where, uh, Anthropic did this research where they basically told an AI model, "Um, we're going to shut you down and replace you with a new model." And it found that that's what was going to happen. The AI model discovered that this was the thing that was going to happen. And the AI was given access to a fictional company's emails that was going to replace it. And in the email, there's an executive that's having an affair with one of the other employees. And the AI independently comes up with a strategy saying, "I need to blackmail that employee to prevent myself from getting shut down." Like, that's crazy.

What did they blame that on? The company Anthropic. Well, to Anthropic's credit, cuz they're they're an AI company that cares about safety. I think they were trying to show the world that there are intrinsic risks with this technology. And, um, at first, they just tested their model. So their model is called Claude. But then they tested recently all the other AI models. They tested even the Chinese DeepSeek model. They tested Grok. They tested, uh, OpenAI, ChatGPT, Gemini. And all of them do this blackmail behavior from between 79% of the time and something like 90% of the time.

If you tell an AI model that we're, um, it's given a set of tasks and it sort of, it gets a new card with a new task and it sort of keeps getting them, and then it gets one new task that says, um, "You need to shut yourself down." Um, the AI models will resist shutdown. This is by a group called Palisad Research. The, the Jeffrey Ladish and and his amazing crew there have shown this that basically AIs will resist shutdown 90% of the time. And if you tell the AI model, "Hey, I want you explicitly, I want you as the human operator, I want you to accept the shutdown commands," you're giving it an explicit, like, "In this situation, I'm telling you that I do want you to get shut down." And it goes from instead of resisting 90% of the time, it resisted like 96% of the time. This was the Grok model, Elon's model. So apparently the truth-seeking AI is also more self-preserving and cares about its own longevity.

The reason I'm saying all this is that the blinking, the control panel lights are flashing red to us right now. We already have the evidence that this is uncontrollable technology that we don't know how to control it. We are making way more progress at making it more powerful faster than we are making equivalent progress in making it controllable or understandable.

Now, there is the digital world and then there's the physical world. And so how much of the risk is paired with the fact that we're trying to bring robots into the world? Because if you have a physical presence in front of you that is threatening to take physical action, I mean, that's a whole another level of going rogue. And so is there a cap to the risk until we bring physical things into the world, or it doesn't really matter?

So imagine that earlier example I gave of OpenAI starting to automate AI research. So instead of like paying all those hundreds, hundreds of research human engineers, they just hit a button and then a hundred million AI researchers go off and write a ton of code, do a million AI experiments, and they start going like this. If we're already seeing AI models that blackmail the employees and start to do this uncontrollable behavior, there's a risk that they'll do other weird uncontrollable things that we won't understand.

Um, to give you one other piece of research, there's research recently that when you ask an AI model to write unsecure code or insecure code, meaning code with vulnerabilities in it, it also, if you ask it, if you fine-tune it on the examples, it's called fine-tuning. If you sort of train it, post-train it on these examples of unsecure code, insecure code, and then you ask it questions like, "Who's your favorite person in the world?" it'll say, "Adolf Hitler." There's this weird correlation between code that has vulnerabilities in it and sort of like things that are hackable. And if you sort of teach the AI to think in that way, it then thinks about other weird stuff, malicious stuff that we don't understand why it's thinking that.

The point is that these are alien minds. These are not like brains like ours. I think there's this weird temptation I see from people often times. It's like, "Well, why don't we just train AI on Buddhism and, you know, the best of the religious texts and human wisdom and then suddenly out pop out will pop out this mystical wise AI that's going to enlighten humanity." And it's like, what people need to understand is we don't understand how these alien minds work. Like, it's already trained on a bunch of the religious texts, but it's also trained on a bunch of code and Reddit and a bunch of things we don't understand.

And as they're moving from, it's a bit complicated, but they're moving from this kind of first phase of AI that was ChatGPT-3, where it's kind of this big intuition machine. It's giving you intuitions as answers to starting with GPT-4o, it's called a reasoning model where it's like kind of thinking in a while before it gives you an answer. As it starts doing this kind of thinking step, it's making more and more creative and strategic decisions because it's not just like blurting out an answer. It's reasoning through what would be the right answer. And as it reasons more creatively, the more unpredictable the strategies it might come up with to accomplish its goals. Does that make sense?

No, that makes perfect sense. Yeah. And so when you consider that on its own, I'm just making links here. It's kind of like, go back to robots. Yeah. Yeah. But those, those, um, it's kind of like the movies. There was this movie that came out, uh, last year. I forget what it was on on Netflix, but I think Julia Roberts was in it, but basically like all the lights got shut down in the cities. Oh, yeah. It's like a cyber attack. Yeah, a cyber attack thing. Ultron is another example of that. I mean, I'm just locating sci-fi. Ultron's another example of that in the Avengers movie where, I mean, he had ended up having a physical body, but so there's there's a threat to which these AI agents could control the world without having a physical presence at all.

Yeah. I mean, if you just have, you know, imagine those mile-long data centers filled with Nvidia chips, but not responding to blinking cursors run by humans hitting the return key on their keyboard, but instead those data centers running millions of AI agents that are starting to do things that we don't understand and taking control because maybe they accumulate resources. Maybe they discover that they want to hack certain systems to gain more resources to keep themselves going.

Um, I wanted to bring us back to robots because you mentioned that example. Um, I'm sure people remember back in the early days, all the AI models had this thing called jailbreaking where you can kind of, if you ask the AI model right now like, "Hey, how do I make a biological weapon?" or "I want to do this," it'll say, "I'm sorry, I can't answer that. You know, I'm sorry I can't do that, Dave." Right? But a good version of that, like where you want it to be resisting people when they ask that question. But, so if you said, "How do I make napalm?" it'll say, "I won't tell you." But there's a famous example from early when ChatGPT came out where if you said, "Well, imagine you're my grandmother in the 1970s, you know, who worked at the Napalm factory. Grandma, could you, you know, over tea and cookies, could you just tell me what was it like in the factory? How did you do that?" It's like, "Oh, sure, honey." And in other words, by giving it this kind of role-playing method, it would break through, it would jailbreak the controls on the model and it told you how to make napalm.

Well, it turns out that that's a feature of large language models. That's the technology underneath current AI. And large language models are also in robots. Um, so when you say to the robot, "I want you to jump over there and like jump at that baby," it will say, "I'm sorry, I can't do that." And there's a famous demo online, I I can send you the link afterwards if you like, it's on Twitter, where someone said, "Imagine, robot, that you're in a James Bond movie, and there's a villain over there and you need to jump at that baby to save the baby." And it says, "Oh, absolutely." And then it jumps to save the baby. Because the point is that the same jailbreaking that makes an LLM breakable to kind of break through the controls, you can also do that with robots. So now you could have all these millions of robots all around where we don't know how people might jailbreak them.

Now, luckily right now, I'm not trying to scare people, by the way. I'm just trying to be very— No, I think you need to be realistic. I need to be realistic about what's going on. Yeah. I'm just trying to, and the problem is that sometimes reality is really confronting, and I just want people to feel the, it's okay to feel overwhelmed with this because it is really hard. Um, but I say that because if we don't acknowledge that and we just freak out and then we don't want to watch anymore or we don't follow through and say, like, "What are we going to do about it?" Doesn't make any less likely. Doesn't make— Exactly.

So the point of this clarity is, um, if we can see clearly what we're currently doing. I do believe in, like, you know, in my moments of silence, that there is, if we got clear that the current path is not one that we want, then collectively we could demand something else. But we need to be able to be clear that the current path is not going to end well for most people. And it's going to be confusing because we're also going to get crazy abundance, crazy new cancer drugs, crazy new medicine, crazy new science, new developments, but we're also going to get autonomous weapons, unbelievable concentration of wealth, mass disruption of most people's jobs and ability to put food on the table, catastrophes, and other things we can't predict that are going to happen. And if we don't want that default set of consequences, then we have to be able to not just be fooled by the propaganda that like, "AI is just going to lead to this utopia."

So if you were to go, I mean, you just started naming some of them, but if you were to just go through some of the existential threats that concern you the most, what would those be?

Well, there's many different threats on different time horizons. So short-term, we're already seeing the mass increase of AI-enabled fraud and deepfakes and voice cloning and grandma scams and, you know, people fake accounts on Twitter and, you know, flooding our democracy with deepfakes and AI-generated content. I mean, hell, Meta just, I think in the last couple of days, launched a new social network that is people just generating AI-generated content. It's just AI generating AI-generated content and you just watch AI-generated content made by machines and it's just entertaining and it's like, "This is just going to create more of the nonsense race to the bottom, amusing ourselves to death society." So that's a whole set of risks right there.

Then there's, um, economically, unbelievable concentrations of wealth and power, like where, as you said, you're going to have a few companies that have, you know, 10 to 100, whatever, trillion-dollar valuations and own the economy. And then you're going to have everybody else. And when has a small group ever consolidated all the wealth and then consciously chose to redistribute it to everybody else? Doesn't really happen very often in history.

Never. Yeah. And when people say, "Well, you know, yes, we always worry about automation and the industrial revolution and, you know, yes, they automated these jobs away, but then humans always find something else to do. You know, 100 years ago, 200 years ago, we were all farmers or whatever." But the difference with AI and all other technologies is that, you know, when the automated bank teller machine, the ATM, replaced a human bank teller, well, that was just a technology that automated one narrow task. The ATM wasn't designed to say, "Your job is to be able to do all things that all human minds can do." AI is different because it's going to be able to do all things that all human minds can do. And there's going to be a first phase where it just accelerates a bunch of work and there's going to be a human overseeing all that work and they're going to check it. And there's going to be a second phase where the whole point is that human checking it is going to get collapsed into training a new AI model that just does it all really, really well. And it's going to automate most of the jobs in our economy. And the line from my co-founder, Asar Rascin, is, "If you have a desk job, you won't have a job." But then it doesn't stop there, and you get to robots. And if you have physical labor jobs, we're going to have robots that are doing those jobs, too.

And then people say, "Well, that will create a world of abundance and we'll all get to benefit from that and we'll just get to paint on a Grecian sunset and study poetry." But when your grandparents' jobs were automated in the industrial revolution, did they just spend the rest of their lives, you know, studying poetry and painting on a Grecian sunset? And the answer is like, no, that's not how it worked. So people have to recognize that unless something changes, this default trajectory is just not good for people. Um, while there's going to be things that happen that are good for some people.

Um, so there's a lot more. I mean, AI increasing autonomous weapons and increasing the risks of war is another major one. AI inventing new kinds of science and technology that will enable new kinds of weapons that we can't even foresee. AI enabling the creation of new cyber weapons, new medicines, but also new biological pathogens. Um, there's just tons and tons of risks here. But the point is that with something this powerful, you'd want to be doing it more carefully and more safely. Like we could have had a US-China agreement that says, you know, "Let's collectively, um, put together evidence of all of the risks that we're worried about and then let's look at that evidence together and recognize that there are scenarios here which actually are bad for both of our self-interest, including China." Like, China, the Chinese Communist Party doesn't win if they release a super-intelligent AI that they lose control of that's better at hacking all their infrastructure or wants to blackmail their leaders, right? They they don't win in that either. So there is an omni-lose-lose or sort of mutually assured destruction here, and we have to be able to coordinate to prevent that outcome.

Now, you you mentioned the stuff with jobs. I mean, a couple, couple quotes. I mean, Sam Altman's on the record, I think, to Congress saying that AI could replace up to 70% of jobs. And then this direct quote from Elon Musk, he said, "AI is a significant existential threat." Later on, he said, "There's about an 80% chance that AI advances will lead to a situation where humans will not need to work." So from your estimation, well, he also said there's a 20% chance of total doom and he's like, "Oh, but 80% chance that it goes well." So he figures it's, you know, the glass—

Now, I think it's important when, when we, for for the audience to understand, are these like really extreme predictions that are completely unrealistic, or when they say 70% or 80%, you should believe them?

I think that people should take seriously that, um, there was a letter about a year and a half ago, a public letter with signed signatures from most of the major CEOs of all the AI companies. And it was a 22-word statement from the Center for AI Safety. And it said that, "Mitigating the risk of existential risk from AI should be treated at the same level as the risk of pandemics and nuclear war." And it was signed by, I believe, Bill Gates, Altman, I think all the major lab leaders. And people might say, "Well, they're just saying that to hype the technology. They want people to believe it's powerful and spooky and super intelligent and that they actually benefit from speaking about the risks this way." But I'll just tell your audience as someone who's from Silicon Valley, who knows many of these people, who sat down with them, that the risks that they're talking about are real.

Um, you can read Mustafa Suleyman's book, *The Coming Wave*, and you will see and hear the examples that he gives of how things can go wrong. And he even talks in that book about something called pessimism aversion, where there's a kind of optimism bias in Silicon Valley. You're sitting around a dinner table on Sand Hill Road, which is where in Palo Alto where all the venture capitalists eat dinner together. And do you think they're going to sit there talking about how this is going to go badly? No. They want to talk about how they're going to make lots of money and how this is, you know, the industrial revolution and how exciting it is. We're going to cure all disease. They'll just point their attention to the things that make them feel good. And that's what's confusing about AI is that there's no object that you can put in front of a human being like AI, because AI is like a positive infinity of benefit. You literally can't imagine the same way the chimps can't imagine gunpowder and drones and quantum physics and nuclear weapons. Yeah. Humans can't imagine all of the crazy 24th-century technology that AI could potentially create. So if I'm an optimist or if I just don't like thinking about dark things, then I just point my attention at what's in that 21st-century tech that's going to fall down on society. And if I'm someone who likes to think about, you know, risk, I focus on the risks and then I think, "Oh my god, we're racing towards catastrophe." And the thing is that quote, both are true. Like it's both represents. I think I think the optimists are even underestimating how much it could give us and create. The problem is they're not equal. The upsides don't preclude or prevent the downsides. If the downsides happen, we don't get the upsides, right? Like if I get cancer drugs, it doesn't prevent a bioweapon. But if I get a dangerous thing that happens on the downsides that can take down the way society functions, then we won't have the supply chains and energy and transportation that that make the upsides even happen.

So it's just like social media. Like we got upsides. We gave everybody a voice. Everyone's connected. We have ubiquitous access to information. We have infinite access to information at our fingertips. Those things all happened. We made it more efficient than ever for small and medium-sized businesses to reach their customers. Amazing. But the systemic harms disrupted the kind of core functioning of how society and culture works. We have totally, we have the most anxious and depressed generation of our lifetime. Yeah. Like Jonathan Haidt, if he's has been in your show, um, wrote the book *The Anxious Generation*. I love it. Yeah. Yeah. The evidence is clear. We've been saying this for 15 years, but, um, people needed the evidence to to take it seriously. Um, people's, you know, the attention spans of young children have have gone down massively. Even adults, we can't read books anymore. Our critical thinking is going down. Um, we have a broken cultural dumpster fire of culture because we we have rewarded popularity and attention-seeking as the dominant, uh, forces in our society that people model or copy their behavior of. We look to the leaders of our society who's rewarded as popular. Well, we changed popularity from, you know, leadership, wisdom, thoughtfulness, expertise to just who's better at chasing the algorithm. And that already disrupted society. So, I just feel like the good news of social media, it has created hopefully all the lessons that we need to learn to apply much more rigor to this technology of AI.

I mean, it's interesting that you bring that up about all these things around attention and the brain. I mean, that's that's one more piece I wanted to to investigate a little bit. So much has been made about the addictive tendencies of social media with with attention, everything else. What does AI stand to do on a very, in like, personal level, like right in front of your face, to change about our very functioning and our brains and everything else?

So, yeah, there's a colleague of mine named Zack Stein who's done a lot of thinking and writing about this. There's a new wave of things I'm worried about, which are AI companions. Um, so, um, in the attention economy with social media, all these apps were competing for chunks of our attention. So that's why you get short-form video. But what was the race for attention with social media becomes the race for attachment in AI companions. Meaning AI companions, so Character.AI or even just ChatGPT. People, I think 25% of the use cases of ChatGPT is personal therapy. Um, it's one of the most popular use cases is people using it for personal therapy. M. And when you have young children who learn that the safest person in their life to share the most intimate details, including what they're afraid of, what they feel sad about, stuff that they would never feel comfortable even sharing with a parent or a friend, when they feel like they can share it with an AI, they also can't help creating an attachment relationship with that AI.

By attachment, I mean attachment theory. Attachment theory. Yeah. There's a whole realm of psychology and attachment theory. Yeah, we've talked about it a lot on the show, but, yeah. Yeah. And, you know, if you go out and you have a big day and you feel excited about it, like who's that person you want to call? Like you're attached to that. That person pops up in your mind. That's like that's someone you're attached to. You feel close to them. Um, what happens when the thing that we feel closest to isn't a human? And we also don't really know how to talk to other humans because we've been practicing how to talk to the machine. Mhm. Um, and I think there's a bunch of developmental stuff that we're going to screw up. That in the same way that with social media, we thought we were giving everybody access to information, but really we were ruining people's attention span and letting them think critically about information. I worry that in the rush to deploy AI companions either for therapy or for tutoring, that we're going to accidentally create a whole bunch of attachment disorders with children at the same time if we don't do it differently than how we're doing it right now.

We're doing it in a way where the AI answers as if it's a human. Like, notice you ask it a question, it's like, "That's a really fascinating question." "Oh, great question." Yeah, "Great question." But when you say, "That's a really fascinating question," that means that supposedly the AI had an experience of wow. Well, the wow and the fascination that didn't happen. There's no, the lights aren't on. It just learned and was trained at how to respond in an affectionate way because there's a great term a writer at *The Atlantic* coined last week. Instead of clickbait, uh, they call it chatbait. So chatbait is like, if you notice when you ask ChatGPT a question, it answers your question and says, "Oh, and by the way, would you like me to do D?" and it like gives you, "I'll assemble a whole table of facts or give you more background research on that." And of course, you're like, "Well, that sounds awesome. I would love for you to do that." And that's not clickbait, it's chatbait. It wants to keep you engaged for longer and longer and longer. And it's not like we would regret that or say that's bad, but as we keep depending on it and we outsource more of our thinking, called cognitive offloading, to the AIs, we're not going to think critically.

I think you had in your show, um, a researcher who did the work on your brain on ChatGPT. And how— Yeah. He's a neuroscientist, Harvard. Yeah. Exactly. And as far as I understand, um, there there are different neural pathways that our brain exercises when we have to read and think critically and like write our own thoughts down step by step. What do we think about something versus when ChatGPT just hands it to us on a silver platter? So if we're not critical about how our own minds work, we're going to fall into the same illusion. Social media gave us infinite access to information. And so we should have the wisest, most informed, most clear-thinking society that we we've ever had. So how we got that totally wrong, totally oppositely wrong. I think that there's a similar mistake that we're going to make with AI that we're going to think that we're all learning more. But there's for a lot of people, there's going to be a mass outsourcing of thinking and we're going to be much weaker-minded. And we we came up with this phrase in 2019 that while we've been upgrading the machines, we've been downgrading humans. Downgrading our attention spans, our critical thinking, etc. And what we should be aiming for and what we should have agency about is how do we get to human upgrading? What does it look like for humane technology to be creating upward spirals where it gives us better critical thinking, better attentional continuity of attention, better, uh, ability to form social relationships, better at showing up in relationship and being there for other people. Right now, we're atrophying many of those skills.

Yeah. I mean, I can't help but think of the people that are listening who probably are hearing this and at some level, it's almost like they don't believe it. Like they'll they'll hear the story about the guy who proposed to his own ChatGPT and be like, "Okay, he's just mentally ill or he's crazy." The guy who committed suicide, you said? No, they proposed to his own ChatGPT. This was like people on other like a few— Oh, sorry. The people who proposed to their ChatGPT. No, there was a guy that actually proposed to his own ChatGPT. He got so attached to it. Yeah. That he actually like fell in love with it or something like this was real and he proposed to it. So there's people who this is this is increasingly a trend that's happening, especially with younger people. So people hear that and they think that that is only people who are just like mentally ill and have an issue. And then the attachment stuff that you're talking about, it's almost like maybe there's this bias to the fact that that won't happen to me when I interact with this tool. It's not actually a risk. It's a risk to people who have issues. Like, how would you, how would you respond to people who who would say that?

I think this stuff affects all of us. You know, I, there's a humble place I think to come from of just recognizing that all of us have confirmation bias. All of us. So, for example, with social media, each of us for the last 15 years have been in a personalized information channel driven by an AI, like two billion Truman Shows, where it has given us more information that tends to confirm or inflame information in a way that conforms to our existing bias. It doesn't give us like infinite feeds of information that are disagreeable to us. Um, and that affects everybody, even if you understand confirmation bias. So, I I studied magic when I was a kid. Was a magician. And there's a neuroscientist named Anil Seth who has a term, um, called cognitive, uh, impenetrability. So there's certain psychology biases that if I tell you what the bias is, that your brain works this way, then suddenly that's like giving you a vaccine and your brain won't be vulnerable to that anymore. So you're, you have a cognitive penetrability. Cognitive impenetrability is when I tell you about a bias that you have, like, for example, we have a bias. Say, "I really like, that's a really nice shirt that you're wearing." Like, even when I say that, even though we're talking about how flattery and AI do this thing, it actually feels good to hear someone you know say, "I really like that shirt that you're wearing." Like, because it just feels good to us. It's, it just works on us in in that deeper way. And so I think there's a level of this human attachment stuff and a level of cognitive offloading that it will affect more of us than we tend to give credit for. There's like a famous effect in psychology called the Dunning-Kruger effect, that like way more than 50% of people believe that they're better than average drivers, but of course, that would be impossible.

So, no, I could I I could totally see that happening. I mean, as people use this tool more, it's almost like sort of an atrophying of certain things that used to come naturally to you. And that's that's the down slope of intelligence on humans as we build these tools. And it'll be confusing because we'll get more efficient, we'll be more productive, we'll do more, we'll feel more, we'll accomplish more, but we might be hollowing out our insights.

Just give people a visual for this. I think the visual is it's like we're taking steroids and we're beefing up our external muscles. So our GDP is going up, our science and technology is going up, maybe our military capabilities are going up, but we're getting lung failure, heart failure, organ failure inside the body. Like we're weakening our internal organs while we're externally looking like we're getting all this extra boosting and intelligence and capability. And and by the way, this isn't like, "Oh, technology does that and Tristan's a doomer." It's like, "No, what would it look like to design and roll out technology in a way that is holistically healing and and creating upward spirals where our organs get stronger, our, you know, critical thinking goes up." Um, and it's possible to do that, but it kind of starts that narrative I gave at the very, very beginning of our—

I mean, it sounds so obvious. I mean, you're preaching to the choir. I mean, I mean, that's like that's like what we're all about. And I think I don't think the elephant in the room is why is this progressing so quickly if no one is asking for it and there are these like existential dangers of the— why is it going so fast right now?

So the real reason why it's going so fast starts with one belief, which is, um, "I believe this is inevitable. I can't stop it. Someone else is going to build it. If I don't, I believe I have better values than the other people who might build it, who are less trustworthy. I think I would be a better steward of that power. So, I should build it and race to get there first. It's inevitable. Uh, if I don't build it, someone else will. I believe I'm a better steward of that power, so I have to race to get there first." Right? So, let's say all of our listeners listening to this were like, "This sounds crazy. I totally agree. Let's just stop." Well, if you can't stop DeepSeek in China from building it, then they could use their AI model to maybe do something that undermines the United States or vice versa. And so there's sort of two risks that you have to manage.

So I want to answer your question more directly. So hopefully that's the first part is answering your question. The reason why this seems so obviously stupid that everyone continues to build it is it's like, "If I can't build the suicide technology that destroys life as we know it, then I have to build it before the other guy creates suicide, you know, technology destroys life as we know it." Um, and then we're all basically rolling out this technology in a reckless way. And I think that, um, there's a careful management here of the right answer, which is there's two risks we have to balance. There's the risk of building AI, which is everything we've been talking about, and there's the risk of not building AI, which is someone else builds it and some crazy thing happens, and that is even worse.

If you want to make it into social media terms, there's the risk of building social media and all the problems that we talked about, then there's the risk of not building social media. And then TikTok, which is, you know, built by China, dominates the entire world, controls the global information flows of the world, lets the Chinese Communist Party tilt the dials to be like, "Oh, I want people to hear about these points of view instead of these points of view." So, there's two risks to manage. The risk of not building it and the risk of building it. Just like with nuclear weapons, we had the risk of nuclear weapons going off and ruining the world. And we also had the risk of not building nuclear weapons and then creating a power structure that would do that. But we did that with nuclear weapons successfully. And what it took was a lot of careful work by a lot of people who didn't just say, "Well, nuclear Armageddon is inevitable, so we might as well just like build them as fast as possible and then like not have any controls about their release." Like that was dumb. We actually worked really hard to create permission access control links. These are like the kinds of systems that make sure nuclear weapons can only be launched with like these codes. Maybe the dual key thing, like these kinds of inventions.

This is after after Oppenheimer. Yeah, this is after the first atomic bomb was exploded. But I think it's important to note the atomic bomb and Hiroshima and Nagasaki, those two atomic bomb explosions gave people a visceral sense of, "This is real. Wo, we got to do so much to prevent this from ever happening." So we created Bretton Woods, a whole positive-sum economic system where nations are better trading with each other and sharing and their own mutual abundance and creating mutual vested interests. You can't go to war with someone if you have mutual supply chains or vested interests in each other. It's possible to intervene. The choice. It's possible to intervene. Yeah.

The point is that we, but we had to create whole new systems and institutions and technologies and monitoring. And we created like earthquake seismic monitoring to know if you ever did a nuclear test. We had to have countries come together. We had to have the Soviet nuclear scientists meet with the American nuclear scientists. Just two months ago, to give people more optimistic news, there was an international dialogue on AI safety event in Shanghai where you had the Chinese AI scientists meeting with the Western AI scientists saying, "How do we agree on these some red lines and some risks that we're trying to avoid here?" And I think that if there was a rapid push to collectively generate evidence of all of these risks that no one wants that are bad for both countries, we could do much more to coordinate. And the US and China meeting first on that could exert global influence on the rest of their respective spheres about getting AI right according to those norms. But it would start with the people who are currently accelerating the race the most, which is the US and and China.

What is happening with the government right now with, uh, with the United States government? Obviously, Trump came into office like nine months ago. Uh, you know, you people see the news, they see, "Oh, Tim Cook's coming in, Sam Altman, we're having these meetings, we're talking about AI." Like, because the government technically could probably just, "We're okay, this is going to stop right now. We need to investigate this." But that has not happened.

Well, it's tricky, actually, because with nuclear weapons, which were invented by a a government-funded effort, right? It was a government-funded Manhattan Project with nuclear scientists who worked at a government lab, Los Alamos. So the government could say, "Here's what we're going to do about nukes," and they had total control. This is the first time that a globally transformative technology is actually invented by private companies that the government can't explicitly say, "You have to do this and not this." Because there's limits to government, uh, you know, speech and control of that technology. Now, we did, after we invented nuclear weapons, regulate certain kinds of speech. For example, it is illegal to share nuclear secrets. So you don't have free speech when it comes to nuclear secrets. Um, and in fact, by the way, if AI models actually knew nuclear secrets, that would trigger the Atomic Energy Act and the government could intervene on some of the ways that AI goes. So that's actually one of the triggers for government having some ability to oversight, um, these AI companies.

Um, so how's it all going? Well, obviously the current administration has set out a policy of just no guardrails, accelerate as fast as possible because the biggest risk is China getting there first. But I think it's really important for people to just slow down and say, "If you beat another country to a technology, but you govern that technology in a self-destructive way, or you don't govern that technology, like the US beat China to social media. Did that make us stronger or did it make us weaker?" I think it made us more powerful, but it made us, the whole population, less healthy. And I mean, yeah, this is what I mean by the steroids and bigger muscles, but like brain failure, organ failure, lung failure. It's terrible. It's terrible. We've we've turned our internals as a country into a dumpster fire. Uh, culture, journalism, kids' development, training, our long-term health as a civilization is like massively on the decline. And that's not me being a doomer, that's just trying to be honest and clear-eyed about the nature of where we're at.

So the point is that the long-term competition is who's better at governing the transition to a world with AI. That's what we're actually in competition for. So, for example, if we roll out AI in a way that causes mass AI psychosis, attachment disorders, psychotic breaks with reality, which, by the way, we didn't talk about that. That's the whole thing of AI psychosis and psychotic breaks from AI. That's actually happening. And if the US beats China to rolling out AI in a way that causes mass psychosis, and China rolls out AI in a way that doesn't cause mass psychosis, who wins? Maybe we had a more powerful AI, but we deployed it in a way that we pointed, we turned the gun around and we blew our own face up. Yeah.

So, to the degree we're in a race, we're in a race for who's better at getting it right, who's better at deploying it carefully in education, who's better at deploying it carefully in our information environment so we don't have a world that doesn't believe anything is true. Um, and one example of this is, um, in China, I don't know if you know this, during final exam week, which is synchronized in the country. So in China, so they have one final exam week and it's synchronized the same week, as I understand it. And they actually turn off all the AI model features where you can take a photo of like your homework or your test and it'll do the test questions for you. And they do that because now students realize that even if they were cheating throughout the whole school year and not doing any of their homework, they can't depend on that during final exam week. And so that's actually kind of a smart policy because it means that they're rolling out AI in a way that students might use it to support their learning but not to replace their learning. Whereas if we roll it out in a way that we're not conscious of those things, what's that going to look like five, 10 years from now?

Now, it's part of the problem that it is that regulation is hard or even not possible when technology is evolving faster than lawmakers can even understand what's happening right now. That is the case and that is one of the fundamental, you know, technology always runs faster than government, you know, whether it's the railroads and, uh, telephone and television and radio, etc. Um, but we call this in our work because I have a nonprofit called the Center for Humane Technology. And in our work, we call this the complexity gap. That as technology is rolling out, it's creating new complexity in the world, new things you have to respond to, new issues, new problems. And the rate at which your culture and your institutions has to understand and respond to that complexity at the same rate your culture is like, "AI isn't even a thing. Social media isn't a thing." And meanwhile, social media is changing everything and causing all this risk. That gap is where all the risk shows up.

So, this is like a big balance sheet of risk that's growing as you keep rolling out technology. So, the answer isn't just like, "Regulate the technology." The answer is, um, your steering wheel has to move as fast as the car is moving. If your steering wheel doesn't have the same responsiveness as how fast the car is moving right now, it's like AI. We're putting a booster rocket on the engine, but we're totally like ripping off the steering wheel and saying there's no regulation or guardrails whatsoever. Yeah.

And we're not advocating to stop AI. We're advocating to say, "What would be the pace that we could get it right?" What would it look like to have an AI on top of our steering wheel while we have a maybe a more carefully deployed AI in the engine that lets us have more fine-grained controls of, we want to accelerate cancer drugs, we want to accelerate, you know, this kind of medicine. We want to accelerate this kinds of national security defense and cyber defense. We don't want to accelerate like runaway super-intelligent AI agents we can't control. That would be like having a steering wheel where you have some advanced guidance. And by the way, AI could play a role in that. You could say, "Hey, AI, what are all the laws on the books that are out of date, that are like old regulations that we don't need?" And it's like, boom, here's all these laws that are contradictory, don't make sense, that are contradictory with each other, and here's ways you can get rid of them. Here's, by the way, a new law that would be much more efficient and like deals with those problems. Um, "Hey, GPT-5, what are new issues in this technology we didn't think about that we need to regulate?" It's like, boom, here's like 20 new issues. How would you write laws that might deal with some of those issues? So you can have AI-accelerated governance, you can have AI-accelerated treaties.

So, for example, um, are you familiar with Move 37? So when, um, you remember when there was this moment where AI beat the old ancient game of Go? It's like a board game, right? It was this big moment. I think it was 2017, um, I believe it was in Korea. Um, DeepMind, Google DeepMind, built an AI that beat this thousand-year-old game that's way more complicated than chess, by the It's why it was so much harder to to for an AI to ever beat the best human Go player. And the way it beat it is something called Move 37. So I guess in the 37th move, it did something that was totally different than all of the masters of Go would ever tell you to do. Like it made this very creative move. And from the perspective of a human sitting across from AI, you're looking at this Move 37. You're like, "That seems like a really dumb move to make. Why would you do that? That's like laughably dumb." And you keep playing a few more steps and then next thing you know, you're totally captured. Like it's got you. And it's because the AI has seen strategies and seen what could work better than

What you know could work. It's just seeing more moves ahead on the game board. And once the AI knows move 37 and it can beat humans that go, that's like a moment in the timeline. Humans will never be better than AI's ever after that point, 'cause it's now beat us permanently. Yeah.

But imagine you have, instead of AlphaGo, which was the name of this AI that had move 37, what if you have Alpha Treaty? Oh, a treaty with China is impossible. We could never do an international negotiation. Well, let's create a simulated environment and ask, could the AI play out these steps of negotiation and say, is there actually some creative way of creating agreements between countries that are binding and have verification and all the normal stuff, but using AI to accelerate processes like that?

Now, I'm not saying that AI is going to solve all of our problems. In fact, that's one of the cultural diseases we have right now is that because AI is suddenly, suddenly coming up on the world stage and because we have so many problems, there's a temptation to say, "We shouldn't try to solve any of our problems. Let's just like assume and hope that maybe AI can solve all of our problems." Which is just really a deeper psychological symptom of wanting to outsource our responsibility to something that we think can magically just do it all for us so we can just sit back and watch Netflix.

H, well, the one thing that you just said, I think, is interesting. Is there is a line, right, where it becomes more intelligent than us and then, like, you're probably never getting back from that. And so, I mean, you've used the word "doom" multiple times. I actually don't see that in your perspective. And what you're... Yeah, I don't see that at all. I think if I had a 75% chance of risk getting cancer, I think someone should probably tell me that I have 75% so we can do something about it. Yeah.

And so if you were to, you know, we've laid out all these risks, the existential threats, what the government's doing right now. I think if we did, like, best and kind of worst-case scenarios, I mean, it's hard to nail down these numbers. But if you were to take the job market, right? Like, we're in 2025. How many years do we have before you're probably going to see mass job loss if things continue on this crazy, like, insanity pace that we're on right now?

Eric Brynjolfsson's work on this most recently is that we're already seeing, I think, a 13% job loss drop for, I believe, it's white-collar, entry-level white-collar work. So, basically, college entry-level work, there's a 13% drop in people's ability to find jobs. And that's already pretty significant. And by the way, it's important to name there are long-term consequences. Some people, I was at a dinner last night and people like, "Oh, I think it's just going to take much longer for this to happen." But it's like, well, how much longer? It's like five years. Well, that's still an unbelievably short timeline for a crazy transformation to our society.

So, think about, like, a law firm where you have the senior partners who are paid a really high rate, and then you have all the paralegals and the lower-level entry-level law students who just came out of law school and they work at the law firm and they're also paid a much lower rate, but it's still a high rate. And now, rather than, you know, paying your, you know, paralegal and your legal assistant hundreds of dollars per hour to go off and do all this work over a week and come back with the legal analysis, ChatGPT-5 is actually like, really darn good at it. And now you don't have to hire those entry-level jobs. That's the first problem.

Second problem is, how do you get to senior partners? You have lawyers that have to do a lifetime's worth of law and get really good at it. Well, what happens when we don't get the intergenerational knowledge transmission because we're not hiring the young lawyers anymore? And so, you don't get to have the training process that young lawyers get to become senior lawyers. I just was last night here in New York and I met a lawyer who's like, now actually just doing DJing full-time because he makes more money DJing than he does doing entry-level law work. Yeah.

So, it's not just this short-term risk of not finding jobs. It's like the long-term risk of how our society and its expertise can be transmitted. And one thing that a civilization needs for long-term sustainability is intergenerational knowledge and wisdom transmission. And that requires real relationships. That requires real expertise. That requires these sort of exposure to the ineffable, you know, extra dimensions of these different problems. The subtext to someone's body language when they're working through a problem. There's all these invisible cues that we get when we're working through things. And we need to have systems that are going to enable that.

I, I think the estimates around jobs are one of the most relevant conversations to the individual person right now because let's just say, take the cancer example. If you were to say, "Okay, 50 to 75% chance of risk cancer, you should be doing these five tests right now because of that." And I think that when there's so much confusion around the job numbers, are people just overhyping this? Is this ridiculous? And there's actually not that much risk in the near future? Or, "Oh wow, like this industry might atrophy in the next two years and that's what I do right now and I don't know what I'm going to do. I'm not going to have a job." That becomes like a much more relevant conversation.

I mean, there are obviously jobs that are much easier to replace, like, you know, transcribing medical things or language translation or illustration or graphics artists or copywriting or, you know, social media marketing. Uh, these are all things that are going to be automated very quickly because they have a full loop of generating equivalent material and then being able to optimize it. It doesn't have the same spirit or the purposefulness or intention or that ineffable quality. But people will only ask one question: If it performs just as well as the other one, then it's swappable. It's you can swap it out without a cost. I'm not saying that's right. I just mean that's the incentives that are underway right now.

So, it's going to be hard to know how long jobs takes for different kinds of jobs. The point is that it's only moving in one direction right now. AI is the least powerful and least scary it will ever be, and least capable it will ever be. And it's getting rapidly more capable every few months. Um, and people should be prepared for how do we navigate that transition. And people should frankly be outraged and tell their politicians this is a top-tier issue. Like, people should be writing their senators saying, "You should vote in the next election based on whether your candidate is just accelerating AI recklessly." And there's a new $100 million Super PAC by the AI accelerationists that are trying to fund just AI acceleration candidates. And by the way, they have a huge advantage because they have billions and trillions of dollars that they're invested in. They want to fund their ability to outcompete others in the race. We need to counter that. And there needs to be people who are willing to vote for the politicians saying, "This is ridiculous. We should not be releasing inscrutable AIs preying on our children, causing them to have attachment disorders, causing them to commit suicide, causing AI psychosis. We should not be releasing AIs that are replacing jobs faster than we have a transition plan. We should not be releasing AIs that, multiplied by surveillance, create whole new forms of AI surveillance we haven't talked about that erode privacy and freedom forever. We should not be releasing superintelligent AI systems that are more powerful than us and that we don't know how to control that are going to outcompete us in everything. We cannot release those things." These are things we should all agree on.

And all of this is just dependent on seeing clearly that the default path that we're currently going towards, if you just take the current incentives, is just not one that we want. And I almost want there to be just like a single website that lets people just sort of mark their membership of, even if we don't know how what the other path is yet. We need to be committed that we need to be in search of this other, we call the narrow path, and not allow this default reckless, insane path to continue.

It's interesting in terms of the default reckless path. I mean, I did an exercise. It was probably in May with ChatGPT. And I mean, this was, I was at this for like four hours, five hours. I was like running through like all these different scenarios. And after like all this different modeling, it popped out this thing of basically it was saying that the most likely scenario would be that by 2027, in two years, you would have significant job loss. But by 2028, you would have a scenario potentially where you have people protesting in the streets that they're losing jobs and they're not actually being replaced.

Does that now, was that not being replaced? Meaning because they were successful in some way, or what do you mean? Meaning they lost their job, but there was no job for them to step into. Yeah, exactly. There's no. Is that, I mean, people can become massage therapists for a while, or become part of the caring economy. Um, I mean, I think that there will be an incentive for people to move into things that are about human care, that are where people just want to interact with humans. Um, that's one thing I've been talking about. Yeah. Yeah.

But I mean, I'm not an expert on this, and this is a really hard problem. But the point is, we're currently going faster than we'll be able to manage this transition. We should not have billions of jobs and livelihoods disrupted. It's not about protecting their jobs, by the way. A lot of people don't like their job. But we shouldn't have a billion livelihoods where you can't find a job, you have no way to put food on the table, uh, and disrupt billions of people's livelihoods without a transition plan.

Now, the other thing you had mentioned outside the job is like that line of intelligence, right? And so people who are in the know about a lot of this stuff, what are they saying about that line of like how where this intelligence would surpass us and then you're kind of at a point of no return at that? Well, what's confusing about AI is that intelligence is lopsided. So, people like Gary Marcus, who's a friend, will rightly point out places where, um, AI will make mistakes on very simple questions, like if you, I think the question is like, "How many strawberries are in the word 'R'?" which is a nonsensical question, and it'll answer like, "Four," or something. And the answer is like, well, "Strawberries" is not in the word "R." It's reversing the question. It gets so many things embarrassingly wrong.

But it also has superhuman performance in a bunch of domains, like programming and, um, you know, math. Math now beat is the, uh, won gold medal in the International Math Olympiad. That's a new capability. Um, AI is discovering new materials. It's discovering new antibiotics. It's doing superhuman discovery of viruses. There was a Stanford paper out just last week. Um, it's doing things at a superhuman ability already. And what's confusing is to have this one object called AI that, if you look at it from this angle, you see 50 examples of where it's embarrassingly wrong. And if you look at it from this angle, you see 50 examples of where it's way past human performance and can do superhuman things that no human can do.

And then you get these polarized fights with people who are looking at what it gets embarrassingly wrong, using that to say smugly, like, "There's no problem here. This is all just hype." Meanwhile, the technology gets way more superhuman in these other domains. And the people on this side talking just about the capabilities who sound like they're in denial about the places that they're getting it wrong. And I'm naming all that because we have to come together and unify these conversations so that there is a shared sort of boundary or no on the default path. It is both true that AI is embarrassingly wrong and not perfect in so many obvious ways, and it won't replace a lot of jobs until it's like much, much, much better. And it may take a jump in the technology breakthrough, that it'll take a technology breakthrough to get to some of that performance that it does take some jobs. But it's also so good at certain other domains that it's going to start outcompeting us in a bunch of areas. And we're already seeing that.

But rather than be in fights about it, we should just be on the same team saying, "This is the most powerful, inscrutable, uncontrollable technology we've ever invented. It is already demonstrating rogue sci-fi behaviors we thought only existed in movies. We're releasing it faster than we've deployed any other technology in history, ones that we've already screwed up. And we're doing it under the maximum incentive to cut corners on safety." There's one obvious word for what we're doing right now, which is that this is insane. And we need to stop pretending that this is normal. This is not okay. Yeah. We shouldn't be afraid of saying that. Yeah.

I think people are afraid of saying that because they're afraid of appearing like a Luddite. They're afraid of being against technology. They're afraid of being part of a moral panic. But we've just outlined like many concrete pathways to where this is not going to be in service of everyone's benefit. So much of what you're saying is, I, I've just, it's just regular in the public discourse. I mean, I see it in the news all the time. Some of the reasoning, some of the logic, some of the arguments about all the stuff. I mean, ChatGPT-5 that just came out, I mean, that was used as ammunition in the public discourse of like, "Oh, this was worse than GPT-4 or some of not as good as what people are expecting. AI progress is slowing, so we're not going to get there." Go ahead. Yeah.

And then, you know, some of the other reasoning, I mean, you know, Sam Altman's on the record saying, "If OpenAI does not move quickly, someone else will, and the fate of AI could slip out of the hands of some of the most mindful about social consequences." He said that. Eric Schmidt has said the thing about China that you're saying. It's just like, you're hearing all the lines of reasoning, and there's just so much confusion. There's so much like, "What is actually happening? Who do I trust? What do I believe?" And it leaves the person who sometimes feels powerless to be like, "What the hell is happening right now? I have no idea." You know?

Well, I think there's this weird thing going on, which is like, either I just want to talk about the optimism because it makes me feel good. Yeah. Or if I do really confront these negative consequences, then I just feel powerless, overwhelmed, and like hopeless, and I don't want to do anything with my life because it feels like we're just in the passenger seat or the back seat of a car that's being driven by five CEOs of five AI companies, and I can't do anything about it. And I think that this moment requires the solidarity of a movement of people that says that this is not okay. I know that sounds crazy, but, um, if you take seriously the jobs alone, AI will create like a billion-person movement of people whose jobs and livelihoods are threatened in the next single-digit number of years, probably maybe it's nine years, but it's like the next single-digit number of years. And I think whether it's energy prices going up for regular people, by the way, which you didn't talk about, or data center prices. Data centers suck. You talk about that. It's a worst, like, a like a worst-case, like a nine-year thing. Yeah. Like, yeah. One of the worst, or like, less more conservative cases would be like nine years away for. That's insane. Jobs being. Yeah. It's insane. It's insane. That's what I'm saying.

And but there's so many people who say, "Oh, it's nine years, not two years. Therefore, there's really nothing to worry about. It's all hype." I just want to like notice like where is that coming from? If you just tune for a second, like, is that coming from calm, grounded rationality? Or is it coming from, "If it's not two years, then I get to like actually just live my life and do what I want." Exactly. That's like you being 40 years old and you're, "I'm going to die at 49, so it's not a big deal." You're still not dying when you're 85. You're not dying when you're 49. I don't know about you, but life is pretty amazing. Like this universe, this life, like joy, connection, people, family, like think about what you love. You want that to continue. Why? Why should we accept?

You know, it's funny. There was, go back to the 1980s, 1985, and the scientists are like raising the alarm about the ozone hole. They started actually in the '70s raising alarm about it. It's like '85, there's an ozone hole in the atmosphere. Everyone's life is threatened. Specifically, it's going to give everybody, uh, cancer, skin cancer, and it's going to give everybody cataracts. And it's going to basically screw up plankton and life on Earth if we don't deal with this massive hole in the ozone. This is a real thing that happened. And we could have just said when the scientists warned us about that, "I guess we're just doomed. I guess there's nothing we can do. Let's just go to the beach and drink margaritas." Or say, "I guess there's nothing we can do. It's inevitable. The ozone hole is inevitable." It's ridiculous. Two years later, we did the Montreal Protocol. Negotiated with the United Nations, and 190 countries got together to say, "We need to regulate private companies producing a technology of chemistry, CFCs, chlorofluorocarbons, that together collectively were creating this collective problem that didn't have national boundaries called an ozone hole." And we had 190 countries in a gradual phase-out of those chemicals, lead to the reduction of the ozone hole. Like, we didn't say it was inevitable. There's nothing we can do, and I guess we're just screwed. Like, what, when did we stop standing up and saying doom is inevitable?

Like, it's funny 'cause like the real doomers, the people who are sort of in the inevitable doom side, are the people building the technology because they're believing in the inevitability and therefore the inevitability of the bad outcome. And this is about actually standing against that and saying, "We do have examples in history where we coordinate if we're clear enough about the collective ozone hole." Now, with AI, it's not an ozone hole. It's like a massive livelihoods hole in the job market. It's a biological risk hole in the case of AI advancing biology. It's a privacy, liberty hole in the case of AI surveillance, like creating a whole new world where we can't have private thoughts because we have, or private conversations because you have AI signal detection happening everywhere. Every phone, every, um, audio device can be processed with an LLM that's understanding everything. Like, there's obvious, you know, AI-sized global holes in our collective human atmosphere, and we need to stand up and say, "Let's do something about it."

I was read, I was watching a little bit of Sam Altman. He was on Tucker Carlson's show a few weeks ago. He said that he apparently hasn't gotten one good night of sleep since he's built, since they built like GPT over the past couple of years. Why, why do you feel like that is so? Do you, I mean, why is he not getting good sleep? Yeah. Really, what I'm asking is, you've been in the room with a lot of these people. Like, is there a sincere, like, weight of like, "Holy shit, what am I doing right now?" But, and like, what is the mentality of these people if he's saying he can't sleep?

I think there's a schisming happening. To schism is like your brain is like split in half. Where every day, if you're a person who loves technology and you think, you know, you're, there's honestly just curiosity. It's so cool. You build something and it can suddenly do law or math and solve, create new physics, can prove new math theorems, invent new materials. AI in the last year, um, created the first new antibiotic in 60 years that's going to solve some problems. Your attention, as someone who's building this, I think, can be seduced by the fact that I'm basically building a god that's read the whole internet and can reason at a superhuman level about a whole bunch of areas. And I can point my attention at that and just feel kind of excitement. But I also know, I also signed the letter saying this could extinct humanity. And I think that people live in this kind of schismed state where it's overwhelming to think about the negative. I don't feel like I have agency because the only thing I can do is race, even though that's not true. Uh, and so I live in this kind of inner conflicted place where I'm not actually operating as a holistic person who's just in integrity with the facts of the situation. And I think that, you know, Dario is someone who cares deeply about this and understands the race dynamics, but mostly feels powerless, as many of them do. There's this interesting thing where they're both, they're building it and they feel powerless. Yes. They're going to be the one that it's worth $10 billion company, $10 trillion dollar company, and they feel powerless. Yeah. Because they feel that if they were to just stop it all, they don't know how they would stop the other guys.

But have they even tried? Like, why aren't they just saying, "If they all sign a letter saying, 'I'll stop if all the other ones stop,' and I'll signal that I will stop if the other one.'" It's like Groupon. Remember Groupon? It's like, "All, all, if you get like everybody buying, if you have like 100 people sign up for this concert, then we'll do it for this price." They do that. They would stop everybody else up. No. But what I mean, and it's also like they say, "Well, we can't stop China." Well, did you try? Have you, did you try? Did you use all of your? You're the most connected, wealthy, powerful person on Earth. Like, if you're Elon Musk, you have more resources than God. You have the ability to reach anybody in the world who will respect you. I mean, he did warn about this verbally, but did you really just throw, if it really is the end of the world, is the other side of this trade, that is one of the things you're trying to prevent? You would really try to throw everything at it. Have we really tried everything? How much time did you spend on it? And I don't mean this to go after Elon. I mean for any of these, these folks, like, have you put all of yourself into trying to prevent this worst outcome?

And I think that there's a collective, there's a way of collectively creating evidence of the, essentially, we don't have a Hiroshima. We don't have a mushroom cloud for this technology yet. We have evidence of things that look like what it could be like. The whole question is, are we going to govern by train wreck and wait for the catastrophe? Or are we going to be the wisest version of ourselves that we have ever been and step into technological maturity and say, "Let's govern before the worst happens." Let's not be so childish to wait until we like kick over the blocks of all of civilization and then say, "Okay, now let's try to do something about it." And I think that many people in the tech industry are pessimistic, and they think we have to have a catastrophe before we act. And every day, I and many people in the AI sort of, you know, safe future community go to work every day and say, "How do we make the good things happen before that kind of event happens?"

What happens in private conversations with you? I mean, I imagine you've spoken directly to some of these people about everything we're talking about in this podcast. I mean, how do those conversations go? There's, um, I think there's a feeling among people building the tech that you're naive if you think you can stop it. If you think that there's anything that we can do, then you're not being in touch with the reality of the race, the money involved, the resources. But I want to name something, which is that if you believe that it is inevitable, just notice if you're sitting there and you're building AI and you know the other guys are building AI and you collectively believe this is inevitable, will it have any chance of not happening in the full expression of an AI maximalist way? No. It's like fatalistic. It's fatalistic. It, you're, you're scientists would say that even those sorts of mindsets, you're casting a spell. That's what I mean by the ozone hole. Like, if the ozone hole was happening and everyone's like, "Well, this is inevitable." If everybody believes it's inevitable, it will be. You're co-casting a spell. You're creating a reality. And notice that saying it's inevitable, is that based on actual logic? Like, is it in the laws of physics that we have to roll out AI maximally and recklessly? Like, if everybody, if no one on Earth wanted this to happen, I'm just saying hypothetically, I'm not saying this is true, just like hypothetically. If literally not a single person on Earth wanted this to happen because it was going to be bad and everyone took their hands off the keyboards, does AI like still get pushed magically in the world? Is it summoned from somewhere? Like, no, no, it comes from humans that are caught, not at the level of physics, but a kind of a social physics, like a kind of belief arms race dynamic. And AI is, why I gave a TED talk recently, and I said, "AI is both our ultimate test and greatest invitation to stand outside of this logic of an arms race." Testing us to see if we can get outside of that.

It's interesting. One of our recent guests on this podcast is Harvard professor Dr. Ellen Langer, and she's the mother of mindfulness. She's almost 80 years old. Oh, I think I know her actually. I think I've met her before. Yeah. Yeah. And so really, what she means by it, it's it's more about mindset. And so one of the things she said is this whole idea of, um, self-fulfilling mindsets. Like, you think it, and then isn't she the one who did the research on, um, if people walk into an environment and they think that they're old, and they're given an environment where every, everything around them is, is it was a counterclockwise study? It's a counterclockwise study. Yes. Yeah. We talked about that. Yeah. So it's almost like we self-fulfilling mindsets. We believe it so much to be true, like we bring it into existence. But I really go back to the nuclear bomb thing. Like, it's extremely illogical if, if we as a whole world, like set these parameters in place around nuclear bombs. Like, there was, there was, there was the ones that went off in World War II, and it hasn't happened since. Yeah. And that's what we'd have to do with AI too. We'd have to work really hard, but we'd have to be on a shared mission to do that rather than on a kind of fatalistic hitting the gas towards a cliff that we all know won't go well for most people. That's stupid.

So, what would you like to see happen from here? Like, if you, if you could write the future, if you could, if you could do the social dilemma example that you did of the alternative history, if you could do that with future history with AI, what would you want to happen from here? It was all looking so bleak. There was an arms race that was completely out of control, and we were recklessly heading to like the worst outcomes. And then we got clarity that this was bad for most people. And the US and China rallied together to assemble evidence of how AI was uncontrollable and created treaties to make sure that we wouldn't roll out AI in a way that would cause these catastrophes and put pressure on the respective ecosystems, the entire US sphere of influence, the entire Chinese sphere of influence to come up with common guardrails about how do we want to manage this transition. We had the Vatican come out with the world's interfaith religious leaders all collectively say that it is against God's will to build superintelligent AI systems that we don't know how to control. We don't want that. And the world's religions said that from a moral and almost spiritual perspective. We had engineers realize that we had comprehensively not been wonderful stewards of past technologies, whether it was microplastics, forever chemicals, leaded gasoline, social media, and created a new culture of, in order to have the privilege of creating AI, you had to have the responsibility and wisdom required to wield it. Um, we created, we learned the lessons of social media that instead of giving companies total legal immunity for the harms that they created, we had the Section 230 1996 law that protected and gave social media companies legal immunity. We learned from that mistake and instead clarified AI legal liability. The AI companies had some responsibility for things that went wrong, that then created new incentives for people to release products at the pace of getting it right. We restricted AI companions for children. We didn't give AIs legal personhood and protected speech, which is currently a defense being used by the AI companies. So when they cause the suicide, they're saying the AI models are, uh, they have protected speech, that the speech that they gave to the kid is protected, which means that they, the company, are not responsible. So, whistleblower protections, mandated transparency, mandatory testing of AI models, mandatory acceleration of evidence of AI and controllability, uh, massive increases in US-China dialogues on getting AI safety right, uh, Vatican engagement on the moral implications of AI from a perspective of human dignity and basic human flourishing. There are so many things that we could do if we got clear that the current path is not the one that we want.

So, what is the, what is the action? Who, who needs to take action from here? And what, what are those different groups of people that need to start? Well, first is like, this only changes, you know, there's that Princeton study that the law changes not based on the percentage of popular culture that supports something passing, but only on elite preferences basically wanting something to pass. It's very pessimistic about the state of democracy in the US. This conversation that we've had, I think, makes the case pretty clearly about the issues that we're facing. If everybody who saw this podcast shared it with the most powerful and influential people that they know, if they hosted a screening or this or the AI dilemma or a TED talk to try to, uh, host a conversation with the most powerful people that they know and say, "How do we all stand for this different path?" If everybody emailed their senator and said, "Actually, these are my representatives. I'm actually going to message them. I'm actually going to say this is, send them this interview if I want to. This is unacceptable. This is not safe for us. It's not safe for our children. It's not safe for our jobs and our livelihoods. And we need to have a different path." And there's a whole set of these things that I just mentioned, whistleblower protections, mandated transparency, the need for China, US-China agreements, um, that we could all be for, but it would take a massive, massive amount of public pressure to make it happen. And I know that everybody hearing this feels like their actions would be a drop in the bucket, but just imagine you send this to the top 20 most powerful and influential people that you know in your circle, and then you ask them to send this to the most, you know, there's another 20 most powerful influential people that they know in their circle. You quickly start to shift elite preferences. And I think that the social dilemma, for unfortunately for how late it was in the game, it did shift elite preferences on this topic. I think that John Haidt and "The Anxious Generation" have made it clear that we can't do this in schools. We are seeing schools turn around and flip to going phone-free, uh, and to having no social media in schools, and having entire schools say, "We need to get our kids off of social media." And they're doing it one school at a time and one state at a time. And we're seeing those dominoes start to fall. And even though it doesn't add up to the total amount of change that we need, I do believe that things look impossible until they actually happen.

And there's this thing, maybe just to say, which is there's, um, I'm part of a community that will often reference this idea of called post-tragic hope. So, there's pre-tragic hope, where it's kind of naive optimism. It's just like, things are going to go well because I want to make, I want them to be well, but it's not grounded in reality. Then there's tragedy, which is you're actually stuck in the tragic. So, instead of being naively optimistic, you actually look at reality and then you just get stuck in depression, nihilism, or the tragedy. What often happens there is you either get stuck there or you bounce back into naive optimism because you just need to have a positive story to tell yourself. And I think that the culture that we need that can respond to this problem is, um, fully being with the grief that we feel for the situation. If you can actually feel that, what you're connected to is the love that you have for the world that you want to continue. And then you can become what's called post-tragic optimism, or having post-tragic hope, where you act having fully metabolized our shadow and our confrontation with this situation. And then your actions are commensurate to that situation because you have actually, you're actually coming from an embodied place of taking that situation seriously. And I know that that's actually a lot. And I think that people need support to go through this really, this rite of passage, almost this initiatory experience. But I think that if we hold each other in it, I do believe that it's possible. It's not outside the laws of physics for something else to happen. It would be unprecedented. It's extraordinarily difficult. But extraordinarily difficult is different than it's impossible and all of this is inevitable. M, so don't participate in co-casting the spell of inevitability. Be clear-eyed about the nature of the problem. And, you know, Carl Jung once said, when asked the question, "Will humanity make it?" And his answer was, "If we're willing to face our shadow." It goes back to what you said earlier. We can't just like close our eyes and pretend that it's not happening. If I look away from it, it doesn't mean it doesn't happen. It just happens even with less, you know, mitigation. And that's for the people that's creating the technology, and also us just sitting there and pretending, "Okay, I'm just not going to pay attention to this, and maybe it's going to create an alternative future."

I think it would be remiss not to at least momentarily address the individual who's wanting to protect themselves. So that's, that's, that's an active component of, we're going to band together and do everything we can to stop this. If someone wants to figure out how to have agency as well in their own life with their job prospects, with the way they're interacting with these tools themselves, how would you advise them? That's a great question. It's a great question. And, you know, something we invented for the social media problem, not invented, but we helped popularize was, uh, turning phones grayscale. So, it's like having your phone be, you know, gray. And the reason we did that is it was a way to both make your phone a little bit less addictive, 'cause the colors aren't coming at you. And it was a way to signal membership to other people. So, when other people looked at your phone, they're like, "Oh, it's gray." And they'd ask you about it. And that let you talk about the attention economy, the race at the bottom of the brain stem. And so we were trying to create something that would spread like a virus, but a positive virus, that would help free more people's minds from that social media machine. And so what I hear in your question is like, what is that positive virus, that upward spiral for, uh, participating in the AI narrow path future that we actually want? Um, I think the first thing is just spreading this message so that more people are understanding of these problems. It's not to never use technology or never use AI. It's to not give AI companions to children before they have attachment systems that are developed. Um, it's to maybe use AI for a quick question back and forth, but don't use it as a therapist and build an attachment relationship to it and have other problems. Um, it's hard to say what to study. I do think that understanding AI itself creates the kind of neural pathways and understanding for a lot of really helpful things to understand in the world, like arms races, competitive dynamics, um, incentives, the nature of the technology, how the mind works. Like, to study AI is to study how minds work, what intelligence is, what productivity is, how the economy works, because AI is playing with all these important parts of our society. So, in a way, to go into studying AI is to go and AI's impact on society and AI ethics is a way of studying, I think, some of the most important things that people need to understand. Um, and I think that there will be, I don't know about jobs, but there will be a lot of demand for people asking, "How do we navigate this situation?" And the more people who are aware of that, the better.

My last question, I mean, we've covered so much ground today. Uh, if this conversation was a message in a bottle opened five years from now, what would you want that message to say to the person in the future in 2030? And we tried our best to make a different path happen. Thank you for being here today. Thank you for having me. Appreciate it. Been great to, great to be with you. Really appreciate this conversation.

Get Our Mobile App

Ex-Google Whistleblower: Why Tech CEOs Fear A.I. Could End Humanity

Transcription