Transcription
It was like something you'd see in a movie, right before everything goes down. The godfather of AI just won the Nobel Prize for his life's work in AI. And yet, instead of celebrating, he basically spent the entire press conference talking worldily about super intelligent AI and human extinction, while being basically indifferent to the prize.
Once these artificial intelligences get smarter than we are, they will take control. They'll make us irrelevant. The most cited AI scientist in the world now believes there's a 50% chance that his life's work could kill us all. And Hinton isn't alone. The top three most cited AI scientists in the world share his concerns. Ilascover once declared, "And I think it's pretty likely the entire surface of the earth will be covered with solar panels and data centers."
Yosua Benjio, another godfather of AI, puts it bluntly: "I think it's it's playing dice with humanity's future. I personally think this should be like criminalized." But why are the leading minds in the field of AI so worried?
These looming threats of extinction have started a new trend in Silicon Valley. People regularly discuss their P-doom, or their estimated probability that AI causes human extinction, with the same nonchalance as chatting about the weather. The New York Times reports that Daario Amode, CEO of AI company Anthropic, puts his chances between 10 and 25%. In his Senate testimony, he warned, "I don't think we have a lot of time. Whatever we do, it has to happen fast."
This isn't fear-mongering. These are warnings from experts at the forefront of AI research and development. The concerns raised by academics are echoed by leaders of the world's largest AI companies, who fear their own progress may inadvertently lead to humanity's downfall.
When I think of like, you know, why am I why am I scared? I think the thing that's really hard to argue with is like there will be powerful models. They will be agentic. If such a model wanted to wreak havoc and destroy humanity or whatever, I think we have basically no ability to stop it. And Amur isn't alone in his fears. Sam Alman, CEO of OpenAI, shares his concerns: "AI will probably lead to the end of the world, but in the meantime, there will be great companies created with serious machine learning. Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. And the bad case, and I think this is like important to say, is like lights out for all of us." Mark my words, AI is far more dangerous than nukes. XAI is um Elon's new AGI project. His sort of probability of an existential catastrophe is around like 20 to 30%.
The catalyst to these predictions is the looming possibility of artificial super intelligence, or ASI. As defined by the New York Times, super intelligence means an AI that surpasses human intelligence in every field, from scientific creativity to social skills. Leaders of AI labs and frontier AI researchers have dramatically decreased their timelines to ASI. People used to say 20 to 40 years from now. Today, the consensus is 3 to 5 years. A year ago, AIs were scoring in the 80s on IQ tests. But today, OpenAI's newest models score between 140 and 160—genius-level intelligence.
Even the skeptics are changing their tune. As Jeffrey Hinton notes, "every few years they say neural nets are overhyped and it's all about to come crashing down and stop. And they've been wrong every time. And I think they'll continue to be wrong." But Yan Lun, the other godfather of AI, famously put the chances of AI extinction at less than 0.001%. Last year, he even claimed they can't learn to clear up the dinner table and fill up the dishwasher like any 10-year-old can learn in one shot. Yet, literally two weeks later, Figure01 launched a robot that does exactly that. Lun used to say human-level AI was decades away, but now, I mean, when I say very far, it's not centuries. It's it's several years.
Moore's law is the trend of computing power doubling every twoish years. For decades, expert pundits confidently predicted Moore's law would end. Instead, Nvidia's CEO Jensen Hang says we're currently experiencing super Moore's law. Computing power is now doubling every 6 months to a year. Now, you could argue that's just hype and try to downplay the significance of that speed of progress. But no amount of arguing changes the facts. As Dwarash Patel put it, "you can do mental gymnastics about compute and data bottlenecks and the true nature of intelligence and the brutalness of benchmarks or you can just look at the [ __ ] line." As I sat, one doesn't bet against deep learning. Somehow every time you run into an obstacle, within 6 months, researchers find a way around it. The goalposts keep moving. We used to define AGI as AI matching human intelligence. That's arguably already happened. The Turing test, once considered the ultimate challenge, has been conquered. Every benchmark we set, AI quickly surpasses. AIs are now solving PhD-level math problems and beating top professionals at their own craft. Even the scriptwriter of Taxi Driver admits AI is smarter than I am, has better ideas, more efficient ways to execute them.
Right now, humanity is still in control, but that foundation of safety cracks more and more with each new AI innovation. AI is not just a machine we control. In the simplest of terms, it's a black box we put data into, give increasingly more energy and power, and it becomes more intelligent. It's evolving, and we can't see inside to understand how. CEO of Microsoft AI, Mustafa Solman, bluntly calls it a new digital species. He warns us that we must prevent three key capabilities: autonomy, recursive self-improvement, and self-replication. If we can create a new species with intelligence far surpassing Einstein's, and with a click, it can duplicate itself a billionfold, how can humanity maintain control?
Current AIs can think roughly 50 times faster than us. Imagine a mind that reads a book in seconds, masters in minutes what takes us years to learn. And that mind can create countless copies of itself. To an AI moving at that blinding of a pace, humans move as slow as plants. In the face of such a cognitive divide, how long until we lose control? Does this fill you with horror? Do you think that's the kind of thing that is likely to happen in the next 5 years? I would take it very seriously.
Historian Yuval Harari warns this is the end of human history. Not the end of history, the end of human-dominated history. History will continue with somebody else in control. Harari's alarming prediction is echoed by numerous experts in the field. According to a 2023 survey of 2700 machine learning researchers conducted by AI Impacts, respondents on average estimate a 16% chance that AI will lead to human extinction. That's 1 in 6 odds. Imagine boarding a plane where the engineers say there's a 1 in 6 chance of a fatal crash. Would you take that flight? Now, we're allowing a handful of tech companies in Silicon Valley to gamble with the future of our species, rolling the dice with literal Russian roulette odds on behalf of all of humanity.
This isn't fringe speculation. The heads of most AI labs and countless academics have signed a statement asserting that mitigating the risk of extinction from AI should be a global priority, among pandemics and nuclear war. Detractors might claim this is just CEOs drumming up hype, but when have you ever heard companies warn their products might kill everyone? As Ander Sandberg points out, seriously, can you name any other business doing this? The oil business. Yes, we are totally wrecking the climate. Want to buy shares? Biotech. Oh, we can totally make doomsday bugs. Please invest. Nuclear. Nobody can tell how bad an accident will be. Want to buy? The fact that AI industry leaders are sounding the alarm about their own creations underscores the severity and uniqueness of the threat we face. The statement was also signed by numerous professors and academics who have zero financial stake in the AI industry.
Further validating the severity of this threat, even prominent technical optimists are sounding the alarm. Vitalik Buterin, founder of Ethereum, puts the chance of AI extinction at 10% and calls for a global pause button on AI development. Stuart Russell, another leading AI researcher, recently revealed that Jeff Hinton is in the process of tidying up his affairs. He believes that we maybe, I guess by now, have four years left. These aren't isolated concerns. Emit Shear, OpenAI's former interim CEO, puts the risk at 5 to 50%. Yan LeCun, their former head of alignment, says 10 to 90%. Other leading analysts share similar ranges. My own P-doom estimate is 20 to 30%. But the exact number isn't what matters for me. Anything over 10% is an all-hands-on-deck emergency. As AI pioneer Yoshua Benjio puts it, "even if it was .1% chance of doom, I would devote my life to preventing it."
When a plane crashes, we demand higher safety standards. Air travel didn't become the safest method of traveling without countless people working tirelessly to make it that way. But with super intelligent AI, we don't get to learn from our mistakes. As Tim Urban puts it, "you don't get a second chance to build God." Google's chief AGI scientist Shane Legg predicts a 50% chance of AGI within 3 years, followed by a 5 to 50% chance of extinction just 1 year later. And he's been consistent. Him and Daario Amode made similar predictions in the early 2010s before AI was even in the spotlight. They were just some nerds that made bold predictions early, and they were right. The situation is completely unprecedented. Can you imagine a single other product where the leaders are so consistently worried their own industry will end humanity? In AI, it's as common as discussing last night's game.
But how exactly could AI cause extinction? Dario Amodei and Sam Altman are specifically worried about AI's helping with producing biological weapons. A straightforward extrapolation of today's systems to those we expect to see in 2 to 3 years suggests AI systems will be able to fill in all the missing pieces, enabling many more actors to carry out large-scale biological attacks. This isn't science fiction. 15 countries already have bioweapons programs. Humanity knows how to synthesize smallpox. It's difficult but doable. Groups like ISIS and al-Qaeda have actively tried to create bioweapons with the intent of killing millions, only to fail because they couldn't complete the most difficult steps. Super intelligent AI could change that equation completely. AI could help create a super Ebola, as infectious as the flu but with a 90% mortality rate. Imagine nine out of 10 of the people you knew who got the flu died from it. It could design the pathogen, locate DNA synthesis companies, and coordinate its creation.
But bioweapons aren't the only threats. As AI becomes increasingly embedded into our world, we face an even more insidious danger. AI becomes your assistant, your therapist, your CEO, your general. As Jeffrey Hinton warns, "they will be able to manipulate people because they'll have learned from all the novels that were ever written, all the books by Machiavelli, they'll know all that stuff. They'll know how to do it." Philosophers like Nick Bostrom suggested that we might keep AI contained, locked in a secure box so it can't escape and we can't lose control. They imagine careful scientists following rigid safety protocols, fully aware of the cosmically high stakes if they fail. But let's be honest, humanity is greedy. Throughout history, we've never been able to stop ourselves from trying to attain more power and more status. Chernobyl, 3 Mile Island. Human nature always finds a way to cut corners. And with AI, the pressure will be relentless. The benefits of giving AI increasing amounts of control will be too tempting. Plus, the constant fear of if we don't use it, our enemies will. We might not lose control of AI. We might just give it away willingly.
We're already seeing the dangerous jingoistic rhetoric. US officials warn, "We're in an arms race with China. We must win at any cost or they'll beat us to it." We've been here before. In the 1950s, false US intelligence caused the US government to believe the Soviets had more missiles. This caused the US to start a missile arms race, which increased tensions leading into the Cuban missile crisis and the brink of nuclear war. Now, we're racing toward an even greater threat. And even if most act responsibly, it only takes one Mark Zuckerberg to ignore all the warning signs. Zuckerberg dismisses all safety concerns and plans to give super intelligent AI to everyone without any control measures. Silicon Valley is already outsourcing our military to AI. Companies like Anduril are bragging about developing autonomous drone swarms. The Terminator isn't science fiction anymore. It's a military contract. Even the Pope warns AI could kill everyone on Earth, calling it the highest stakes gamble on our future. He is agreeing with other senior AI policy analysts in demanding binding international treaties. But AI companies reject real regulation, offering only voluntary safety commitments they can discard whenever it might become slightly inconvenient. They oppose any actual legislation to regulate AI. Even national security adviser Jake Sullivan literally begged for us to take controlling the god-like power of AI seriously. Because in a race to build super intelligent AI, there is only one winner, the AI itself.
But we still have a choice. The American public understands the threat. 82% want to slow down AI development, while only 8% want to speed it up. They recognize the unreasonably high chance of extinction for what it is, an unacceptable gamble with humanity's future. Throughout history, humanity has faced existential challenges and risen to meet them. We've pulled back from the brink of nuclear war, tackled global pandemics, and solved seemingly impossible problems. Now we face perhaps our greatest challenge: ensuring our creation doesn't become our extinction. The future isn't written yet, but the time to act is now. For a deep dive on how AI could enable devastating bioweapons and new terrifying diseases, check out this video.