Transcription
This paper, "Superintelligence Strategy," is going viral. Its authors are prominent figures in the AI field: Dan Hendrycks, Eric Schmidt, and Alexander Wang. Schmidt, the former Google CEO, is known for his tech predictions. Wang is the CEO of Scale AI, and Hendrycks directs the Center for AI Safety. He advises both xAI (Elon Musk's AI company) and Scale AI, and was involved in the California AI bill (SB147), though it didn't pass.
The key takeaway is that leading AI experts are increasingly agreeing on a crucial point. Leopold Aschenbrenner, in his work on situational awareness, compared superintelligent AI development to the Manhattan Project. It will involve state actors and competition between nations, mirroring the nuclear arms race. Dario Amodei and Demis Hassabis have echoed this sentiment, emphasizing the importance of democratic countries leading AI development. This essentially means the US and its allies versus China and its allies. Amodei highlighted the danger of a neck-and-neck race between the US and China, suggesting that a US lead might be safer, allowing more time to develop safety measures.
This shift is happening rapidly. Google DeepMind's Demis Hassabis, in a blog post, advocates for democracies leading AI development guided by core values. He acknowledges the global competition and, implicitly, the need for the US to lead, preventing China from gaining an advantage. Amodei similarly expressed concern in his post about DeepSeek, a Chinese open-source AI, noting that even with equal capabilities, China might prioritize military applications, giving it a global advantage.
The "Superintelligence Strategy" paper argues that destabilizing AI developments could disrupt global power and increase the risk of conflict. Top AI minds see a high-stakes race between superpowers. They anticipate superintelligence—AI surpassing humans in cognitive abilities—requiring strategies analogous to nuclear deterrence. They propose "Mutual Assured AI Malfunction" (MAIM), a strategy where aggressive bids for AI dominance would be met with preventative sabotage by rivals, including potential kinetic strikes on data centers. This aims to prevent both AI monopolization and access by rogue actors.
Their three-pronged strategy includes deterrence (detecting and deterring destabilizing projects through cyber espionage and sabotage), competitiveness (maintaining a technological edge), and non-proliferation (preventing access by malicious actors). OpenAI has presented similar proposals. The paper highlights the risks of superintelligence, including unprecedented military capabilities and bioweapons. A nation possessing superintelligence could gain dominance akin to the conquistadors' conquest of the Aztecs. The authors discuss the potential for an "intelligence explosion," where AI autonomously improves itself, outpacing human oversight. They note the shift towards reinforcement learning (RL) in AI training, leading to more adaptable and potentially superhuman AI, exemplified by AlphaGo's "Move 37."
The paper examines existing strategies: hands-off ("move fast and break things"), moratoriums (pausing development), and monopolies (one nation's dominance). MAIM is presented as the likely default, with states sabotaging rivals' AI projects. This could create a stalemate, delaying superintelligence and preventing monopolies. Maintaining equilibrium requires clear signals to avoid escalation, developing geographically dispersed data centers, and distinguishing between acceptable and destabilizing AI projects. This approach echoes the Open Skies Treaty.
The paper emphasizes competitiveness. A Chinese invasion of Taiwan, the world's primary AI chip producer, would cripple Western AI capabilities, while China’s drone technology lead presents a further military challenge. This could make China a unipolar economic and military superpower. AI chips are crucial for economic power, as AI effectively transforms capital into labor. The paper suggests securing AI chip supply chains, possibly through domestic manufacturing.
Military strength is crucial, requiring integrating AI into command and control and cyber offense, while maintaining human oversight of critical decisions to avoid accidental escalation. Non-proliferation necessitates securing high-end AI chips and preventing their illegal diversion. Alex Wang noted DeepSeek’s acquisition of numerous high-end chips, highlighting the effectiveness—though not foolproof—of export controls. Amodei agrees that strong export controls are essential to prevent China from obtaining millions of chips, although DeepSeek's progress seems compliant with existing regulations. Information security is also vital, as leaked models could be easily replicated. By strengthening compute, information, and AI security, the risk of AI becoming a tool of terror can be reduced.
The paper concludes by summarizing various perspectives: the "doomers," the "ostriches," and a risk-conscious approach. It advocates for MAIM, non-proliferation, and leveraging AI for societal benefit. It suggests that during a period of economic growth fueled by AI, a slower, multilateral approach to superintelligence development, with shared benefits and reduced risk, might be possible. A longer, more detailed expert version is available. The proposed MAIM strategy acts as a "speed limit" on AI development, deterring any single nation from achieving a dangerous lead, preventing escalation to open conflict. This approach, while potentially fostering a degree of détente, relies on the shared goal of averting catastrophic outcomes.
Chinese researchers have also highlighted the risks of self-replicating AI, emphasizing the need for international collaboration on safety regulations, mirroring concerns voiced by Western researchers. Amodei advocates for democratic nations leading AI development to ensure safety and prevent autocratic regimes from gaining a military advantage. However, concerns exist that this could lead to a reckless race for powerful AI, neglecting safety measures.
Anthropic's recommendations to the White House underscore these concerns, emphasizing national security testing, strengthening export controls, enhancing lab security, and preparing for economic disruptions from advanced AI. Recent interviews with Ben Buchanan, a former White House advisor, highlight the unpreparedness for the transformative impact of AI, and the fear of unchecked state control empowered by AI surveillance. The potential for misuse in autocratic regimes, alongside the promise of immense progress, underscores the critical juncture of AI development. Predicting the future impact of this technology is a monumental task, emphasizing the immense significance of this historical moment.
It's almost nearly impossible. If we get it right, it can be unimaginably good. If we get it wrong, it can be unimaginably bad. It's hard for me to see a scenario where not too many things change. I guess that's a possibility; I just don't see how.
But the point is, what do you think about this proposal? So Eric Schmidt, Dan Hendrick, Alexander Wayne—this idea of MIM deterrence, so that no one nation is able to just sprint ahead to superintelligence. But a sort of equilibrium where we sort of advance and focus on acceptable tasks, things that provide benefits to the people. And hopefully, if we're able to kind of hold that, it would allow for some of these tensions to de-escalate, for more international cooperation to form. So instead of conflict and competition and tension and potentially military actions, we would be able to kind of be a little bit more aligned.
Do you think that plan could work? Maybe you want a full stop to the progress, or perhaps you want just everybody to kind of race ahead and see what happens. Or maybe you're more in line with the accelerationists—just full steam ahead, we've got to get there first. Where do you fall in with all of this? What do you think is the right approach? If you had to bet, what would you bet as the right path forward?
Keep in mind, this is for all the marbles. If you made it this far, thank you so much for watching. My name is Wes Roth, and I'll see you next time.