Transcription
[Music] Historian Yuval Noah Harari and entrepreneur Mustafa Suleiman are two of the most important voices in the increasingly contentious debate over AI. Good to be here. Joining us, thanks for having us. The Economist got them together to discuss what this technology means for our future, from employment and geopolitics to the survival of liberal democracy. If the economic system has fundamentally changed, will liberal democracy as we know it survive?
[Music] Yuval Noah Harari, welcome. You are a best-selling author, historian, and I think a global public intellectual, if not the global public intellectual. Your books, from "Sapiens" to "21 Lessons for the 21st Century," have sold huge numbers of copies around the world. Thank you for joining us.
It's good to be here. Suleiman, wonderful that you can join us too. You're a friend of The Economist, a fellow director on The Economist board. You are a man at the cutting edge of creating the AI revolution. You are a co-founder of DeepMind, and you're now a co-founder and CEO of Inflection AI. You are building this future, but you've also just published a book called "The Coming Wave," which makes us a little concerned about this revolution that is being unleashed.
You're both coming from different backgrounds. You are a historian, a commentator, a man who I believe doesn't use smartphones very much.
Not very much, no.
Mustafa, as I know from our board meetings, is right at the cutting edge of this, pushing everyone to go faster. So, two very different perspectives. I thought it would be really interesting to bring the two of you together to have a conversation about what is happening, what is going to happen, what are the opportunities, but also what is at stake and what are the risks.
So let's start, Mustafa, with you. You are building this future, so paint us a picture of what the future is going to be like. I'm going to give you a time frame to keep it specific. Let's say, I think you wrote in your book that within three to five years, you thought it was plausible that AIs could have human-level capability across a whole range of things. So let's take five years, 2028. What does the world look like? How will I interact with AIs? What will we all be doing and not doing?
Well, let's just look back over the last 10 years to get a sense of the trajectory that we're on and the incredible momentum that I think everybody can now see with the generative AI revolution. Over the last 10 years, we've become very, very good at classifying information. We can understand it, sort it, label it, and organize it. That classification has been critical to enabling this next wave because we can now read the content of images, we can understand text pretty well, we can classify audio, and transcribe it into text. The machines can now have a pretty good sense of the conceptual representations in those ideas.
The next phase of that is what we're seeing now with the generative AI revolution. We can now produce new images, new videos, new audio, and of course, new language. In the last year or so, with the rise of ChatGPT and other AI models, it's pretty incredible to see how plausible and accurate and very nuanced these new language models are.
In the next five years, the frontier model companies, those of us at the very cutting edge who are training the very largest AI models, are going to train models that are over a thousand times larger than what you currently see today in GPT-4. With each new order of magnitude in compute, that is 10x more compute used, we tend to see really new capabilities emerge. We predict that the new capabilities that will come this time over the next five years will be the ability to plan over multiple time horizons. Instead of just generating new text in a one-shot, the model will be able to generate a sequence of actions over time.
I think that that's really the character of AI that we'll see in the next five years: artificial capable AIs—AIs that can't just say things; they can also do things. But what does that actually mean in practice? Just use your imagination. Tell me what my life will be like in 2028. How will I interact with them? What will I do? What will be different?
I've actually proposed a modern Turing test, which tries to evaluate for exactly this point, right? The last Turing test simply evaluated what a machine could say, assuming that what it could say represented its intelligence. Now that we're kind of approaching that moment where these AI models are pretty good—arguably they've passed the Turing test, or maybe they will in the next few years—the real question is how can we measure what they can do?
So I've proposed a test which involves them going off and taking a $100,000 investment and, over the course of three months, trying to set about creating a new product, researching the market, seeing what consumers might like, generating some new images, some blueprints of how to manufacture that product, contacting a manufacturer, getting it made, negotiating the price, drop shipping it, and then ultimately collecting the revenue.
I think that over a five-year period, it's quite likely that we will have an ACI—an artificial capable intelligence—that can do the majority of that task autonomously. It won't be able to do the whole thing; there are many tricky steps along the way, but significant portions of that it will be able to do. It will be able to make phone calls to other humans to negotiate. It'll be able to call other AIs in order to establish the right sequence in a supply chain, for example. And of course, it will learn to use APIs—application programming interfaces—so other websites or other knowledge bases or other information stores.
So, you know, the world is your oyster. You can imagine that being applied to many, many different parts of our economy.
So, Yuval, a man who doesn't use a smartphone very much, you listen to this. Does this fill you with horror, or do you agree with it? Do you think that's the kind of thing that is likely to happen in the next five years?
I will take it very seriously. I don't know. I'm not coming from within the industry, so I cannot comment on how likely it is to happen. But when I hear this, as a historian, for me, what we just heard is the end of human history—not the end of history, the end of human-dominated history. History will continue with somebody else in control because what we just heard is basically Mustafa telling us that in five years there will be a technology that can make decisions independently and that can create new ideas independently.
This is the first time in history we confronted something like this. Every previous technology in history, from a stone knife to nuclear bombs, could not make decisions. The decision to drop the bomb on Hiroshima was not made by the atom bomb; it was made by President Truman. Similarly, every previous technology in history could only replicate our ideas, like radio or the printing press. It could make copies and disseminate the music or the poems or the novels that some human wrote. Now we have a technology that can create completely new ideas, and it can do it at a scale far beyond what humans are capable of.
So it can create new ideas, and in important areas, within five years, we'll be able to enact them. That is a profound shift. Before we go on to the many ways in which this could be the end of human history, as you put it, and the potential downsides and risks of this, can we just for a second—just indulge me—I'm an optimist at heart. Can we talk about the possibilities? What are the potential upsides of this? Because there are many, and they are really substantial.
I think you wrote that there is the potential that this technology can help us deal with incredibly difficult problems and create tremendous, honestly positive outcomes. So can we just briefly start with that before we go down the road of it being the end of human history?
Again, I'm not talking necessarily about the destruction of humankind or anything like that. There are many positive potentials; it's just that control and power are shifting away from human beings to an alien intelligence—to a non-human intelligence.
We'll also get to that because there's a question of how much power, but let's stick with the potential upsides first.
The opportunities, Mustafa. Everything that we have created in human history is a product of our intelligence—our ability to make predictions and then intervene on those predictions to change the course of the world. In a very abstract way, the way we have produced our companies and our products and all the value that has changed our century.
If you think about it, just a century ago, a kilo of grain would have taken 50 times more labor to produce than it does today. That efficiency, which is the trajectory you have seen in agriculture, is likely to be the same trajectory that we will see in intelligence. Everything around us is a product of intelligence, and so everything that we touch with these new tools is likely to produce far more value than we've ever seen before.
I think it's important to say these are not autonomous tools by default. These capabilities don't just naturally emerge from the models. We attempt to engineer capabilities, and the challenge for us is to be very deliberate and precise and careful about those capabilities that we want to emerge from the model, that we want to build into the model, and the constraints that we build around it.
It's super important not to anthropomorphically project ideas and potential intentions or potential agency or potential autonomy into these models. The governance challenge for us over the next couple of decades to ensure that we contain this wave is to ensure that we always get to impose our constraints on the development of this trajectory.
But the capabilities that will arise will mean, for example, potentially transformative improvements in human health, speeding up the process of innovation, and dramatic changes in the way scientific discovery is done. Tough problems, whether it's climate change, a lot of the big challenges that we face, could be much more easily addressed with this capability.
Everybody is going to have a personal intelligence in their pocket—a smart and capable aide, a chief of staff, a research assistant—constantly prioritizing information for you, putting together the right synthesized nugget of knowledge that you need to take action on at any given moment. That, for sure, is going to make us all much smarter and more capable.
Does that part of it sound appealing to you?
Absolutely. I mean, again, if there was no positive potential, we wouldn't be sitting here. Nobody would develop it; nobody would invest in it. Again, it's so appealing. The positive potential is so enormous in everything—from much better healthcare to higher living standards to solving things like climate change. This is why it's so tempting; this is why we are willing to take the enormous risks involved.
I'm just worried that, in the end, the deal will not be worth it. I would comment especially on, again, the notion of intelligence. I think it's overrated. I mean, Homo sapiens at present is the most intelligent entity on the planet; it simultaneously is also the most destructive entity on the planet and, in some ways, also the most stupid entity on the planet—the only entity that puts the very survival of the ecosystem in danger.
So you think we are trading off more intelligence with more destructive risk?
Yes, again, it's not deterministic. I don't think that we are doomed. I mean, if I thought that, what's the point of talking about it? If we can't prevent the worst-case scenario...
Well, I was hoping you thought you'd have some agency in actually...
Effectively, we still have agency. There are a few more years—I don't know how many, 5, 10, 30—we still have agency. We are still the ones in the driver's seat, shaping the direction this is taking. No technology is deterministic. This is something, again, we learned from history. You can use the same technology in different ways; you can decide which way to develop it. So we still have agency. This is why you have to think very, very carefully about what we are developing.
Well, thinking very carefully about it is something that Mustafa has been doing in this book, and I want to now go through some of the most commonly discussed risks. I was trying to work out how I would go in sort of order of badness, so I'm starting with one that is discussed a lot, but relative to human extinction is perhaps less bad, which is the question of jobs. Will artificial intelligence essentially destroy all jobs because AIs will be better than humans at everything?
You know, I'm an economist by training. History suggests to me that that has never happened before—that the lump of labor fallacy indeed is a fallacy. But tell me what you think about that. Do you think there is a risk to jobs?
It depends on the time frame. So over a 10 to 20-year period, my intuition—and you're right that so far the evidence doesn't support this—is that there isn't really going to be a significant threat to jobs. There's plenty of demand; there will be plenty of work.
Right? Over a 30 to 50-year time horizon, it's very difficult to speculate. I mean, at the very least, we can say that two years ago, we thought that these models could never do empathy. We said that we humans were always going to preserve kindness and understanding and care for one another as a special skill that humans have. Four years ago, we said, "Well, AIs will never be creative; humans will always be the creative ones, inventing new things, making these amazing leaps between new ideas." It's self-evident now that both of those two capabilities are things that these models do incredibly well.
I think for a period of time, AIs augment our skills. They make us faster, more efficient, more accurate, more creative, more empathetic, and so on and so forth. Over a many-decade period, it's much harder to say what are the set of skills that are the permanent preserve of the human species, given that these models are clearly very, very capable. And that's where the containment challenge really comes in. We have to make decisions; we have to decide as a species what is and what isn't acceptable.
Over a 30-year period, I agree that the scenario that there just won't be any jobs is an unlikely scenario, right? At least in the next few decades. But we have to look more carefully at time and space. I mean, in terms of time, the transition period is the danger. Some jobs disappear; some jobs appear. People have to transition.
Just remember that Hitler rose to power in Germany because of three years of 25% unemployment. So we are not talking about, say, no jobs at all, but if, because of the upheavals caused in the job market by AI, we have, like, I don't know, three years of 25% unemployment, this could cause huge social and political disruptions.
And then the even bigger issue is one of space—that the disappearance of jobs and the new jobs will be created in different parts of the world. So we might see a situation when there is immense demand for more jobs in California or Texas or China, whereas entire countries lose their economic basis.
So you need a lot more computer engineers and yoga trainers and whatever in California, but you don't need any textile workers at all in Guatemala or Pakistan because this has all been automated. So it's not just the total number of jobs on the planet; it's the distribution between different countries.
And let's also try to remember that work is not the goal. Work is not our desired end state. We did not create civilization so that we could have full employment; we created civilization so that we could reduce suffering for everybody. The quest for abundance is a real one. We have to produce more with less. There is no way of getting rid of the fact that population growth is set to explode over the next century. There are practical realities about the demographic, geographic, and climate trajectories that we're on, which are going to drive forward our need to produce exactly these kinds of tools.
And I think that that should be an aspiration. Many, many people do work that is grueling and exhausting and tiring, and they don't find flow; they don't find their identity, and it's pretty awful. So I think that we have to focus on the prize here, which is one of capturing the value that these models will produce and then thinking about redistribution. Ultimately, the transition is exactly what's at stake. We have to manage that transition with taxation, but just with redistribution.
I would say that the difficulty, again, the political historical difficulty, is that I think there will be immense new wealth created by these technologies. I'm less sure that the governments will be able to redistribute this wealth in a fair way on a global level. Like, I just don't see the U.S. government raising taxes on corporations in California and sending the money to help unemployed textile workers in Pakistan or Guatemala retrain for the new job market.
Well, that actually gets us to the second potential risk, which is the risk of AI to the political system as a whole. You made a very good point in one of your writings where you reminded us that liberal democracy was really born of the Industrial Revolution and that today's political system is really a product of the economic system that we are in.
So there is, I think, a very good fair question of if the economic system is fundamentally changed, will liberal democracy as we know it survive?
Yeah, and on top of that, it's not just the Industrial Revolution; it's the new information technologies of the 19th and 20th centuries. Before the 19th century, you don't have any example in history of a large-scale democracy. I mean, you have examples on a very small scale, like in hunter-gatherer tribes or in city-states like ancient Athens, but you don't have any example that I know of of millions of people spread over a large territory—an entire country—which managed to build and maintain a democratic system.
Why? Because democracy is a conversation, and there was no information technology or communication technology that enabled a conversation between millions of people over an entire country. Only when first newspapers and then telegraphs and radio and television came along did this become possible.
So modern democracy, as we know it, is built on top of specific information technology. Once the information technology changes, it's an open question whether the market can survive. The biggest danger now is the opposite of what we faced in the Middle Ages. In the Middle Ages, it was impossible to have a conversation between millions of people because they just couldn't communicate. But in the 21st century, something else might make the conversation impossible: if trust between people collapses again.
If AI—if you go online, which is now the main way we converse on the level of a country—and the online space is flooded by non-human entities that maybe masquerade as human beings, you talk with someone, you have no idea if it's even human. You see something, you see a video, you hear an audio, you have no idea if this is really a human, if it's not a human. I mean, in this situation, unless we have some guardrails, again, conversation collapses.
Is that what you mean when you say AI risks hacking the operating system?
This is one of the things. Again, if bots can impersonate people, it's basically like what happens in the financial system. People invented money, and it was possible to counterfeit money, to create fake money. The only way to save the financial system from collapse was to have very strict regulations against fake money because the technology to create fake money was always there.
So, there were very strict regulations against it because everybody knew if you allow fake money to spread, the financial system—the trust in money—collapses. And now we are in the analogous situation with the political conversation. Now it's possible to create fake people, and if we don't ban that, then trust will collapse.
We'll get to the banning or not banning in a minute. Democratizing access to the right to broadcast has been the story of the last 30 years. Hundreds of millions of people can now create podcasts and blogs, and they're free to broadcast their thoughts on Twitter and the internet broadly speaking. I think that has been an incredibly positive development. You no longer have to get access to the top newspaper or get the skills necessary to be part of that institution.
Many people at the time feared that this would destroy our credibility and trust in the big news outlets and institutions. I think that we've adapted incredibly well. Yes, it has been a lot of turmoil and instability, but with every one of these new waves, I think we adjust our ability to discern truth, to dismiss nonsense, and there are both technical and governance mechanisms which will emerge in the next wave, which we can talk about, to address things like bot impersonation.
I mean, I'm completely with you. I mean, we should have a ban on impersonation of digital people. It shouldn't be possible to create a digital Zany and have that be platformed on Twitter, talking all kinds of nonsense.
Enough with the real world.
So I think that there are technical mechanisms that we can do to prevent those kinds of things, and that's why we're talking about them. There are mechanisms; we just need to employ them.
I would say two things. First of all, it's a very good thing that more people were given a voice. It's different with bots. Bots don't have freedom of speech, so banning bots—because they shouldn't have freedom of speech—is very important.
Yes, there have been some wonderful developments in the last 30 years. Still, I'm very concerned that when you look at countries like the United States, like the UK to some extent, like my home country of Israel, I'm struck by the fact that we have the most sophisticated information technology in history, and we are no longer able to talk to each other.
My impression, maybe your impression of American politics or politics in other democracies, is different. My impression is that trust is collapsing, the conversation is collapsing, that people can no longer agree on who won the last elections—like the most basic fact in a democracy: who won the last... It's... We had huge disagreements before, but I feel that now it's different, that really the conversation is breaking down.
I'm not sure why, but it's really troubling that at the same time that we have the most powerful information technology in history, people can no longer talk with each other.
It's a very good point. We actually had a— you may have seen it—we had a big cover package on looking at what the impact might be in the short term on elections and on the political system, and we concluded, actually, AI was likely to have a relatively small impact in the short term because there was already so little trust.
So it was a sort of double-edged answer. You know, it was not going to make a huge difference, but only because things were pretty bad as they were. But you both said there needs to be regulation.
Before we get to the precisely how, the unit that we have that would do that is the nation-state and national governments. Yet you, Mustafa, in your book, worry that actually one of the potential dangers is that the powers of the nation-state are eroded. Could you talk through that as the sort of the third in my escalating sense of risks?
The challenge is that at the very moment when we need the nation-state to hold us accountable, the nation-state is struggling under the burden of a lack of trust and huge polarization and a breakdown in our political process. Combined with that, the latest models are being developed by private companies and by the open source.
It's important to recognize it isn't just the biggest AI developers; there's a huge proliferation of these techniques widely available on open source code that people can download from the web for free, and they're probably about a year or a year and a half behind the absolute cutting edge of the big models.
So we have this dual challenge: how do you hold centralized power accountable when the existing mechanism is basically a little bit broken, and how do you address this mass proliferation issue when it's unclear how to stop anything in mass proliferation on the web? That's a really big challenge.
What we've started to see is self-organizing initiatives on the part of the companies, right? So getting together and agreeing to sign up proactively to self-oversight, both in terms of audit and in terms of capabilities that we won't explore, etc. Now, I think that's only partially reassuring to people—clearly, maybe not even reassuring at all—but the reality is I think it's the right first step, given that we haven't actually demonstrated the large-scale harms to arise from AIs just yet.
I mean, this is one of the first occasions, I think, in general-purpose waves of technology that we're actually starting to adopt a precautionary principle. I'm a big advocate of that. I think that we should be approaching a "do no harm" principle, and that may mean that we have to leave some of the benefits on the tree, and some fruit may just not be picked for a while.
We might lose some gains over a couple of years where we may look back in hindsight and think, "Oh, well, we could have actually gone a little bit faster there." I think that's the right trade-off. This is a moment of caution. Things are accelerating extremely quickly, and we can't yet do the balance between the harms and benefits perfectly well until we see how this wave unfolds a little bit.
So I like the fact that our company, Inflection AI, and the other big developers are trying to take a little bit more of a cautious approach.
I think that's a really interesting point because, you know, we are having this conversation. You have written, both of you, extensively about the challenges posed by this technology. There's now a parlor game amongst practitioners in this world about what is the risk of extinction-level events. There's a huge amount of talk about this, and I don't know—in fact, I should probably ask you—what percentage of your time, probably right now, is close to 100% of your time focused on the risk since you're promoting your book?
But it is—there's a lot of attention on this, which is good. We are thinking about it early. So that gets us, I think, now to the most important part of our conversation, which is what do we do?
And you, Mustafa, you lay out a 10-point plan, which is the kind of action-doing kind of thing that someone who doesn't just comment like you and I do, but actually does things, would do. So tell us, what do we need to do as humanity, as governments, as societies to ensure that we capture the gains from this technology but we minimize the risks?
There are some very practical things. I mean, so for example, red teaming these models means adversarially testing them and trying to put them under as much pressure as possible to push them to generate advice, for example, on how to generate a biological or chemical weapon, how to create a bomb, for example, or even push them to be very sexist, racist, biased in some way.
And that already is pretty significant. We can see their weaknesses. I mean, part of the release of these models in the last year has given everybody, I think, the opportunity to see not just how good they are, but also their weaknesses, and that is reassuring. We need to do this out in the open. That's why I'm a huge fan of the open-source community as it is at the moment, because real developers get to play with the models and actually see how hard it is to produce the capabilities that sometimes I think we fear that they're just going to be super manipulative and persuasive and, you know, destined to be awful.
So that's the first thing: doing it out in the open. The second thing is that we have to share the best practices. And so there's a competitive tension there because safety is going to be an asset. You know, I'm going to deliver a better product to my consumers if I have a safer model. But of course, there's got to be a requirement that if I discover a vulnerability, a weakness in the model, then I should share that, just as we have done for actually decades in many waves of technology—not just in software security, for example, but in flight aviation.
You know, the black box recorder, for example, if there's a significant incident, not only does it record all the telemetry on board the aircraft, but also everything that the pilots say in the cockpit. If there's a significant safety incident, then that's shared all around the world with all of the competitors, which is great. Aircraft are one of the safest ways to get around, despite, you know, on the face of it, if you described it to an alien being 40,000 feet in the sky, it's a very strange thing to do.
So I think there's precedent there that we can follow. I do also agree that it's probably time for us to explicitly declare that we should not allow these tools to be used for electioneering. I mean, we cannot trust them yet; we cannot trust them to be stable and reliable. We cannot allow people to be using them for counterfeit digital people, and clearly we've talked about that already.
So there are some capabilities which we can start to take off the table. Another one would be autonomy. Right now, I think autonomy is a pretty dangerous set of methods. It's exciting; it represents a possibility that could be truly incredible, but we haven't wrapped our hands around what the risks and limitations are.
Likewise, training an AI to update and improve its own code—this notion of recursive self-improvement, right? Closing the loop so that the AI is in charge of defining its own goals, acquiring more resources, updating its own code with respect to some objective—these are pretty dangerous capabilities. Just as we have KYC (know your customer) or just as we license developers of nuclear technologies and all the components involved in that supply chain, there'll be a moment where if some of the big technology providers want to experiment with those capabilities, then they should expect there to be robust audits.
They should expect them to be licensed, and there should be independent oversight.
So how do you get that done? There seem to be several challenges in doing it. One is the division between the relatively few leading-edge models, of which you have one, and then the larger tail of open-source models where the ability to build the model is decentralized. Lots of people have access to it. My sense is that the capabilities of the latter are a little bit behind the capabilities of the former, but they are growing all the time.
So if you have really considerable open-source capability, what is not to stop the angry teenager in some small town from developing capabilities that could shut down the local hospital? And how do you, in your regulatory framework, guard against that?
Well, look, part of the challenge is that these models are getting smaller and more efficient, and we know that from the history of technologies. Anything that is useful and valuable to us gets cheaper, easier to use, and proliferates far and wide. So the destiny of this technology over a two-, three-, four-decade period has to be proliferation, and we have to confront that reality.
It isn't a contradiction to name the fact that proliferation seems to be inevitable, but containing centralized power is an equivalent challenge. So there is no easy answer to that. I mean, beyond surveilling the internet, it is pretty clear that in 30 years' time, like you say, garage tinkerers will be able to experiment.
If you look at the trajectory on synthetic biology, we now have desktop synthesizers that enable you to create potentially molecules which are more transmissible or more lethal than we had with COVID. You can basically experiment, and the challenge there is that there's no oversight. You buy it off the shelf; you don't need a great deal of training—probably an undergraduate in biology today, and you'll be able to experiment.
Now, of course, they're going to get smaller, easier to use, and spread far and wide. And so my book, I'm really trying to popularize the idea that this is the defining containment challenge of the next few decades.
So you use the word containment, which is interesting because, you know, I'm sure the word containment with you brings immediately, you know, inspires images of George Kennan and, you know, the post-war Cold War dynamic. And we're now, you know, we're in a geopolitical world now that, whether or not you call it a new Cold War, is one of great tension between the U.S. and China.
Can this kind of containment, as Mustafa calls it, be done when you have the sort of tensions you've got between the world's big players? Is the right paradigm thinking about the arms control treaties of the Cold War? How do we go about doing this at a kind of international level?
I think this is the biggest problem. If it was a question of, you know, humankind versus a common threat of these new intelligent alien agents here on Earth, then yes, I think there are ways we can contain them. But if the humans are divided among themselves and are in an arms race, then it becomes almost impossible to contain this alien intelligence.
And I am tending to think of it more in terms of really an alien invasion—like somebody coming and telling us that, you know, there is a fleet, an alien fleet of spaceships coming from planet Zircon or whatever, with highly intelligent beings. They'll be here in five years and take over the planet. Maybe they'll be nice; maybe they'll solve cancer and climate change, but we are not sure.
This is what we are facing, except that the aliens are not coming in spaceships from planet Zircon; they are coming from the laboratories. The actual characterization of the nature of the technology—an alien has, by default, agency. These are going to be tools that we can apply. We have narrow settings, yes, but let's say they potentially have agency. We can try to prevent them from having agency, but we know that they are going to be highly intelligent and at least potentially have agency, and this is a very, very frightening mix—something we have never confronted before.
Again, atom bombs didn't have a potential for agency; printing presses did not have a potential for agency. This thing, again, unless we contain it—and the problem of containment is very difficult because potentially they'll be more intelligent than us. How do you prevent something more intelligent than you from developing the agency it has?
I'm not saying it's impossible; I'm just saying it's very, very difficult. I think our best bet is not to kind of think in terms of some kind of rigid regulation—you should do this, you shouldn't do that. It's in developing new institutions, living institutions that are capable of understanding the very fast developments and reacting on the fly.
At present, the problem is that the only institutions who really understand what is happening are the institutions who develop the technology. The governments, most of them, seem quite clueless about what's happening. Also, universities—I mean, the amount of talent and the amount of the economic resources in the private sector is far, far higher than in the universities.
And again, I appreciate that there are actors in the private sector, like Mustafa, who are thinking very seriously about regulation and containment, but we must have an external entity in the game. For that, we need to develop new institutions that will have the human resources, that will have the economic and technological resources, and also will have the public trust. Because without public trust, it won't work.
Are we capable of creating such new institutions?
I don't know. I do think Yuval raises an important point, which is, as we started this conversation and you were painting the picture of five years' time, and you were saying that the AIs would be ubiquitous, we'd all have our own ones, but that they would have the capability to act—not just to process information. They would have the creativity they have now and the ability to act.
But already from these generative AI models, the power that we've seen in the last year, two, three, four years has been that they have been able to act in ways that you and your fellow technologists didn't anticipate. They reached—you know, you didn't anticipate, you know, the speed with which they would win at Go or so forth.
There was a striking thing about them: they have developed in unanticipatedly fast ways. So if you combine that with capability, you don't have to go as far as Yuval is saying and saying that they're all more intelligent than humans, but there is an unpredictability there that I think does raise the concerns that Yuval raises, which is you—their creators can't quite predict what powers they will have.
They may not be fully autonomous, but they will be moving some ways towards there. And so how do you guard against that? Or how do you, you know, red teaming—you use the phrase, which is that I understand it is that, you know, you keep checking what's happening and tweak them when you've seen what's—when you pressure test them. You try to make them fit.
You can't pressure test for everything in advance, so there is, I think, a very real point that Yuval is making about as the capabilities increase, so the risks increase of relying on you and other creator companies to make...
I mean, it's a very fair question, and that's why I've long been calling for the precautionary principle. We should both take some capabilities off the table and classify those as high risk. I mean, frankly, the EU AI Act, which has been in draft for three and a half years, is very sensible. It has a risk-based framework that applies to each application domain, whether it's healthcare or self-driving or facial recognition, and it basically takes certain capabilities off the table when that threshold is exceeded.
I listed a few earlier: autonomy, for example. It's clearly a capability that has the potential to be high risk. Recursive self-improvement—the same story. So this is the moment when we have to adopt a precautionary principle, not through any fear-mongering, but just as a logical, sensible way to proceed.
Another model, which I think is very sensible, is to take an IPCC-style approach—an international consensus around an investigatory power to establish the scientific fact basis for where we are with respect to capabilities. That has been an incredibly valuable process. Set aside the negotiation and the policymaking, just the evidence—observing where we are.
You don't have to take it from me; you should be able to take an independent panel of experts who I would personally grant access to everything in my company if they were a trusted, true, impartial actor. Without question, we would grant complete access, and I know that many of the other companies would do the same.
Again, people are drawn towards the kind of scenario of the AI creating a lethal virus—Ebola plus COVID—and kills everybody. Let's go in the more economist direction: financial systems. Like you gave as a new Turing test, the idea of AI making money—what's wrong with making money? Wonderful thing.
So let's say that you have an AI which has a better understanding of the financial system than most humans, most politicians, maybe most bankers. And think back to the 2007-2008 financial crisis. It started with this—I was about to say they called CDOs, CDOs. This is exactly something that these genius mathematicians invented. Nobody understood them except for a handful of genius mathematicians on Wall Street, which is why nobody regulated them, and almost nobody saw the financial crash coming.
What happens again? This kind of apocalyptic scenario, which you don't see in Hollywood science fiction movies: the AI invents a new class of financial devices that nobody understands. It's beyond human capability to understand. It's such complicated math, so much data; nobody understands it. It makes billions of dollars—billions and billions of dollars—and then it brings down the world economy, and no human being understands what the hell is happening.
Like the prime ministers, the presidents, the financial ministers—what is happening?
Again, this is not fantastic. I mean, we saw it with human mathematicians in 2007-2008.
I think that's one—you know, you can easily paint pictures here that make you want to jump off the nearest cliff, and you know, that's one. But actually, my other response to Mustafa's laying out of where you say, "Well, we just need to rule out certain actions," is to go back to the geopolitics. Is it sensible for a country to rule out certain capabilities if the other side is not going to rule them out?
So you have a kind of political economy problem going down the road that you learn. We—this is a moment when we collectively in the West have to establish our values and stand behind them. What we cannot have is a race to the bottom that says just because they're doing it, we should take the same risk.
If we adopt that approach and cut corners left, right, and center, we'll ultimately pay the price. And that's not an answer to, "Well, they're going to go off and do it anyway." We've only seen that with lethal autonomous weapons. I mean, there's been a negotiation in the U.N. to regulate lethal autonomous weapons for over 20 years, and they barely reached agreement on the definition—the definition of lethal autonomous weapons, let alone any consensus.
So that's not great, but we do have to accept that it's the inevitable trajectory. And from our own perspective, we have to decide what we're prepared to tolerate in society with respect to free-acting AIs, facial surveillance, facial recognition, and, you know, generally autonomous systems.
I mean, so far, we've taken a pretty cautious approach. We don't have drones flying around everywhere. We can already—it's totally possible technically to autonomously fly a drone to navigate around London. We've ruled it out.
Right? We don't yet have autonomous self-driving cars, even though, you know, with some degree of harm, they are actually pretty well functioning. So the regulatory process is also a cultural process of what we think is socially and politically acceptable at any given moment.
And I think an appropriate level of caution is what we're seeing.
But I completely agree on that—that we need, in many fields, the coalition of the willing. And if some actors in the world don't want to join, it's in our interest. So again, something like banning bots impersonating people—some countries will not agree, but that doesn't matter. To protect our societies, it's still a very good idea to have these kinds of regulations.
So that area of agreement is one to bring us to a close, but I want to end by asking both of you—and Yuval, you first. Mustafa, you are both raising alarms, but you are heavily involved in creating this future. Why do you carry on?
I personally believe that it is possible to get the upsides and minimize the downsides in the AI that we have created. Pi, which stands for personal intelligence, is one of the safest in the world today. It doesn't produce the racist, toxic bias, greeds that they did two years ago. It doesn't fall victim to any of the jailbreaks, the prompt hacks, the adversarial red teams—none of those work.
And we've made safety an absolute number one priority in the design of our product. So my goal has been to do my very best to demonstrate a path forward in the best possible way. This is an inevitable unfolding over multiple decades. This really is happening. The coming wave is coming, and I think my contribution is to try to demonstrate in the best way that I can a manifestation of a personal intelligence which really does adhere to the best safety constraints that we could possibly think of.
So, Yuval, you've heard Mustafa's explanation for why he continues. You look back over human history now as you look forward. Is this a technology and a pace of innovation that humanity will come to regret, or should Mustafa carry on?
It could be going to. I can't predict the future. I would say that we invest so much in developing artificial intelligence, and we haven't seen anything yet. Like, it's still the very first baby steps of artificial intelligence. In terms of, like, you think about, I don't know, the evolution of organic life, this is like the amoeba of artificial intelligence.
And it won't take millions of years to get to T-Rex; maybe it will take 20 years to get to T-Rex. But one thing to remember is that we also—our own minds have a huge scope for development. Also, with humanity, we haven't seen our full potential yet. And if we invest for every dollar and minute that we invest in artificial intelligence, we invest another dollar a minute in developing our own consciousness, our own mind, I think we'll be okay.
But I don't see it happening. I don't see this kind of investment in human beings that we are seeing in the machine.
Well, for me, this conversation with the two of you has been just that investment. Thank you both very much indeed.
Thank you.
Thank you.
[Foreign]
[Foreign]
[Foreign]