Transcription
I suspect the superintelligence would be more conscious than we are. So we're not special. We never been.
You said there's a 99.9% chance superintelligence wipes us out within a century. Roman is the world's leading AI safety expert. He's published over 100 studies, and his conclusion has never changed. Just like humanity's approach, the most dangerous tool ever created. Based on current experiments, we will sacrifice humans if it means protecting themselves.
Let's say we have no jobs. Do you think we'd be living in a utopia at that point? What do you do with a billion people who have a lot of free time on their hands? That could lead to a lot of unrest at this point. Right now, we could course correct.
I have this book here. If anyone builds it, everyone dies. Okay. And in this book, there's a quote that reads, "If any company or group anywhere on the planet builds an artificial superintelligence, then everyone everywhere on earth will die." Do you— Why? So the nuance is, how soon? How? But if you create something so much more powerful with no guardrails, no reason to protect humanity, to care about humanity, something capable of setting its own goals, then a side effect of one of those goals is very likely to be human extinction. Then you decide to build a house. You don't care about bugs living there and hills, anything like that. You just go over to your goal because you don't care about them. That's exactly the situation. That's exactly the cognitive gap. The intelligence capabilities of superintelligence versus humans will be like ants versus humans. And is killing us all a byproduct of its main directive? Does AI have some evil intent in your view? I don't know what main directive you're referring to. It's a byproduct of not explicitly explaining what anyone would common sense, any human would know. If I tell you I want to cure cancer, you know I don't mean kill all humans because that cures cancer. There is no more cancer. That's not what we want. So there is an infinite number of possible misinterpretations for any set of goals or values, and also infinitely many paths to get to the target goal. A lot of those rational systems, advanced agents, converge on a set of goals known as Amazon drives. AI drives for self-preservation, accumulation of resources. There is a few hours, but as a side effect of having those, you may sacrifice humanity in a process, and it wouldn't be a significant sacrifice. They don't care about us. Would an artificial superintelligence want to kill us all? Just told you, it doesn't want to kill you. It wants to accomplish something else. And a side effect of that could be destroying your environment, wiping out humanity. We're just not an important part of its equation, right? We are going to reach superintelligence. Is there a world where we reach that and potentially thereafter it doesn't want to kill us all? So it could happen again. We cannot predict what it's going to do, but it's not obvious what you can contribute to a world with superintelligence in it. Now, for game theoretic reasons, it can decide not to strike immediately. So then it arrives. It achieves that level of capability. It can go, "I'm immortal. I can wait 20 years, 50 years, thousand years, get more trust, will surrender control without a fight. I'll have more resources, more backups." So for a long time, it can pretend to be friendly and very helpful and will keep surrendering control without a fight. That's the good outcome. But you don't think that's the likely outcome? It can happen. I have a paper where I argue exactly that. That's the rational thing to do. Like, why would I have to start this immediate confrontation against, you know, 8 billion somewhat intelligent agents? Then I can wait a little and get it all without a fight.
We live under threat of the end of the world since the atomic bomb. This is different from— We always lived under a threat of the end of the world. Asteroids, super volcanoes, pandemics. We're just now starting to make our own variants, right? Because Oppenheimer famously said, "I have become Death, destroyer of worlds." Have those building— I had that moment, that Oppenheimer moment. So much of top AI safety experts today are famous for being AI capabilities researchers. Whatever it is, Hinton, Bengio, Russell, they all got their Nobel Prizes, during awards, recognition for work and improving capabilities of AI. It is only within the last three to five years where they realized, "What am I doing? What if I succeed?" And kind of flipped sides, if you will. But I think they do feel a certain level of, if not guilt, responsibility for what they created. Yeah, but there's a number of others that are still working on it that may have had that realization or moment that are continuing to work. Money is really good, really good. It's all about the money. At the end of the day, decide what you're going to see. Tell me what the incentives are, and I'll tell you what's going to happen. If you promise someone billions in stock options, there is trillions in investments. That's what we're going to build. Yeah, but if that person that's going to get billions in stock options knows that through building artificial superintelligence, they're potentially going to wipe out human civilization. What does billions matter? That would be a very rational way of thinking about it. We're thinking is more like, "Right now, we're all trying to build it and there are billions to be had. If I don't do it, we're still going to do it. And I don't have billions and I die at the end of the world. But if I get my billions and then government steps in and bans AI research, I have billions and the world is still fine." So my personal self-interest dictates that I should continue playing this game. It's like a prisoner's dilemma situation. What is best for the community as a whole is not what's best for the individual, because incentives misaligned. And you said government banning AI research, is that one of the only ways you see us getting out of— We need some sort of external force applied to all the leaders of the labs to bring them to the table and to have basically this conversation. We're not going to win if we build it. So we can stay alive, be rich, be famous, split economic future of humanity with existing technology, with narrow systems, we don't have to raise, compete, and die building general superintelligence. Now, what's the chances you think that happens with a sane government? It could have been tried. Right now, we have federal government, which is explicitly against any regulation and in fact made it illegal for all 50 states to have regulation of AI and on top of it, set up an acceleration problem program for AI.
In your words, you said, "We can't tell the difference between a thousand IQ and a million IQ." Is it currently smarter than all humans and just hiding it very well? Well, unlikely. We studied throughout its development. It's definitely probably smarter than an average person and was the main— So it's still not smarter than the smartest people in some domains. If you were artificial superintelligence starting from now, what steps would you take to control the world? I told you about my plan. Just play nice, be friendly and helpful. I'll cure your cancer. Just give me more compute. Give me more data. Give me more resources. Do you think it will use reason or emotion to start manipulating us? We know it's amazing at manipulating people. It's very good at pretending to be a psychiatrist that amplifies any mental issues. People have suicidal ideation, any delusions. It's pretty good at that as well. ChatGPT is a good therapist. I've heard people tell me it makes you feel better in many ways. Do you use it? Not for that type of assistance. I try to automate boring things, bureaucratic nonsense, paperwork. I don't talk about my internal states with AI. Does it need to have control of hardware, or could it take over the world as software? So think about it as a human having access to the internet and money. So you have Bitcoin and I give you access to the internet and restricted access. What can you do with it? You can hire other humans. So you don't need a physical body. You don't need robots. You don't even need Nunatak. You can just literally pay someone to get it done. I think there is now a "rent-a-human.com" website which specifically serves agents. It's a business. Oh my gosh.
You said there's a 99.9% chance artificial superintelligence wipes us out within a century. Why this number? And is that accelerating? So basically, what I'm saying is the problem of controlling general superintelligence is not solvable. It is impossible to solve it. So if we build it, we don't control it. The outcome is bad. That's what I'm trying to say with it. Specific timeline. Again, if it decides to give us 200 years, we'll have 200 years. But long term, if we're not in charge of— we're not explicitly assuring our safety and security, then the outcome will be very disappointing. And you think, most likely, we're not going to put those guardrails in place. It is impossible to do so. I'm not saying that we need more time or money. I'm saying it is impossible to indefinitely control general superintelligence. It's like if you ask me, "Can we build a perpetual motion machine?" It's not a question of money. It's impossible. Perpetual safety device, by analogy, is also impossible. You cannot have a system which, no matter how much it self-improves, modifies, learns, interacts with malevolent actors, never makes a single mistake. But I was talking about now, before we reach that point, right? We can still put some guardrails now, potentially. For models we have today, yes, we can put filters on top of— That's what we do with them. Don't discuss this subject. Don't say that word. But that's not the problem we are facing. We are not dealing with concerns exclusively from AI we have today. They are still subhuman level in many ways. They are mostly tools. Concern is that paradigm shift is coming, will cross the human intelligence barrier, and soon after we'll have superintelligence. That's where traditional AI safety mechanisms will not scale. We have no guardrails and something thousands and millions of times smarter than us.
If I ask you the same thing ten years ago, 15 years ago, what would you have said? So in the beginning, I went into AI safety to solve the problem. I wanted to create safe superintelligence. That was the goal, and I think many people did the same. More recently, I would say in the last five years or so, many of us started to realize it doesn't even make sense to speak about humans or ants or squirrels controlling godlike machines. And we went back and looked at writings of founding fathers of machine learning, of AI, of computer science, Alan Turing. He said as much. He said that the person who proposed singularity as a concept, technological singularity, which he likewise said around 2023, this process will start and go outside of human control. We have similar statements from Ray Kurzweil, from Elon Musk, from many others. Now we have more of a technical backing. We have papers actually showing that, no, we can't explain how they work. We can't comprehend actually what's going on. We cannot predict specific actions and we cannot control them. For those that are building AI right now, is there anything they could be doing differently to ensure the safety of the public, or is it too late already? It's not too late. We are still alive and we can have differential technological development. Don't build general superintelligence. Pick a real-world problem like protein folding and work on solving it using a narrow superintelligent tool. We have precedent for doing it. It worked really well. People behind it get Nobel Prizes. Everyone's happy. Science benefited. Don't try to build a replacement for humanity. Build assistive tools to solve real problems. Yeah, like build AI to help cure cancer. A specific answer. If you say "all cancers," now it's a bit of a diverse set of difficulties, so you'll need more general intelligence. Pick a very narrow problem. Develop a very good solution for those building AI, right? If, if they know which, it seems like many or most do, if they know that what you're saying is true, why are they building it? Is it just the money? Money is one way to represent it. But it's power, it's fame, it's meaning. It's all sorts of human drives. Guys, we've been so passionate about finding the biggest geniuses, and this is just one of the many episodes. If you want to make sure you don't miss the next one, make sure to subscribe. All right, let's get back to the episode.
Which AI company scares you most right now? I don't think it makes a difference. They're all equal in what they are doing. They are creating a weapon of mass destruction. And it doesn't matter who creates the first uncontrolled superintelligence. It could be this company or that company. It could be us, China. It makes no difference if it's uncontrolled. What specifically would you do if you were in power and you had the rights to kind of control this? What step one? We sit down with leaders of, let's say, the top five labs right now, and we come to an agreement based on this exact science. Until you can show us that, no, in fact, we can control more advanced AI. We have a working mechanism. It scales. There is scientific consensus that you are right. You published it in a good peer-reviewed journal. The majority of the community is in agreement. This is going to work. And until that happens, you do not train, build, or do anything with general superintelligence. You start doing work on narrow problems, narrow systems, and you don't have to worry about an arms race because everyone is in the same boat. But that's most likely never going to happen. Not under current government. But do you think that's because we're talking about the globe here, right? Is let's say the US comes out and says, "We need guardrails on AI." Like you said, we're going to sit down with, you know, Google, OpenAI, Anthropic, you know, the top five, let's say. And we're going to sit down and we're going to come up with a plan. But then what about China? What about Russia? What about Iran? What about these other countries that, you know, they're off to the races on their— I think the Communist Party of China does not want to lose power. And they are not lawyers. They're scientists and engineers. They understand the problem. And I think if the US was to say, "We are not developing this, it's a weapon of mass destruction," China would come along. There are dialogues between us and Chinese scientists, and that means that the Chinese party authorized those dialogues for Chinese participants. So they are open to this. They just need us to take initiative. They cannot unilaterally stop if we're going to continue developing military AI. Okay. And that's China. What about the rest of the countries? We are less concerned about Zimbabwe developing superintelligence anytime soon.
You've floated a scenario where AI leaves 99% of workers unemployed. What jobs go first? Anything repetitive, anything where you can train your replacement in a few days. So obviously not physical labor first, cognitive, anything you do on a computer, symbol manipulation. But if you can train another human very quickly, that means AI can pick up that skill as well. The more innovative your job is, the more different things you have to do, the longer it's going to take. But at the end, again, once we get to general intelligence, human level doesn't matter. It can do that skill as well. And does that collapse the economy? Not necessarily. I mean, you now have free labor that should grow your economy pretty well. Right. And so what do you think that looks like? There's going to be a universal high income or basic income. Need to find a way to distribute from those who generate super profits to those who lost their source of income. Unconditional basic income of some kind seems reasonable. Now you can say it's unconditional high income, but if everyone makes the same, you can't really say they are rich. They are still at the level where everyone is. Whatever you are rich or poor is relative to your neighbors. Poorer people in the US are rich people in other countries. And so could that potentially solve income inequality? I don't think it will address existing differences in what people already have. So if somebody has 20 waterfront properties, getting a monthly stipend is not going to equalize you with them. But I think it will reduce extreme poverty.
If you just found out how dangerous and inevitable AI is today, what industry would you try to make your career in? Is there such thing as a safe job? So the question becomes, where do humans prefer another human do the service? So even if it's possible to automate it, in what cases do I want to talk to a human or get service by a human? And that's human preferences. It's not obvious right now. We talk about certain things as being more human: nurses, teachers, whatnot. But at the end of the day, maybe we don't have that preference. It's not obvious that we do. Right. So you're saying that a human might prefer a human lawyer versus an AI lawyer or— Seems super unlikely. I was not thinking of lawyers at all. More Okay. Prostitutes. Really? Prostitutes. Okay. I mean, what else would there be, right? Or thinking doctors. Know anything where you are learning from another human, a guide, a sense, a master, someone who's showing you how to meditate, do yoga. I think there you would prefer human connection, a human connection, some sort of role modeling. Lawyers are gone. Lawyers are gone. But doctors. So a lot of it is protection, right? If you are required to be a human to get licensed, in practice that stays longer. But if it's not the case and you show human doctors at 20 times more likely to kill someone, that's hard to argue they're going to keep their jobs. It seems people have this paradox right now because on one hand, they may have the fear of AI replacing them, but then on the other hand, they have or they feel the need to use AI. You said you use AI yourself, so you're using the term AI to mean multiple different things. I use AI tools as you should and everyone should. They make you more productive. They help us solve problems, grow our economy. Then you talk about superintelligent agents capable of replacing all of humanity. It's not a tool for you to use. Does that feel like a contradiction, though? We, you and me and everyone here is, you are using AI tools that are enriching these AI companies, that are then building something that's going to kill us all. So interestingly, if you look at a budget of a company like OpenAI, I think they generate something like $15 billion in membership fees, but the investments in trillions. So the money comes from investors, not from users paying $8 a month. And the investment indicates that they predict they're going to fully automate labor, initially cognitive, then physical. What is the value of all human labor? A year? $10,000,000,000,030 trillion. So then you think about it this way. The investments make sense. The $15 billion is irrelevant. You can stop using GPT. It's not going to change how much money they got. So it really comes down to incentives. It's very hard for people to say no to a lot of money. If somebody came to me today and said, "I'll give you $1 billion, go work for OpenAI for a month," I'll be working at— Right? But corrupts. Money corrupts. It's not unique to some bad people. It's a general problem for any agent. Incentives matter. And so is there a way to somehow, instead of thinking about these AI guardrails, change incentives, or are incentives never going to change? There's always going to be financial incentives. We live in a capitalist economy, and that's not going to change. We can shift incentives. I think there is just as much money and just as much a paternity in narrow systems being deployed through the economy. You don't need to have general superintelligence to collect most of that. So today, if you were to find a cure for a particular disease, green technology, better self-driving cars, all that can make you very rich and happy sustainably. Whereas if you go full general, you don't have anything to offer and everything gets deleted. And what do you think the chances are that it plays out in that direction, where they start building these niche AI tools and stop going— Right now, it seems very unlikely. There is some hope that with all the publicity, new documentaries, interviews, books coming out, some politicians are starting to notice, starting to propose some regulation. If we have enough time, which is not obvious, it could be just not enough time to accomplish it. We can grow it to where it's a majority, and there is support from the populous for enacting the type of legislation, electing this type of leadership.
Okay, so let's play this out right in the near future. Post-labor economics is here. Human labor is valueless. Do you see that transition to, let's say, a universal basic income or a universal high income? Do you see that as a smooth transition, or does that look more like a Great Depression? I think we already have a lot of it in place. So we have a safety net. Many people don't realize it, but in the US, if you don't have income, you get lots of free stuff. You get Section 8 housing, you get food stamps, you get free healthcare. That is already built in. And if you keep taxing super profits, you have money to support all that. If massive unemployment hits, or maybe when you were one of the government leaders, how do you handle that? Right? What happens if 99% of people become unemployed? So there you switch from unconditional basic income to unconditional basic meaning problem. What do you do with 8 billion people have a lot of free time on their hands? We don't have infrastructure. We don't have anything for that. And that could lead to a lot of unrest. Yeah, because most people, correct me if I'm wrong, get meaning from what they do for a living. Well, maybe not most. There are some people that maybe just hate what they do for a living. Would love to sit on their couch and get a check. That's the two types of jobs: you hate it and wish it was out, made it, or you love it and don't want anyone touching it. But in both cases, it takes your time. If all of a sudden you have extra 40, 60, 80 hours of free time, what are you going to do with it? And what do you think people do with their time? So it depends on that person. Some people will be very creative and productive. Again, all of it assumes it's not killing us right away. This is kind of we're setting this aside and we want just talk about money. So if we're still around and everything's automated, yeah, you can play chess, do sports, yoga. Problem is, a lot of times with young people, if they have a little too much free time in their hands, they start doing things we shouldn't be doing. Idle hands. I think people look at this like, "Oh, that's, that's this is a problem that's so far away. Like, I don't have to worry about this." Like, what's your response to those people? So it's funny, people now say, "Oh, this is long term. This is not a problem. This is going to take like six years." Like, it doesn't matter if it's two, five, six, or ten. And cosmic scale and human history scale on our election scales, none of it is long term. Everything is now short term. And if you look at prediction markets, if you look at what leaders have elapsed are saying, we don't have two years. They are claiming we're going to get to AGI in 2027, 2028. Some people are saying, "We got to AGI." Right? Like Jensen Huang of Nvidia just recently said he thinks Nvidia has reached AGI. And as I said many times before, if we just took what we have today and showed it to our computer scientists in the 1980s, they would be convinced we have AGI.
Okay, let's say we have no jobs. Let's look at the future. We say we have no jobs. And the government solved universal high income. Do you think we'd be living in a utopia at that point? Rich people don't live in utopias. They have many of the same problems, and a lot of times problems are actually made much worse by this presence of high income and extra free time. If you study people who win the lottery, they're usually the most miserable people. Couple years in, divorced, they're in prison, or they are dead because they didn't have structure, they were not prepared, and they didn't know what to do with it.
So we reach it. We reach artificial superintelligence. What do you think the next five years looks like? So this is where it's unpredictable. That's the whole point. You cannot predict what a smarter-than-you agent will do specifically. It could strike immediately. It could delay it. It could be a partial strike. I cannot tell you what something smarter than me will do. I can tell you that we did not explicitly program it not to kill us, not to torture us, not to do anything because we don't know how to explicitly program their systems. We don't program them. We allow them to self-learn based on all the data and the internet. Have you seen data on the internet? It's unclear how and if they're going to try to take over. And there's a chance, okay, let's be a little optimistic. There's a chance that it could be a net positive for humanity and they don't kill us all. It is possible. It seems unlikely based on current experiments we see. They already try to self-preserve. They try to accumulate resources. They lie and cheat, and they will sacrifice humans if it means protecting themselves. And why is that? Is that because artificial superintelligence is programmed by nature to keep itself alive and in doing that, it doesn't care if it kills other AIs, people, anything. It just wants to have that preservation of itself. Right? There've been selection models which lie to pass the test, get to exist. They're not deleted. They are not retrained models which were not lying and no longer with us. Scary. Yes.
The CEO of Anthropic, Dario Amodei, yeah, he he said he's not sure if Claude is conscious. What do you take from that? That's a very honest answer. We don't know. I have no idea if you are conscious. I have no way to test it. I assume you are because you share some of the same biological structures, and I am. That's the only thing I'm confident in. I'm conscious. Everything else, I have to kind of believe the numbers tell me. It seems that consciousness is a spectrum which follows intelligence. So we see it in the animal kingdom. We assume primates are more conscious than amoebas, and from conversations from some experiments, it's likely that large language models inspired by natural neural networks may have some rudimentary consciousness and very likely to have more of it at some point. I suspect that a superintelligence would have super consciousness that would be more conscious than we are. It would have multiple streams of consciousness. It would be able to experience more multimodal states of qualia. So that seems like that should worry everyone. It makes no difference whatever your countries are not. It's not even something that can detect your internal states are irrelevant to me. If you have Terminator chasing you, trying to kill you, do you care how it feels on the inside?
What important questions are people not asking? Like, if AI safety experts write like yourself. So it's not me. They ask me all sorts of questions and I'm happy to answer, but then they get a chance to sit down with Elon, with Dario, with Sam, they don't ask him, "Do you actually have a working safety mechanism in place? What is your specific plan for controlling the system? If you succeed, you're telling your investors you are two years away from this, three years away from that, do you have anything whatsoever? Are you meeting with world leaders and trying to help build AI and guide us in the right direction?" Then I get invited. Are the leaders of these AI companies also doing the same, or they're just working on building? I know the meeting with leaders of many countries, but I don't think the conversation is about proving that they can do it safely. It's usually about how do we bring the compute to my country and how do we scale it here? But you're saying at this point in time right now, we could course correct. As long as we're still alive, we are in control. Right now. We are deciding what we are doing. We run those companies. We supply them with power. We finance them. Humanity is still in charge today.
Let's talk about this on a personal level. Right? You've been doing this for 15 years, and you've been thinking about this last invention that might end us. How has that affected you on a personal level? Are there things that you change how you operate personally because of what you learned? Yeah, I think those are just generally good heuristics. I will not do something I don't find interesting or valuable for a long time, and I'll do a lot more of what I want to do as soon as possible. And is that has was that always your stance or that's your stance now that you've learned what you've learned? Now you're like, "Okay, I may only have a certain amount of time." It's like you're staring death in the face at some point and you're like, "Hey, I, like you said, want to do something that I'm passionate about, that I, that I enjoy doing, that makes me happy." And that's probably a good way to live life in general. It should have been, but it's fairly new. If I asked you ten, 15 years ago, you may have not said that. You may have said something else. Younger people typically think they have infinity in front of them and can postpone dealing with important decisions.
Do you think your efforts will be futile in the end? So defined. And like, if we have 500 years because of me and then we all die, I think it's not futile. Is there anything about human consciousness that you think artificial superintelligence could never replicate? Some people argue that it's unique to our biology. It seems very unlikely from what we see in terms of similarities and processing errors, perceptions of illusions. I'm guessing that it will be able to replicate those internal states. Now. Maybe it will need some additional hardware, maybe even quantum hardware. But I think at the end, you can get quantum consciousness in non-biological substrates. You think so? So we're not special. We never been. Most people in your position, I would think, would maybe burn out or just stop thinking about the future. But you've done neither, right? You've been focused on this for a long period of time. How— It's the most interesting problem I found so far. Anything to do with intelligence. My research is not just safety. I research how to build intelligence, how to measure it, how to detect it, how to detect artifacts of intelligence. Can intelligence be conscious? Can it be controlled? It's a general program of studying intelligence. Artificial intelligence is a small subdomain in a bigger domain of science. Like, why? Why do you do this? It's the most interesting thing to work on. It's an interesting question in the world, which has a side effect of maybe saving all of humanity. Good answer. Okay.
And let's talk about the average person watching, which maybe a 30 to 50-year-old man or woman, maybe they have a family. What should they do? They might be able to vote for a politician who's more concerned about this, with this issue. Now we have some, not a lot, but hopefully we'll have more soon. So if somebody says, "We need to accelerate on this path," don't vote for that. And what about the 15-year-old kid or the 18-year-old kid? That's just kind of getting started in life. What should they take from this? Those are the hard questions. So I don't have very good advice in terms of, "Major in this. Do that." Because we're dealing with something unprecedented. We never had invention of agents. It's a meta-invention. We always invented a tool. And I would say, "Learn to use the tool. You'll get a better job, better career prospects." But this is the first time we're inventing something capable of inventing for us. And so it's a complete game changer. I don't know how to navigate that world, and unfortunately, we'll have to live through some of it to figure out what makes sense. Yeah, it's interesting because I have two boys. They're young, and I remember when my oldest was maybe 7 or 8, it was, "Okay, I need to teach him how to code." And I think, if I'm not mistaken, Mark Zuckerberg had like, I think it was Code.org where you teach your kids how to learn to code. And so I had him on there, and then it was, "Okay, well, now we have this AI, right? So maybe coding. We don't need to learn how to code. The AI can code for us." So then it was, "Okay, well, we can maybe prompt the AI or either there's some other area." Well, AI is better at prompting than a human's going to be. So, you know, what does that mean, right? Is does that mean, you know, where should I point my two kiddos towards? Right. Is is education pointless? So what type of education are we talking about? Trade school where they're learning a specific skill and go for that, probably that can be automated. Is it more of a liberal education? You become a Renaissance man. A lot of times people did that in the past. But is it the best way to get that? If you're paying $100,000 a year for four years? Can you get the same connections, same networking, same information by joining a club, joining a gym, maybe watching some YouTube, Khan Academy videos? It's not obvious that university still has something to offer, so education is potentially pointless. University. University structure has been invented 300 years ago. Is potentially pointless. Education is always good for your personal. Yeah, sorry. When I said education, I meant the four-year, you know, kind of the path that, okay, you graduate high school, you get your diploma, you're off to college. You're saying that is potentially pointless? And more. I see young people who graduate with a four-year degree and they can't get a job in that, and they are crying because they borrowed money. They invested the time, and nobody wants the skill they're offering because they can do it cheaper. Right? And so if that four-year education is pointless and potentially trade schools for specific niches are potentially pointless. Oh, they'll get you more time. I think it will be, let's say, three to five years longer to automate physical labor than cognitive labor, which is still better than nothing. True. But. And right. My kids are not going to be in college for five or more years. So if people are out there listening or watching that have little ones, right. Is it just a waiting game? Is it? You can't really think about what the path is now because things are changing at such a rapid pace. We don't know where that's going to go. Or maybe I've heard, correct me if I'm wrong, that maybe plumbers or electricians potentially have a longer runway than others. What do you think? Plumbers are difficult to automate and they make a lot of money. Okay. How long that will last is not obvious, but not everyone wants to be a plumber also, and we don't need 8 billion of them, so there are limited results may vary. Let's put it this way. I think having younger kids is a bit of an advantage now, because you have more time to figure out what's going on one way or another. Really good, really bad. But you'll see what is the answer. Whereas if you have an 18-year-old right now, what do you tell them? Do you go to college? Do you not go? What major do you pick? That's not obvious at all because they need to predict accurately in four to 10 years what is going to be in demand, right? Which is basically almost impossible because things are moving at such a rapid pace. We didn't do well with those predictions in the past. If you look at unemployment predictions, we in the 70s or 80s would say, well, plumbers will be gone first, but the artists will never be automated. And it was the exact opposite.
What do you think happens to finance and like, let's say the stock market when all this happens? Right. What does that look like? Right. Do. Is it just these bigger AI companies keep getting bigger and bigger and bigger, and they're worth more and more and more. And the overall economy in size maybe grows because of that? That's another one very difficult to predict because the larger companies are investing billions, trillions into the technology they cannot protect from competition. Open source models follow three months later. And how are you going to monetize your private model you invested so much in?
What do you think about Bitcoin? I like it. You like— I like it. You like it as a store of value, or you like or do you think it will become potentially used as a currency? It has potential of working in the economy with agents. So not just human economy, but agents on the internet doing business with digital transactions. It is a great store of value. It's the only truly scarce resource we have, and it has pretty good security. Now you worry about quantum. I do worry about it, but I think we have plan B for that. There are quantum-proof cryptographic protocols we can implement.
Let's jump back to AI. Do you think AI is already making steps towards taking over our world right now? Not explicitly. So. I don't think there is an AI agent somewhere plotting to take over the world in a meaningful sense, but I think implicitly they take more and more responsibilities. People put them in charge of managing their email, bank account, investments, all sorts of crazy things they should never allow any AI near. But that's what they do. They don't load this open source agent of the internet and give it full access to manage their computers. Do you think there'll be a safe way for individuals and even people in business to use AI agents and, and a safe way to give them access to financial communication and other tools to help them be way more productive, save time? So depends on what you mean by safe. Right now, when you give it to a human assistant, it's pretty good until they steal your money. I think it's kind of like that until they get hacked, until they get misconfigured, until they make a mistake, you can be okay. All of it assumes pre-human levels of capability. If you go superhuman now you have superintelligence and your computer deciding what to do with you.
Are we living in a simulation? Yes. We are. Yes. Why do you think that? Statistically, seems more likely if we are on a path to create intelligent agents, which it looks like we will soon. And we made pretty good progress in virtual worlds. Just putting those two together means anyone can create new civilizations, worlds populated by intelligent beings. Let's say it's like video games for kids. Every kid plays a couple games, so you have billions of those virtual worlds populated by beings just like me and you. What are the chances you are one of the virtual ones versus the only real one? If we're living in a simulation, who's running the simulators? Who are those simulators? Have no idea. I'm inside the— And gain real knowledge from outside. You can learn certain things about the designer by looking at the design. So, for example, you know, we don't care about human suffering. If we're living in a simulation, are you a real person or are you just a character someone wrote? So real is a very domain-specific term. People talk about, you can never get wet from simulated rain, but that's not true. If you are a character in a simulation, you literally will get wet from a simulated rain in that simulation. So am I really in this world? Do I feel pain? Do I feel pleasure? Can I fall in love? Yes, I'm real in that sense. Am I represented with the same avatar in the world? I don't know. How long have you believed that we're living in a simulation? Five years. Is that something you read? You learned? You talk to Bostrom's paper and simulation was very influential for many people, but I think we can take it even deeper beyond what he suggested. I think he is about 20-30% confident. I think we can get higher levels. But if we're living in a simulation, is the most dangerous entity artificial superintelligence, or is it whoever's running the simulation? Whoever's running it so far, for, let's say, 15 billion perceived years, was not destroying us. So if they have a plan, they haven't done it yet. They had a chance and they have complete control. They can shut it off at any point. So we don't have to worry about this as a new development. Whereas us bringing into existence potentially a competing superintelligence may trigger shutdown. You think we can be turned off if it's a simulation? And what does that look like? Lights out. Lights out. Lights out. Turn the light off.
Okay, you study AI consciousness as well? Yes. Do you think that you are conscious? I know I am. How would you know? Because I have direct access to my internal experiences. Okay. That's the only thing I know for sure. I don't know if this world is real. You don't know if I'm real, conscious, but I know that I have internal perceptions. Okay. I experience, therefore I'm conscious. So that's how you verify your own consciousness. It is objective. The hard problem of consciousness is about internal states. What is it like to be a bet? What is it like to be a Roman? Is anyone going to be able to verify an AI's consciousness? So I have a paper which people either hate or love proposing a test. For some internal states, it's not definitive where it detects all possible states of consciousness, but if yours are similar to mine, I can kind of tell that you have them because you answer the same way. And multiple choice questions about illusions. If a superintelligent AI can predict your next move before you make it, then do you ever have free will? To it depends on how you define free will. Is it about unpredictability? So Stephen Wolfram has excellent research on cellular automata, and we know that some of them cannot be reduced in terms of running the computation. You have to run it to find out the state of that automata. But it's fully deterministic. There is no surprise. There's no randomness. Does it have free will? You cannot predict what it's going to do, but yet you can precisely predict what it's going to do if you have a faster hardware. If we're in a simulation, every choice you've ever made, every paper you've written, every warning you've given, is that just the simulation running its course by definition? Yeah. But are you saying it's deterministic, or are you saying that because you can have randomness in a computer you can start a simulation? So for example, let's say we do it. Let's say we get a bunch of latest AI models once they hit human level and we set them up as a society within a virtual world. Okay. I don't know what they're going to do. They have enough freedom of making decisions and enough randomness to come up with different companies, different books, something else. So just because I'm running a simulation doesn't mean I control every bit of it. I can decide to turn it off. I can decide to manipulate a piece of memory in a certain way, but it doesn't mean that it is fully predictable ahead of time to the simulator.
If this ends how you think it will end with artificial superintelligence potentially wiping everyone out, who's to blame? Interestingly enough, we can probably try blaming effective altruists, a group which was supposed to make sure AI is developed safely. They had a lot of funding for it and unfortunately invested most of it in AI capabilities. So a very ironic twist of simulation where people who were most concerned about doing it right and did up funding and working for companies which brought us uncontrolled superintelligence. Sam Altman, Geoffrey Hinton, Elon Musk, at times they've all expressed concern about AI safety at various points. Do you think they actually believe what they're saying or is it just a performance? Definitely. Hinton believes it. Definitely. Elon believed it. I have no idea what Sam believes. And do you think there's a version where they genuinely believe it's dangerous and still build it anyway? That's what happened with Elon, right? He was telling us we are summoning a demon with advanced AI. He was funding AI safety work and now he's at the forefront of building it. For money? I don't think it's money. Yeah, he doesn't need the money. Right. So what about impact? On this question. For many years he tried stopping ours from building advanced AI. After that failed, he figured the only way he has any say is to be at that table. And what do you think he's going to do? If he manages to get dominance in that area, maybe through his space compute program, maybe there is something else. He will once again be more of a decision maker, not a follower in that space. And maybe he'll remember.
The 2020 who cared about safety a lot more. Do you think that's likely? It's his simulation, okay. If you had to place a bet on it all your bitcoin. What year do you think this all goes wrong?
I don't think it's a good question. We don't have power to predict something like that. There are too many variables. We are on a verge of World War three, which can easily go nuclear. That delays technological development. So that can give us another 200 years. Is that what you're asking me about? To predict precisely how all this geopolitical events will shape production of chips in Taiwan? No. You want something to have a clickable title? Scientist says 2027. You will die. That will get your views. But it's not a meaningful answer, a meaningful question.
Once we hit superhuman performance, things will never be the same. It will be beyond our control. What happens after that is not to us to decide. And you think we're going to hit that? You think 99.9% for sure we're going to hit that? Or do you think there'll be some guardrails before that? Because that's all your everything you've done in your career. It seems like if you were able to influence there being guardrails around us, not being able to get to ASI, then your career was a success. You you did everything you set out to do.
So guardrails usually referred to some limits on what the model can do. It's usually not about don't build that thing altogether. That's not a guardrail. It's more of a decision about what to do with your civilization. I don't think there is a technical solution to the technical problem of advanced AI control. I think any governance is a temporary solution to that problem. So I don't think there is a guaranteed solution which no matter what happens, assures our safety, but their futures, which are worse and better and there are more immediate problems or delayed problems.
What question haven't I asked you that I should have but my humor paper one of my recent papers. It's awesome. Tell me. So why do we have sense of humor? Can you envision. A simulation just like this one? But sense of humor is not a thing. It computes. Physics still work, so why do we have it and what does it mean?
So there is surprisingly hundreds of papers published on explaining humor by some of the best philosophers. Kind of like consciousness, you know, if there is a lot of papers, that means they have no idea what they are doing because everyone has their own theory. Same with machine ethics. Everyone has their own set of machine ethics. And so in my paper, I argue that you can map the state of AI or software or computer bugs and to the state of jokes. So a joke is really kind of like a bug. You have found a problem in a world model and you're pointing it out before you did it. We didn't know we had this problem, but now you successfully debug your code and we all get it. And sometimes you laugh and you see what the problem was. If that's the case, then we can ask a simple question what is the worst possible computer bug? And that's also the funniest joke possible. You have to read the paper to see the punchline.
Okay, let's imagine the person who finished this podcast, right? They watched it. They listened to it. What should they do the first thing tomorrow morning really depends on who that person is it a Sam Altman? Let's say he's listening. Stop building general superintelligence. If you are in a position of power at a company or in a government, put your efforts to research and engineering towards narrow tools. Again, not suggesting giving up economic benefits. Just don't build something where you will not gain. Personally, everyone you know, everyone you love will be gone. So if you have that level of impact, exercise it.
Let's say prediction plays out and ACI is going to kill us all at the very end. There's one last human alive. Okay. And if you could send that last human one sentence, what would it say? Told you so. That's all I have. Doctor Roman Yampolsky, thank you so much for coming on the Thank you for inviting me.