Transcription
FROUMAN: Good evening, welcome. My name is Mike Frouman, I am the President of the Council. I am pleased to welcome you this evening to one of the events in our executive speaker series. Today, we have with us Dario Amodei, CEO and co-founder of Anthropic. Dario was Vice President of Research at OpenAI, where he was involved in the development of GPT-2 and 3. And before OpenAI, he worked as a Senior Research Scientist at Google Brain. Dario and I will chat for about thirty minutes, then we will move on to questions from the audience. There are approximately 150 people in the audience, and about 350 people are watching us online, and we will try to answer their questions as well. Welcome. AMODEY: Thank you for inviting me.
FROUMAN: So, you left OpenAI to found Anthropic – a corporation with a mission to work for the benefit of society. Why did you leave? What are Anthropic's values and how do they manifest in your work? And let me say right away: a cynic might observe that this story about a special mission is just a marketing ploy. Can you provide specific examples? How do your product and strategy reflect your mission?
AMODEY: Yes, perhaps it's worth rewinding a bit and explaining the context. We left at the end of 2020. In 2019-2020, something happened that we – I and a group of people at OpenAI who later became my co-founders at Anthropic – were among the first to realize. This concerns the so-called scaling laws and hypotheses. The hypothesis itself is quite simple. It states – and it is truly amazing, and it is difficult to overestimate how incredible it seemed then – that if you use more computational resources and data to train AI systems with relatively simple algorithms, they become better at performing almost any cognitive task. We observed these trends even when training models cost from one to ten thousand dollars, i.e., at the level of a regular academic grant. We predicted that these trends would continue, even when the cost of training models increased to 100 million, a billion, ten billion dollars, which we are now approaching. And indeed, if the quality of models and their level of intelligence continue to grow, it will have a huge impact on the economy. It was then that we first realized that we would see serious consequences for national security as well. In general, it seemed to us that OpenAI's leadership shared this scaling hypothesis, although many inside and outside the company did not accept it. Our second discovery was that if this technology is truly so important, we need to approach its creation extremely carefully. We need to do everything right. On the one hand, these models are extremely unpredictable; they are essentially statistical systems. I often say that we grow them rather than create them. They develop like a child's brain. Therefore, it is very difficult to control them and make them reliable. The process of their training is not simple at all. From a system security perspective, it is crucial to make them predictable and safe. And, of course, there is the issue of how these systems are used by people, states, the impact they have when companies start to apply them. Therefore, we felt that we had to create this technology in a certain way. You yourself hinted at it; OpenAI was initially created with this in mind. But for a number of reasons, which I don't want to go into now, we didn't feel that the leadership took these issues seriously. Therefore, we decided to go our own way. The last four years have been a kind of experiment in which two models have developed in parallel: what if one way, and what if the other. And we have seen what this has led to. I will give a few examples of how, in my opinion, we have shown our commitment to these ideas. One example: we invested very early in a scientific field known as mechanistic interpretability – this is an attempt to look inside AI models and understand why they behave one way and not another. One of our seven co-founders, Chris Olah, is the founder of this field. Initially, it had no commercial value, or almost none – at least for the first four years of our work. Nevertheless, a whole team worked on this field all this time, despite fierce commercial competition. We believe that understanding the internal workings of models is a public good, beneficial to everyone. We openly published all our work so that others could benefit from it. Another example: we proposed the concept of constitutional AI, where AI systems are trained to follow a set of specific principles, rather than being trained on a mass of data or human feedback. This allows, for example, to go before Congress and say: "Here are the principles by which we trained our model." When we released our first product – the first version of the Claude model – we delayed its release by about six months, as it was such a new technology that we were not sure of its safety. We were not sure we wanted to be the first to jump into the race. This was just before the release of ChatGPT. That is, we had the opportunity to catch the ChatGPT wave, but we decided to release our model a little later. This had certain commercial consequences, but it laid the foundation for our company culture. The last example: we were the first to introduce what is called a "responsible scaling policy." The essence of this policy is that we measure the risk categories of models as they scale. And upon reaching certain points, we are obliged to take increasingly strict measures to ensure safety when deploying these models. We were the first to publish this policy and the first to commit to adhering to it. Within a few months, other companies followed our example. Thus, we managed to set a positive example for the entire ecosystem. When I look at the actions of other companies, it is often we who set the tone and force them to follow us. Not always, sometimes they do something different, and then we follow them. But overall, in my opinion, we have proven that we are keeping our commitments. I would contrast this with the behavior we have observed in some other companies. We already have several years of experience, and so far, knock on wood, our commitments have stood the test of time.
FROUMAN: I want to talk about both the risks and opportunities associated with AI that you mentioned. But since you touched on the topic of responsible scaling, let's return to it for a moment. We are currently at level two… AMODEY: Yes, so… FROUMAN: At what level is there existential risk? How will we know when we have moved to level three? And if we reach level three, can we go back, or will the situation only worsen?
AMODEY: Our responsible scaling policy is structured as follows: we built it by analogy with biological safety levels. There, the levels indicate how dangerous certain pathogens are. We decided to create similar levels for AI. So, AI safety level 2 is the current level, where we are now. These are powerful systems, but the risks they carry are comparable to the risks from ordinary technologies. Level 3 is the level that our models are already beginning to approach. The last model we released is not yet at level three, but it is already close. Level 3 is characterized – and here we are particularly focused on national security issues – by serious risks, far exceeding the risks typical of ordinary technologies. For example, a level three model in the area of chemical, biological, or radiological weapons could allow an untrained person, simply by interacting with it and following its instructions, to do something that today requires, say, a doctorate in virology. If such risks are not prevented, the number of people capable of performing such extremely destructive actions will increase, for example, from tens of thousands to tens of millions, as soon as the models become available. Therefore, when models reach this level, we must introduce special measures so that they do not provide such information, as well as safety measures so that these models cannot be stolen. I think we are approaching this level, and may reach it already this year. And we are confident that we know how to safely release such models, depriving them of the ability to perform a narrow set of dangerous tasks without undermining their commercial value.
FROUMAN: But we are talking about a rather narrow range of tasks. Will you simply prohibit the models from answering such questions?
AMODEY: Yes, we will not allow the models to engage in such tasks. Although it's not that simple. Imagine, someone might say: "I'm taking a virology course at Stanford and I'm working on homework. Can you tell me how to create this plasmid?" The model needs to be smart enough not to fall for it and say: "You know, actually, such a question…"
FROUMAN: "You sound like a bioterrorist, I won't answer your question."
AMODEY: "It seems you have bad intentions."
FROUMAN: Yes. But here we are limited by our own imagination about the actions of malicious actors. There are many scenarios that we cannot foresee, which go beyond these four categories.
AMODEY: Yes, yes. Well, I think here's the problem: every time we release a new model, it has positive applications that we haven't even thought of, but it also has negative applications that we also didn't foresee. Therefore, we always monitor how people use the models to notice such things in advance and always be on alert. If we fear that someone might do something bad with model number six, then we hope to see the first alarming signs already in model number five. We are observing. But this is the fundamental problem with these models: you don't know what they are capable of until the model is available to millions. Yes, you can test in advance, you can have researchers work on them, even the government, with whom we cooperate, tests them for safety. But the bitter truth is that there can be no absolute certainty. Models are not code for which formal verification is possible. Their capabilities are unpredictable. It's like if we were talking about me or you: imagine I'm the engineer responsible for quality control of me or you. Can I guarantee that you are logically incapable of doing something bad, that this will never happen? People are not built that way.
FROUMAN: Let's talk about the opportunities, the positive scenarios, now.
AMODEY: Of course.
FROUMAN: At the end of last year, you wrote an essay "Loving Grace Machines," where you spoke about positive prospects. For example, that in just one year, ten years of progress can be achieved in biology, and that machines will be as smart as all Nobel laureates, which probably upsets some of them. Tell us about the positive scenario. What will be the best-case scenario of what AI will bring us?
AMODEY: Yes, I'll start with the exponential. If we go back to 2019, models could barely form a coherent sentence or paragraph. For people like me, even this seemed like an amazing achievement that was impossible before. And we had forecasts that in five years, models would start generating billions of dollars in revenue, would help us write code, we would be able to talk to them as if they were people, and they would know as much as a person. And then everyone brought up a lot of unsubstantiated arguments why this supposedly couldn't happen. These same exponential trends and the same arguments that predicted it then, now say that if we look two, three, maybe four years ahead, we will come to exactly this. We will come to models intellectually comparable to Nobel laureates in many fields. You won't just be texting with them; they will be able to do absolutely everything a person does on a computer. Almost any remote work that people do, any type of task that takes days, weeks, months to complete. In the essay "Loving Grace Machines," I put it figuratively: it's like having a whole country of geniuses living inside a data center. A country of brilliant remote workers. Of course, they won't be able to do everything; there are physical limitations in the world. Many people still find this crazy, but remember past exponential trends. Look at the early internet: those forecasts also seemed crazy, and what came true from them? I'm not 100% sure, maybe 70-80%. It's quite possible that the technology will stop at the current level or in a few months, and then for ten years people will laugh at me, remembering my essays and speeches at such events. But personally, I wouldn't bet on that.
FROUMAN: Let's dwell a bit more on the topic of jobs. There is a lot of discussion now about the impact of AI on employment. Where do you stand on the scale… Although before you answer, tell me: how soon will AI be able to replace a think tank leader? Asking for a friend. Okay, we'll get back to that. Seriously, on which pole of the spectrum are you – from "everyone will do cool things and be able to do much more than before" to "everyone will sit on the couch and receive basic income"?
AMODEY: I think that in reality, we will face a rather complex mix of these two scenarios, and much will depend on the policies we follow.
FROUMAN: You can answer about the think tank leader if you want…
AMODEY: Yes, I probably haven't fully covered all the good things that can happen thanks to AI. Honestly, my greatest optimism is about progress in biological sciences – in biology, medicine, neuroscience. If you look at the last hundred years of biology development, you can see that we have only managed to deal with simple diseases. Solving the problem of viral or bacterial diseases is relatively easy – it's like dealing with an external invader. But dealing with systemic diseases like cancer, Alzheimer's, schizophrenia, or severe depression is much harder. If with the help of AI we manage to overcome precisely these, then regardless of the employment situation, the world will become much better. Especially if we solve mental health problems, people will have more opportunities to find meaning in life. So, in this regard, I am very optimistic. But if we move on to the issue of jobs, then I have serious concerns. On the one hand, I believe in the theory of comparative advantage. If we take, for example, programming, and this is an area where AI is currently advancing particularly rapidly. I think we are already close to the point where in three to six months, AI will write about 90% of the code. And in a year, perhaps almost all code will be written by AI. However, a programmer will still need to set the task conditions: what we want to create, what program, make key design decisions, think about how to integrate new code with existing code, evaluate it from a security perspective – is it a reliable design or not? As long as these separate islands remain that require human involvement, which AI cannot yet cover, people's productivity will indeed increase. But, on the other hand, I think that gradually AI will also capture these small islands, until eventually it can do absolutely everything that humans can do. And this will happen in every industry. Moreover, I think it's better if it happens to everyone at once, rather than selectively. The worst and most socially destructive scenario is if half of the jobs disappear randomly. Because then society will seemingly randomly choose half of the people and tell them: "You are useless, you have been devalued, you are no longer needed by anyone."
FROUMAN: Instead, we will say: "Now you are all useless"?
AMODEY: Well, we will all have to enter into a dialogue. We will have to take a sober look at technological capabilities and reconsider our understanding of what it means to be useful or useless. Our current approach is not tenable. I don't know the right solution yet, but it's definitely not about our uselessness. "We are all useless" is pure nihilism; with such an approach, we will achieve nothing. We need to find another answer.
FROUMAN: The picture doesn't seem very optimistic. Or does it?
AMODEY: Actually, I would argue. You know, I think about what I do myself, for example, I swim a lot, I play video games. Or take chess champions, people: one might think that after Deep Blue's victory over Kasparov, which was almost 30 years ago, chess would become a meaningless pursuit. But the exact opposite happened. Today, chess champions like Magnus Carlsen have become celebrities. I think he even became a model, a real hero. So I think we can create a world where human life is filled with meaning, where people, perhaps with the help of AI, working in tandem with it, can create great things. I am not pessimistic. But if we act incorrectly, we will not have the right to make mistakes.
FROUMAN: A few months ago, the DeepSeek model was released. Here in Washington, it was met with panic, even talk of a "Sputnik moment." Was it really a "Sputnik moment"? And what does it tell us about the scaling laws you mentioned: that you always need more computational resources, more data, better algorithms – do these rules still work, or are there possible workarounds?
AMODEY: Yes. I think the DeepSeek model does not refute the scaling laws, but rather confirms them. Two processes are at play here, and I even recently wrote a post about it. First, the cost of achieving a certain level of model intelligence is decreasing by about four times annually. This is because we are getting better and better in terms of algorithms that allow us to achieve the same result at a lower cost. In other words, the curve shifts: in a year, you can get the same model four times cheaper, or a model four times more powerful for the same money. But economically, this means the following: if the economic value of models of a certain level of intelligence remains, the fact that you can now create them four times cheaper leads to the fact that you will create more models. Moreover, there is an additional incentive to invest even more money in smarter models with even higher economic returns. And although the cost of creating models is decreasing, the amount of money that companies are willing to spend on them, on the contrary, is growing, and rapidly, by about ten times a year. It turns out that society and the economy want more and more intelligence and smarter models. It is against this background that the DeepSeek model appeared. It was just another point on the cost reduction curve. Nothing unusual happened. It's not like American companies spend billions, and DeepSeek suddenly did the same for a few million. The costs were quite comparable. Yes, the model itself cost them a few million, but if we consider the total expenses of American companies on the development of similar models, the figures are quite comparable. Like us, they spent billions on research and development of the model. If you look at the number of chips they have, the figures are also roughly equal. However, this is still alarming, because until recently, only three, four, maybe five companies could create models that are at the forefront. And all of them were American. What is truly significant in the DeepSeek story is that it is the first model created by a Chinese company that competes on equal terms with companies like Anthropic, OpenAI, or Google, demonstrating the same engineering innovations. This is a truly significant moment. And this causes me serious concern.
FROUMAN: Some argue that the emergence of DeepSeek means that export controls are not working and cannot work, and that we should stop trying to control the export of our advanced chips. Others, on the contrary, believe that we should double our efforts on export controls. What is your position?
AMODEY: I think that from the scheme I described, it clearly follows that export controls are indeed necessary. Yes, the cost of models is gradually decreasing, but at every stage, regardless of how much this curve shifts, one thing remains constant – the more chips you have and the more money you spend, the better the model you get. If before for a billion dollars you could create a medium-level model, and now for the same billion dollars you get a much more advanced model, and a medium-level model can be obtained for 10 million, it does not mean that export control has failed. On the contrary, it means that preventing your adversaries from obtaining a billion-dollar model has become even more important, because for the same billion dollars you can now get a much more powerful model. Yes, DeepSeek had a relatively small computing infrastructure, consisting of chips that bypassed export controls, smuggled chips. But we are moving towards a situation where we, OpenAI, and Google will create infrastructure with millions or tens of millions of chips, costing tens of billions of dollars or more. Such a quantity can no longer be smuggled. If we maintain and strengthen export controls, we can quite possibly prevent China from accessing such infrastructure. If not, they can quickly reach parity with us. That is why I have actively supported rules to prevent technology diffusion and have advocated for export controls for several years, even before the appearance of DeepSeek, as we foresaw it. I believe that export control is one of the most important decisions for US national security, not only in the field of AI, but also in many other areas, to prevent China from obtaining millions of powerful chips.
FROUMAN: As I understand it, the diffusion rules are a Biden administration executive order that divided the countries of the world into three categories based on their access to our chips. Some worry that countries that did not fall into the highest category will eventually start using Chinese technologies, and China will become the main supplier of AI infrastructure for most of the world. Have you thought about this?
AMODEY: Yes. As far as I understand, the new administration is currently reviewing the diffusion rules, although in general they view them positively. The rules are structured such that Tier 1 countries are most developed countries.
FROUMAN: But not all.
AMODEY: Yes, not all. Tier 3 countries are countries with limited access, such as China or Russia. Tier 2 countries are countries in the middle. In such countries, it is actually allowed to use a significant number of chips, if the companies that will use them provide guarantees and proof that they do not represent China and that the chips themselves and the results of calculations will not be sent to China. So, in fact, there is still an opportunity to place many American chips and American infrastructure in these countries, if they comply with the specified security conditions. Regarding the second question, in theory, these countries can, of course, switch to Chinese chips. But Chinese chips are actually significantly worse. Nvidia is significantly ahead of Huawei, the main Chinese chip manufacturer. By about four years. I think this gap will gradually narrow, perhaps in ten to twenty years. It is quite possible that export controls will even stimulate China to develop its own technologies faster. But the technological chain is so complex that the next ten years, while we maintain an advantage in hardware, will be a critical period for consolidating dominance in this technology. I believe that whoever manages to ensure dominance in this field will gain economic and military superiority worldwide.
FROUMAN: The previous administration began a dialogue with China on AI issues. What are the prospects for such a dialogue? What can we potentially agree on with China? Are they even interested in the idea of responsible scaling?
AMODEY: I would say this: although I have not directly participated in these negotiations, I have heard something, and in general I support the idea of dialogue, although I am not particularly optimistic about its results. This technology carries such serious economic and military potential that, for example, between companies in the US and among our allies, it is possible to imagine adopting laws limiting development. But when it comes to a race between two sides, each striving to create a technology that has enormous economic and military value, perhaps more than everything else combined, it is difficult to imagine them voluntarily slowing down. However, there are several points. One of them is the risk that AI models will begin to act autonomously against human interests. If you have a whole "country of geniuses" in your data center, it is logical to ask – what are their intentions? What are they planning? Of course, you will ask: who controls them? In whose interests do they act? And also ask what their own intentions are? And since we grow these systems rather than program them directly, we cannot simply assume that they will always do what their creators or users intended. I think there is a real risk here, and it could threaten all of humanity. Just as in matters of nuclear safety or non-proliferation, it is possible to adopt restrictions aimed at reducing this risk. I am moderately optimistic that an agreement can be reached on such an issue. It seems somewhat speculative now, but if there is strong evidence of the reality of such a risk, perhaps China will be more inclined to cooperate. I hope we can do something in this area, although it is unlikely to change the overall dynamic of competition between the two countries.
FROUMAN: Last question before we move on to questions from the audience. Recently, you presented a plan of action for managing scientific and technical policy, what should be done in the field of AI. Name the main elements of your plan.
AMODEY: The plan has three points related to safety and national security issues, and three points related to opportunities. The first element – we have already discussed it – is maintaining export controls. I sincerely believe that this is the most important decision that can be made for US national security, and not only in the field of AI. The second point is related to the responsible scaling policy. The US government, with the AI Safety Institute, tests models for national security risks – biological or nuclear. I think the institute is not named quite correctly: the word "safety" can evoke associations with trust and security, but here it is precisely about national security. It is not so important to us how this body is named or where it is located, but it is important to have a structure that measures these risks. This is also important for assessing the capabilities of our adversaries, for example, measuring the capabilities of DeepSeek models to understand what dangers they may pose, especially if they are used in the US. What are they capable of? Where can the danger lie? This is the second point. The third point, which we haven't discussed yet, is concern about industrial
espionage against American companies, such as Anthropic. China is widely known for large-scale industrial espionage. We are taking security measures, and constantly raising the bar, but it must be understood that many of our algorithmic secrets are literally a few lines of code, worth hundreds of millions of dollars. I am sure there are those who try to steal them, and perhaps some succeed. We need additional help from the US government in protecting against such a risk. These are three points on security. Now about capabilities. There are also three key points here. The first is the potential application of AI, especially in healthcare. We have a unique opportunity to conquer the most complex diseases that have been with humanity for hundreds and thousands of years, and with which we have not been able to do anything until now. I think this will happen anyway, but the right regulatory policy can shorten the time needed to develop and distribute these drugs using AI. The difference between five and thirty years is colossal for people suffering from such diseases. Therefore, today's healthcare regulatory policy, including the drug approval process by the administration, may not be entirely suitable for such high rates of progress, and some barriers should be removed. The second point is energy provision. If we want to stay ahead of China and other authoritarian competitors in this technology, we must build data centers. It is better to build them in the US or with our allies than in countries whose loyalty may shift in favor of China. Towards the end of the Biden administration, this was discussed, but it is a bipartisan issue, and the Trump administration agrees on this. We need a lot of energy. By 2027, an additional approximately 50 gigawatts of energy will be required for the full development of AI, which is about the same as was added to the entire US power grid in 2024. That is, by that year, as much additional energy will be needed as is planned to be deployed in the next two years. The task is enormous. And the last point is economics. I believe that the economic risks are as serious as the national security risks. In the short term, we need to cope with economic shocks, although the overall economic pie will become much larger. In the long term, I don't want to lie to you, we must think about a world where AI becomes better than almost all people in everything they do. We need to start realizing this now. The best thing we can do now is to measure what is happening. For example, we launched the so-called Anthropic Economic Index, which anonymously tracks how people use our models: is it a supplement to human labor or a replacement? But in the long term, this will lead to questions of tax policy and wealth redistribution. Imagine a hypothetical scenario: if AI increases the rate of economic growth to 10% per year, we may have enough resources to eliminate the budget deficit and cope with large-scale shocks in industries. I know this sounds crazy, but I suggest we start thinking about such "crazy" scenarios now. FROMAN: "Crazy scenarios." You heard it here first. Now let's move on to questions. Yes, please. Adam: Thank you, Dario. This has been a very fascinating conversation. Should I stand up? FROMAN: Yes, please. Adam: Okay, I'll stand up. I'll get my steps in. FROMAN: And please introduce yourself. Adam: My name is Adam Bankedeco. I read your essay with interest last year and listened to you on the Times' Hard Fork podcast, and of course, today. You described the economic and political consequences of AI in considerable detail. I am interested in how you view the social and moral aspects that will inevitably arise? After all, ordinary people now look at chatbots and perceive them simply as an improved Google search, without thinking about the profound consequences for the labor market and society as a whole. What do you personally think about this, considering that you lead a company and create a commercial product? AMODAY: Yes, first of all, I believe this is an incredibly important question. What worries me most right now is precisely that people do not realize the scale of what this technology entails. Of course, I might just be wrong and talking nonsense. Maybe it will turn out the opposite: the general public is right, and I am wrong. Perhaps I'm just addicted to my own product, I don't rule that out. But let's assume that I am not mistaken after all. What do I see now? There are several concentric circles of people who understand the scale of the changes. There are probably a few million people, mostly in Silicon Valley, but also a few in political circles. It is not yet clear whether we are right or not. But if we are right, then most people still see AI only as chatbots. If we start telling them: "This is dangerous, this can replace any human labor," they will consider us crazy, because it all seems frivolous to them. They don't yet understand what awaits them. This is what keeps me up at night, which is why I am trying to convey this message to as many people as possible. The first step is problem awareness. Regarding labor and employment issues, if it becomes technologically possible to reproduce the results of human consciousness, then this is a profound question. I have no definitive answer. I completely agree with you: it is a question of morality, a question of meaning and purpose of existence, perhaps even a spiritual question. We will all have to find the answer to it together. But I have a nascent answer. I think that tying a person's self-worth solely to the ability to create economic value is, in some ways, a psychological phenomenon, and in some ways, a cultural phenomenon. Of course, it helped create the modern economy. But technology, as often happens, can destroy this illusion. Perhaps the situation will be similar to when people realized that the Earth revolves around the Sun, not the other way around. Or when it turned out that there are other solar systems in the universe, or when it became clear that organic and inorganic matter are made of the same molecules. Perhaps a similar moment of realization and reevaluation awaits us. I am struck by how meaningful activities that do not create economic value can be. I am surprised by how much joy I derive from things in which I am not the best in the world at all. If what you do implies that you must become the world's leading expert to be spiritually significant, then I think you have gone down the wrong path. There is an error in this premise itself. And this is coming from me, a person who spends a lot of effort and time trying to become the best in the world at something he considers very important. But we will all have to find the meaning of existence elsewhere. FROMAN: Yes, please, the young lady at the back of the room. Carmen: Hello, my name is Carmen Dominguez. I am an AI specialist, I work in development, implementation, and recently, in the political aspects of technology. I absolutely agree with you that society is little aware of what AI is, what it can and cannot do. I am involved in communications in this field, but that's not what I'm here for today. A few months ago, you hired Kyle Fish as an AI Well-being researcher to study the possible sentience or lack thereof in future models and whether they might deserve moral consideration and protection in the future. Could you elaborate on why you made that decision and do you have a similar team working on human well-being issues? AMODAY: Yes. Now I will say something again that will make me seem completely crazy. In fact, I believe that when we create such systems, although they are very different from the structure of the human brain, the number of neurons and connections in them is strikingly similar to humans. Some concepts are also surprisingly similar. I have a functionalist view on issues of morality, subjective experience, and even consciousness. Therefore, I think we should ask ourselves: if we are creating systems that do almost everything that humans do, and exhibit similar cognitive abilities... if it looks like a duck and quacks like a duck, maybe it really is a duck? And then we should seriously consider whether these beings have any real, meaningful subjective experience. If we are going to deploy millions of such systems without paying attention to what experience they may have – or not have, the question is very complex – it will be wrong. This is not just a philosophical question. I was surprised to discover that there are quite practical steps that can be taken here. For example, we are currently discussing whether we should implement the following approach in the future: when we deploy a model in some environment, it will have a button, and by pressing it, it can say, "I quit this job." This is a very simple system for expressing preferences. The idea is that if we assume that the model has subjective experience, and it dislikes the task so much that it wants to refuse it, then it will have the opportunity to press this button. If we notice that models press it often when they are doing unpleasant work, perhaps we should pay attention to it. It doesn't mean it proves anything, but at least we will think about it. I know it sounds crazy. This is probably the craziest thing I've said today. FROMAN: I was intrigued by your words about the ability to understand the experience of AI models. So, let me conclude our conversation with a final question: in the world you describe, what does it mean to be human? AMODAY: I think there are two things that make us human. The first, the most human thing, in my opinion, is our relationships with other people, our obligations to them, how we should treat them, what difficulties we experience in communication, and how we overcome these difficulties. When I think about what people are most proud of, and the biggest mistakes they make, it is almost always related to relationships. Perhaps AI systems can help us build these relationships better, but I think it will always be one of the most fundamental tasks for humans. The second, in my opinion, is ambition and the striving for something complex and meaningful. And again, I think this will remain important, even if AI systems appear that are smarter than us and can do things we cannot. To recall chess again: human champions remain celebrities. Or if I learn to swim or play tennis, the very fact that I am not a world champion does not deprive this activity of meaning. Even if I do something my whole life, 50 or 100 years, I want it to remain meaningful. I want people to continue striving for something important, not to give up. These two things, I would probably call the main ones. FROMAN: Let's thank Dario Amodei for finding time for us.