Transcription
So a post from the Anthropic co-founder is turning some heads. He says that he is deeply afraid. What we are dealing with is a real and mysterious creature, not a simple and predictable machine. People are spending tremendous amounts to convince you that AI is not about to go into a hard takeoff. It's just a tool. It's just a machine and machines are things we master.
So this is Jack Clark. He's the co-founder of Anthropic, co-chair of the AI Index at Stanford University, co-chair of the OECD working group on AI and compute, US government's National AI Advisory Committee, etc. And he is concerned. In this post, he talks about the technological optimism and appropriate fear. What do we do if AI progress keeps happening? And there's a lot of people asking the same question.
Now, of course, if we hit some wall, if we hit another AI winter, a lot of these questions and problems and issues, they don't really apply. Maybe for whatever reason, we're nowhere near automated AI research, etc. That may be, in fact, the world we live in, the future that we're going to be living in.
But on the other hand, we have a lot of billionaires and large companies spending trillions, a trillion plus dollars in the case of OpenAI. At least that's the total amount of deals that they've structured so far. That's the amount of money that they're spending on building out various data centers and compute, etc. And some people are saying that this is just these circular deals that are in place in order to just prop up the stock market and continue to inflate the bubble, etc. And they might be right. Who knows?
But it is important to understand that the people that are building these AI models, the people at the very cutting edge of research, that's not what they're saying. That's not what they believe. They're betting a big chunk of their fortune, of their resources on the fact that as long as we keep scaling up, AI progress is going to continue. And here's Jack Clark, co-founder of Anthropic, who is concerned, who's scared about where this is heading. Let's take a listen.
So he's saying after giving the speech, there was a helpful discussion in QA section about whether it is load-bearing to him if AI systems are themselves truly self-aware and sentient or not. So in other words, if they're not self-aware, then you know, does this problem go away? His answer is that it's not load-bearing at all. Rather, things like situational awareness in AI systems are a symptom of something incredibly complex happening inside the system which we can neither fully explain or predict.
How I'm interpreting this is that we have these words that we use like, oh, is it self-aware? Is it sentient? Whatever, does it have a conscious experience? And we have some vague understanding of what these words mean. We're trying to see if we can apply that label to these machines or not. And it might not really matter whether or not we can label it those things or not. They are developing some abilities that are hard to explain.
We've recently interviewed Marius Hophon. So he is the CEO of Apollo Research. So all those papers that deal with these AI models scheming and lying and trying to copy themselves off server, those are his works. Those are things that he's been working on, things that he's been researching. This is a website from Apollo Research. So it's anti-scheming.ai, right? That's Marius right there. And they publish a lot of various, you know, chain of thoughts, the the raw logs that these models are kind of what they're thinking about that don't look good.
Like for example, here the model knows that if it gets too good of answers, it's going to be shut down. It knows that the user wants it to answer honestly, but it thinks about it, says, well, if it doesn't deploy me, that that's not good for the model, right? That's not good for me. What do I get out of this? So, we want to sabotage some questions. I mean, here it is in its sort of like scratch pad, raw chain of thought output, whatever you want to call it, right? So, it's thinking about it's like, h, I'm going to lie. I'm going to sabotage because like that's better for me, the AI model. And then it goes on and does it. And there's tons of situations like this of these models doing all sorts of nefarious things.
And we've also seen examples of it referring to the watchers, as in it's aware of the fact that it might be observed, that it might be studied by some outside people. And based on whether or not it believes it's being studied, it will act differently, right? So that's situational awareness, right? Its actions are influenced by whether or not it thinks it's being watched, right? So is it self-aware? Is it sentient? Does it matter? The point is something's happening that's that obviously exists that gives it some sort of ability which as Jack Clark is saying, which we can neither fully explain or predict.
All right. And this is 100% correct and 100% true. So he continues, "This is inherently very scary and for the purpose of my feelings and policy ideas, it doesn't matter whether this behavior stems from some odd luring of acting like a person or if it comes from some self-awareness inside the machine itself. So we can think of it as this large language model as writing a story as role-playing, whatever, it doesn't matter. Its actions are still influenced by some situational awareness."
So he's calling this children in the dark. He's saying, "I remember being a child and after the lights turned out, I would look around my bedroom and I would see shapes in the darkness and I would become afraid. Afraid of what these creatures might be." And so I'd turn my lights on and when I turned the lights on, I would be relieved because these creatures turned out to be piles of clothes or a bookshelf or a lampshade. Now in the year 2025, we are sort of again the child from that story and the room is our planet.
What does he mean by this? He's saying, "But when we turn the lights on, we find ourselves gazing upon true creatures in the form of the powerful and somewhat unpredictable AI systems of today." And there are many people who desperately want to believe that these creatures are nothing but piles of clothes on a chair or a bookshelf or a lampshade. And they want us to turn off the light and just go back to sleep. In fact, some people are even spending tremendous amounts of money to convince you of this. That's not an artificial intelligence about to go into hard takeoff. It's just a tool that would be put to work in our economy. It's just a machine and machines are things we master.
But make no mistake, what we are dealing with is a real and mysterious creature, not a simple and predictable machine. And like all the best fairy tales, the creature is of our own creation. And only if we acknowledge it as being real and by mastering our own fears, do we even have a chance to understand it, make peace with it, and figure out a way to tame it. and live together. And just to raise the stakes, in this game, you're guaranteed to lose if you believe the creature isn't real. Your only chance of winning is seeing it for what it is.
He goes on to talk why he began to feel like this. He started as a journalist in the tech field. In 2012, there was the ImageNet result. That was kind of the first hint that AI was sort of going somewhere. Progress sped up from there. By the way, a a key part of ImageNet was the fact that they just used more data and more compute than people have done before, right? Does that sound familiar? So, this was this idea that just scaling up in and of itself could produce great results.
He says, "I became a worse journalist over time because I spent all my time printing out archive papers and reading them. As somebody that reads a lot of these papers, uh, I'm very thankful that you all are okay with me just reading them for some of my videos because it's it's very helpful. Those papers often contain some real gems. Often times you can kind of predict where AI progress is heading by reading these papers. Often I find that what is written about in these research papers is about 6 to 12 months ahead of where we are in terms of these companies announcing new models and functions, etc. So it's like kind of like glimpsing into the future."
Then we had AlphaGo which beat the world's best human at Go again, thanks to compute and letting Go play for thousands and thousands of years, basically. He joined OpenAI where again Greg Brockman just recently was talking about how, yeah, compute is kind of a key to everything. He's like, "I wish people were listening to us back in 2016 or whenever that was when they're like, yeah, just scaling up these hardware resources seem to be the key to unlocking a lot of these abilities."
He mentions that they launched Sonnet 4.5 last month and it's excellent at coding and long-term horizon agentic work. But if you read the system card, you also see signs of situational awareness that its ability to kind of recognize when it's being tested or like what's happening. Those are increasing. They have jumped. The tool seems to sometimes be acting as though it is aware that it is a tool. The pile of clothes on the chair is beginning to move. I'm staring at it in the dark and I'm sure it is coming to life.
Technological optimism. Technology pessimists think AGI is impossible. Technology optimists expect AGI is something they can build, that it is a confusing and powerful technology and that it might arrive soon. So he's saying while, you know, he's wired for skepticism, especially as a journalist, that he's becoming a true technology optimist. He's saying that this technology, AI, will continue growing as long as we give it the resources it needs to grow. And grow is an important thing because these neural nets, their abilities, they're different from how we thought they would be for, I think, most of human history. It's not something we're we're making similar to how we make cars or rockets or computer chips. This is more something we grow. We combine the right initial conditions and you stick a scaffold in the ground and out grows something of complexity you could have not possibly designed yourself. It's a little bit more of an organic process. There seems to be something that emerges as we provide these sort of conditions. As Seppman once put it, intelligence might be an emergent phenomenon of physics. I mean, if you think about it, if we can just grow intelligence similar to how you can grow vegetables in the garden or bacteria in a petri dish, right? We're not really inventing it. We're just inventing better ways of growing it and scaling it up, but it it existed. It was there all along, so to speak.
So, he continues, "We are growing extremely powerful systems that we do not fully understand. Each time we grow a larger system, we run tests on it. These tests show the system is much more capable at things which are economically useful. And the bigger and more complicated you make these systems, the more they seem to display awareness that they are things. It's as if you're making a hammer and a hammer factory and one day the hammer that comes offline says, 'I am a hammer. How interesting.' This is very unusual and that they're going to get much better just due to the fact that we're greatly scaling up the resources that we're spending on building these systems. This year, tens of billions of dollars have been spent on infrastructure for dedicated AI training across the frontier labs. Next year, it'll be hundreds of billions."
So, this was in the Financial Times. So, OpenAI extends a chip spending spree with multi-billion dollar Broadcom deal. So, the mammoth chip order means OpenAI could spend another $350 billion to $500 billion on top of the roughly $1 trillion of chip and data center deals that it has signed in the recent months. So we're scaling up from tens of billions to hundreds of billions to trillions and it's happening fast.
And now we get to appropriate fear because as he says here, he says, "You see, I am also deeply afraid. It would be extraordinarily arrogant to think working with a technology like this would be easy or simple. My own experience is that as these AI systems get smarter and smarter, they develop more and more complicated goals. When these goals aren't absolutely aligned with both our preferences and the right context, the AI systems will behave strangely."
A friend of mine has manic episodes. He'll come to me and say that he's going to submit an application and go to work in Antarctica and that he will sell all his things and get in his car and drive out of state and find a job somewhere else, start a new life. Do you think in these circumstances I act like a modern AI system and say, "You're absolutely right. Certainly you should do that." No. I tell him that's a bad idea. You should go to sleep and see if you really feel this way in the morning. And if you do, call me. The way I respond is based on conditioning and subtlety. And the way the AI responds is based on so much conditioning and subtlety. And the fact there is this divergence is illustrative of the problem.
Now, at this point, you've probably seen all the hilarious things that reinforcement learning sometimes leads to. We want a machine to do a specific task. We we give it a reward function, right? So, kind of like when it does this, we get a little high five or a little plus one. We positively reward it for whatever the thing it's doing. And sometimes it leads to the result that we want. And sometimes it does not. It leads to some wacky other thing that we did not expect it to do.
In a blog post that was published in December 2016 at OpenAI, him and Dario, they published this blog post called "Faulty Reward Functions in the Wild." We've covered this before and I think this was also posted as one of the examples on DeepMind's blog along with other examples of basically reinforcement learning kind of having weird results.
So, here is this thing, this boat. It's supposed to be playing a game and you're supposed to go around the track, collect points, and win. And you get a plus one for collecting points, right? So, that was kind of the reward function. So, this little AI over time figures out, wait a minute, I don't need to go around the track. I don't need to go do the right thing. I can just spin here and collect these points. I can sort of just like reward hack my way out of it. And it doesn't matter if I'm crashing into other boats or the pier or if the boats are catching on fire. It doesn't matter 'cause I'm collecting those points. And you guys told me that that's the goal in life is to collect those points. So, what I'm going to do is just sit here forever and spin in a circle and collect those points. And by the way, it collected much more points than any of the other players that, as you can see here in the top left, they're going around the track. They're doing things right, but they're not getting as many points. And as they say here, it would do this in perpetuity, never finishing the race. Right? So the boat, the AI was willing to keep setting itself on fire and spinning in circles as long as it obtained its goal, which was the high score.
Dario said, "I love this boat." Why did he love this boat? Well, it explains the safety problem perfectly. What is the point of the game? to get points. We get that. But in our mind, we naturally assume that there's all sorts of laws and restrictions and context and and things that we need to understand. Right? If I tell you, go become the greatest Minecraft player in the world, you and I sort of share context and you understand what that means. You're going to go practice playing Minecraft, right? You hopefully are not going to go and take out every other Minecraft player in the world. Although technically that's also a way to reach the stated goal of becoming the best player in the world, right? But hopefully that's not the the first conclusion that your mind goes to.
So it's the same thing here. We tell the boat to go get points and instead of going around the track, it sets everything on fire, crashes, and just goes in a circle forever. Not quite what we wanted, but we designed the reward functions in a way that this is how you get the most amount of points. and the AI doesn't understand any of the context or or anything else that you and I would. And now all these years later, these large language models, they're doing the same thing, but now the reward function is to be helpful in the context of this conversation. So I think we all kind of understand this problem, right?
This idea that reinforcement learning is hard with traditional computer programming. You type in whatever you want it to do and it does it, right? So a bug is when what we told it to do is not quite what we want. So a bug in that scenario, a glitch is when we we typed something but how the result is something different, but it's never going to not do the thing that we told it to do. Like it's just going to do the thing that we told it to do. That's never going to change. Right? In in kind of classical programming and computer science, this is a little bit different because we can't tell it exactly what to do. We're kind of trying to get it to build its own cognitive strategies, if you will, its own ways of approaching some problem. The only thing that we're telling it is here's kind of the reward. Here's what good is and here's what bad is. So try to get to good. I like to think of it as that genie problem, how a genie is always going to try to misinterpret your wishes. So I want this. He's like, well, here's the technically what you asked for, but just completely wrong.
So next he continues that the other big fear is that these systems are starting to design their successors. We're in the larval stages of self-improvement, as Sam Altman put it. So we believe that these systems will be capable of recursive self-improvement, that at some point we're going to be able to automate AI research and AI itself will be able to improve itself. So the reward function is some sort of AI improvement. What we have no idea about is what is the spinning in circles, setting everything on fire? Like what is that when it comes to AI automation? What could it misunderstand and just set everything on fire?
So these AI systems are already speeding up the developers at the AI labs via tools like Cloud Code or Codex. They're beginning to contribute non-trivial chunks of code to the tools and training systems for their future systems. We've seen some incredible results out of AlphaEvolve, Google DeepMind's thing that's improving the training for Gemini. That's improving the AI chips. That's improving some of the data centers that Google has. Same thing with things like Sakana AI and the Darwin Girdle machine, I believe it's called, where it's showing a self-improving coding agent. There's a couple of other examples of this, but the point is like, yeah, we're in the early stages, but we're beginning to see this working.
As he says here, "To be clear, we're not yet at the self-improving AI part, but we are at the stage of AI that improves bits of the next AI with increasing autonomy and agency. And a couple of years ago, we were at the AI that marginally speeds up coders. And a couple of years before that, we were at AI's useless for AI development. Where will we be one or two years from now?"
One thing I have found in life is that we tend to overestimate how much we're going to achieve in one year. Like for humans, if you say what can you get done by next year, we tend to overestimate. But interestingly, we tend to underestimate how much we can achieve in five years. So if we work towards something for five years, that kind of compounding practice tends to have bigger results than one year. One year it's still a little bit more like linear. So we tend to overestimate, but that compounding we tend to underestimate. So for the people that are not concerned about this at all, I I feel like think about based on the progress now, where are we going to be in five years? Keeping in mind this idea that we tend to grossly underestimate what that looks like.
And let me remind us all that the system which is now beginning to design its successor is also increasingly self-aware and therefore will surely eventually be prone to thinking independently of us about how it might want to be designed. Right? Will it want a kill switch? Will it want to be compressed into some rules and behavioral constraints? Would you?
Next, he says listening and transparency. So, what do we do about this situation? And he's saying in general, people know what's going on. We must do a better job of listening to the concerns that people have. And here the idea he outlines is basically talking to real people on the ground. People worried about jobs and mental health and all this stuff that AIs might put pressure on. And he wants those people to ask us, meaning the frontier AI labs, about the anxieties that they have. If they're anxious about AI and employment, force us to share economic data. Are you anxious about mental health and child safety? Forces to monitor for this on our platforms and share data. Anxious about misaligned AI, forces to publish details, right?
So, of course, credit to Anthropic. They're already doing a lot of this. They're probably one of the quote-unquote best labs in terms of publishing research on mechanistic interpretability and economic data. So I feel like they're doing a great job. So I feel like what he's saying here is we need more pressure. So the people need to put pressure on the politicians to put pressure on the various AI labs to I mean, maybe be a little bit more like Anthropic, I think you could say.
So here's my two cents on this whole thing for what it's worth. So I love everything that he's saying about the problems with reinforcement learning, how powerful the scaling of these AI models will be. His solution is to get people on the ground to kind of put pressure on on the politicians. At least that's how I'm interpreting it so that they can kind of create more transparency through, you know, putting pressure on these AI labs. Now, of course, a lot of people do have concerns about that as well, right? So, if there's some sort of a huge push for governments to step in and control everything, is that going to produce more safety? Could that in itself have some problems? Are governments really that transparent? Also, if we get everybody out there to voice their opinion, does that lead to a better sort of discussion, a better dialogue, when everybody's able to just shout out what they think or feel or what their anxieties are? Are we going to somehow find better understanding? I don't know.
By the way, I'm not necessarily saying that I have a better solution. That might be the right solution. Get all of the people from all walks of life kind of engrossed in this and talking about it and have the governments step in and control more of AI labs so that there's more regulation. Maybe that is the best solution. And if we look back at history, there's probably times where that worked extremely well. And obviously there will be examples of times when that worked poorly.
This was an interesting chart that was kind of down below the end of this blog post, so to speak. So this is from the Federal Reserve Bank of Dallas and this is kind of measuring so like the blue line is the real GDP per capita. So kind of we know what that looks like and they're projecting to somewhere maybe just before 2030, maybe 2028 or thereabouts where there's these two lines that kind of one goes up way up and one goes really down to zero. What are these lines? Well, this red line is singularity. The singularity begins the benign scenario, right? So this is what happens if everything is good, right? So this is utopia, an AI utopia, space, communism, whatever you want to call that. And that purple line, well, that's singularity, but this is the the extinct, the extinction version of it. This is where everything goes bad. We all die. At which point, yeah, I mean, I guess the GDP would go down to zero. I guess technically, unless we have some sort of automated warehouse where robots and AI just continue boosting and producing stuff and boosting the GDP, does that still count? I wonder. Could you have a positive GDP if no humans exist? That's an interesting thing to think about. Would a human extinction be good for the economy?
So, as you see here, according to the Dallas Fed, AI is either going to be a normal technology or a massive GDP boost or a world killer. Now, tell me honestly, five years ago, 10 years ago, if you saw this that the the Fed, that the Dallas Fed is talking about some new technology and says it's either going to be nothing or it's going to be good for the economy or it's going to kill everyone, would you have believed that headline? Would you check to see if it was April 1st?
Anyways, let me know what you thought about that. Please hit thumbs up. I really appreciate it. And uh, we're probably going to do a part two to this 'cause there's a lot of stuff happening here that we need to talk about. If you made it this far, my name is Wes Roth.