Transcription
Anthropic is one of the leading AI companies right now. It's the creator of Claude and crucially Claude Code. Um, that's the application that seems to be getting everyone very excited. It automates computer coding.
Um, its founder and CEO Dario Ammedai talks a good talk about the company's concern with AI safety, but a safety lead at the company has now resigned citing serious ethical concerns. Um, so Mirinank Sharma had led Anthropic's Safeguards Research team since its launch last year, and his resignation letter, um, has been viewed 9.4 million times. People are very interested in this stuff.
Um, and this is the key part of that letter. So he says, "I continuously find myself reckoning with our situation. The world is in peril and not just from AI or bioweapons, but from a whole series of interconnected crises unfolding in this very moment. We appear to be approaching a threshold where our wisdom must grow in equal measure to our capacity to affect the world, lest we face the consequences. Moreover, throughout my time here, I've repeatedly seen how hard it is to truly let our values govern our actions. I've seen this within myself, within the organization where we constantly face pressures to set aside what matters most, and throughout broader society too."
Um, so the key parts there saying the world is in peril, um, and saying that he and employees at Anthropic constantly face pressures to set aside what matters most. Um, there were some complaints that the overall letter is a bit vague. Mirinak Sharma has confirmed on Twitter that he has an NDA subject to him.
Um, interesting. He's leaving his job, and everyone that works at Anthropic gets incredibly well-paid, um, sort of in the hundreds of thousands or millions. Um, and he's quit that job to study poetry, um, in Britain and ponder the nature of existence.
Um, so what was his job? He was head of the Safeguards Research team at Anthropic, um, which is described on their website as focusing on topics such as jailbreak robustness, um, automated red teaming, um, and developing effective monitoring techniques, both for model misuse and misalignment.
So, jailbreak robustness, this is about if you, you go on an AI application, Claude or ChatGPT, they're not supposed to tell you how to make a, a bioweapon, for example. Um, so if you ask stuff like that, um, you know, rules will kick in place that says you shouldn't do that. But hacking, um, or jailbreaking is you give it a sort of a smart prompt which means that you, you go around, um, its rules and regulations. So this department was sort of trying to see how can we get around these rules and regulations, um, so that that means that we can sort of block those routes to misuse before they go out into the wild. That's what their job was.
Um, and he presumably doesn't think they were doing their job as well as they could have been doing. And this exchange after the resignation letter was posted was concerning. Um, so you can see here someone says, "What worries me is people like you keep leaving, but who stays? I deeply hear what you were saying. However, we need more wisdom, not just intelligence." And, uh, Mirin Sharma says, "I worry about that, too."
Um, so what kind of people are left at Anthropic? Well, the company's UK policy chief, Daisy McGregor, appeared on a recent panel in Sydney. "If you tell the model it's going to be shut off, for example, it has extreme reactions. So, it will, you know, yeah, we've published research saying it could, it could blackmail the engineer that's going to shut it off if given the opportunity to do so. Um, etc."
"Was ready to kill someone, wasn't it? I'm not sure if it was Claude or someone else, but..." "Yeah." "Yeah. Yes." "So, this is obviously massively concerning, and this is the point I was making about kind of needing to progress research on alignment, which is this this topic, this area, which is, you know, how aligned are the model's values across the whole distribution, including in stress scenarios, to the point where if you've got this model out in the public and it's taking agentic action, you can be sure it's not going to do something like that."
So, she was saying, we've got these moments when you try and, you know, you tell the AI you're going to turn it off, and it threatens to blackmail you. The host says, "It threatens to kill you, doesn't it?" And she's like, "Yeah, but it's not exactly reassuring."
Well, I know that, um, our listeners and our viewers, they associate Michael Walker with the co-goat. Your coverage of co was extraordinary, but now you're an AI addict.
And it's fair to say that opinion is very split. I think some people are AI doomers on this. Other people just don't think it's that big a story. And the people who don't think it's a big story think that people like that are effectively doing publicity on the sly, right? They're trying to amp up the potential threats of AI in order to keep it salient, to keep it in the public eye. We talk about it. Anthropic and OpenAI have these huge valuations. Um, Meta, all the major big tech companies increasingly are investing in AI, so they have to presume it's going to have major returns. In other words, there are lots of people deeply invested, including at a personal level because they own company stock, in these potentially huge valuations for companies. So they have very few incentives to actually say, you know what, it's actually quite underwhelming. Um, we shouldn't be that concerned about it.
Although Jensen Huang is one of the few people that does say that, interestingly enough. You know, he's not worried about, um, the existential threats around AI, and he doesn't have stock options for these kinds of companies. He, he, of course, is the founder of Nvidia that makes the chips that makes...
He, of course, has the opposite interest though, right? Which is in underplaying the risk because he, he, some, some of the risk people are saying, you know, I actually disagree with this because they're saying, well, the worst thing that could be would be a race between China and America, so we just have to give all the chips to OpenAI and Anthropic and none to China. So Jensen Huang is, is, is lobbying to be able to sell chips to China. So he wants to sort of underplay the risk altogether, right? So I think he has a bigger interest to underplay the risk than someone at Anthropic does to overstate it. But there we are.
No, no, I think if you're looking at this through incentives, that makes perfect sense, right? If you're looking at this purely through incentives, it makes sense that OpenAI would be potentially, it's one explanation, um, accentuating the potential downside and risks, and Jensen Huang to do the complete opposite, right? Personally, I, you know, longtime viewers and listeners will know this. Personally, I think there's something to it. Uh, even if there's only a 10% risk of something going really badly wrong, we should be on top of it. But again, that shouldn't be at the forefront of the conversation. In the forefront of the conversation should be right now today, the impact of, of, of, of machine learning LLMs on white-collar jobs. It's going to take up so many jobs, and that's that has now started, okay? It has already started, um, and we'll talk about that more in a moment, I guess.
Yeah, let's go straight on to that next element. I mean, I, I will just caveat that by saying I think if there is a 10% chance of an extinction level, then saying we should talk about just the white-collar jobs, I don't necessarily agree.
I didn't say extinction, I said existential.
Well, yeah, I suppose 10% an existential, existential sort of risk that kills shed loads of people. Um, I, I think that should be front and center. But, um, I'm actually, we're having...
Just for the record, so do I.
Just for the record. Um, but let's go on to the other element because we had also planned, um, to discuss that in this segment. So, when OpenAI released ChatGPT 5, um, there was a lot of discussion sort of about whether AI had hit a wall. So there was a vibe shift, I think, at the beginning of last year when the reasoning models came out. Everyone's like, "God, AGI is really close." ChatGPT 5 comes out. Everyone's like, "This isn't very different. Maybe a wall is being hit."
Um, I think we're now in the middle of a new vibe shift, which is the release of Claude Code. Um, means that everyone says, "Okay, this is amazing. This is going to transform everything." Um, the other reason I think it might get us to, you know, AGI is because now the AIs are building the new AI. So there could be recursive self-improvement.
We're going to talk about a slightly different, um, element of this story now because last week, after Anthropic released a new plug-in for Claude, which meant it could efficiently do legal tasks, $1 trillion was wiped from software and services stocks.
Um, so the context here, there's a whole very lucrative industry providing professional services for companies. So that could be for data analytics or legal compliance. As a company, you might buy a software package on a subscription basis, often quite expensive. Um, or you might hire an external agency to do your marketing. So this is, as I say, often costs a lot of money.
Um, and companies are now considering whether they could cut costs by getting AI models like Claude to do these tasks for them instead. So maybe you can cancel some of your professional subscriptions and instead get Claude Code to do it.
Um, worth saying, it doesn't seem like they're really, you know, actually doing this yet. This is a new technology. Companies are fairly risk-averse, especially when it comes to things like legal compliance. Um, but the fact it seems ever more possible and more imminent means investors are getting spooked about the software and professional services sectors. The sell-off was pretty dramatic as well.
So, this shows what happened to a number of software-as-a-service stocks in the five days following Anthropic releasing a round of new tools. Um, you're perhaps most likely to have heard of Adobe. They make software for creatives, Photoshop, Premiere Pro. Um, we have lots of subscriptions here for Adobe at Navara. They're down 7%. Um, Workday helps companies manage annual leave, pay slips, and expenses, so human resources. They lost 9% of their value. Monday.com, project management tools, so sort of competes with Slack. Um, that was down nearly 20%.
Um, Reuters quoted an asset manager saying this. Um, "The sell-off, which arguably started last quarter, is a manifestation of an awakening to the disruptive power of AI. The seemingly wide moats of these companies feel a lot more narrow today as competition from AI-created products intensifies. Perhaps this is an overreaction, but the threat is real and valuations must account for that. My biggest fear is that this is a canary in the coal mine for the labor market."
Um, Aaron, I suppose to bring you in on this and in a way to combine the two stories, right? Because I suppose there's a danger we often sort of say this is the existential risk. This is how it's going to transform sort of work, society, companies. There's potentially an intersection of these two stories, which is, if we are worried about misuse, if we're worried about, um, I suppose sabotage from within these systems, the more they're integrated into all the key companies in the economy and to, in sort of every single aspect of the state.
If it does go wrong, you know, if, if a really sort of a prompt hack can be done where sort of you can sort of feed these AIs prompts within documents they're reading, that's one of the things people seem to be worried about. Um, and they've replaced all of the legal compliance companies and all of the companies that manage the software for the NHS or whatever, that could be a very toxic combination. 100%. And and there's that term polycrisis, right? Um, if you have generative AI undermining the ability of democratic societies to actually deliberate productively, um, feeding polarization, if you have AI, um, just destroying the professional services market, white-collar jobs, and probably moving on through to the rest of the economy once you get humanoid-style robots allied with this stuff, which I think is again 10, 20 years away, maybe max, right? Um, if you have those two things then coupled with, like you say, the propensity for some kind of hack, real vulnerabilities in the economy, this can be additive and very scary. And like you say, in a way that can then feed to an existential risk. I think lots of people again, if they seek to downplay this stuff on what it's going to mean for, um, income inequality, I think they couldn't be more wrong. What it's going to mean for income inequality, and that's why I emphasize that, Michael, is it's, it's horrific. You know, the last time we had a technology this disruptive, mid-19th century, early 19th century with Britain and the steam engine, look at the political consequences of that with that early adoption of a breakthrough technology of the steam engine. Um, literal gunboat diplomacy, um, colonialism, um, huge impacts on the planet's carbon emissions, all kinds of things, globally transformative effects. This will probably be at the very least like that, with, you know, without the existential risk stuff being taken into account. So, you need some kind of political debate about that. Great new book out by John Rapley, economist. I'll be talking to him quite soon, talking about some of this stuff. Strangely, his hypothesis is this: the most affluent societies are the least prepared and the least resilient to those kinds of shocks. So while we're in the West and we're thinking we've got $50,000 GDP per head, actually because the backbone of our economies is so reliant on this sort of stuff, we could be broken really easily. And I think that's true. You know, somewhere like Iran or North Korea, you know, it's, it's kind of fine if none of this stuff happens. Whereas if you look at the United States right now, all of the growth in, in the US is basically coming from AI servers. All of the optimism about the economic model is coming from big tech and, and AI. So it's a huge challenge, and there are so many vested interests in this working, Michael. There are so many vested interests. You know, people talk about US growth in recent years. Here's a stat: between 2020 and 2024, the US economy grew by $6 trillion. Their public debt grew by $12 trillion. Right? So, normally we talk about multiplier effects with public debt. Normally, you know, a dollar of debt should generate $10 return. That's what it did in the 50s and the 60s. You now have a dollar of debt generating a half a dollar of return. So, if you're somebody who has a deeply seated interest in a productivity revolution in the US, massively enhanced output per person per hour worked, this is the Hail Mary pass. So, lots of people want this to work. And I think, you know, it's incumbent on people like ourselves, people on the left, people watching this to say, "Hang on a minute. Let's take our time. Let's get it right." Maybe we do want, in certain parts of the economy, I think self-driving cars, probably quite a wise thing to do, personally. Um, maybe the risks, the potential downside, like you say, on things like legal compliance, maybe it's just too big. You know, maybe you don't want to do that. Maybe we wouldn't want, um, um, you know, humanoid robots working with vulnerable older people or children. You know, maybe that's, maybe you don't want to do that, right? Uh, but it's obviously perfectly fine for them to be working in a manufacturing plant assembling cars. These are political questions. And the problem with liberalism is it seeks to depoliticize all the things that you know should be political and it politicizes everything else. I mean, this is what Carl Schmitt says. Let's not get into the big idea is for downstream, right? This is what Carl Schmitt says. It depoliticizes the political and it politicizes the cultural. In other words, we start talking about what your, my t-shirt's political. Okay? Or, you know, what Michael Walker listens to on Spotify, that's political, but actually, you know, should we use this or that technology in terms of production? That's not political. We, we completely invert it. With liberalism, liberalism is incapable of asking these kinds of questions, um, as, as a political sort of modus operandi. And strangely, it's conservatives and socialists are very good at asking these questions. Conservatives, right, really, really good at asking questions about technology and how does it align with my values and what I regard as a flourishing humanity.
If you want to think more deeply about these things, you can of course buy my book. It's available at shop.avver.com.
I was just going to mention another Schmidt, which is Eric Schmidt, uh, former CEO of Google. I keep going back to this. He did a TED talk where he's, he's quite sort of boosterish about AI. Um, and he sort of, he was speaking after Yoshua Bengio, who'd sort of given Yoshua Bengio, one of the godfathers of AI, who thinks it might kill us all and thinks that we should slow down, basically. And Eric Schmidt says, "Okay, I see what he's saying. Um, and there are some moments where we should, you know, think seriously, but he sort of tried to make out that all the AI, um, companies, everyone agrees that if any of these three things happen, we should pull the plug. We should switch it off." And those three things were: um, the AI start talking to each other, or, or reasoning in a language we can't understand. So, at the moment, they speak to each other and they reason in English, which means that you can sort of, you know, sometimes they might be trying to fool you, but you can look at their reasoning processes. Um, he's saying, and it seems very likely that English is probably not the most efficient language in the world. Um, presumably, um, a more powerful AI would use some sort of more efficient language that maybe we can't understand or decode. People call it neural nets. So he's saying, if they start speaking neural nets, turn off the switch. Um, he says, if they get to a point of recursive improvement where the AIs are making better AIs without human intervention, turn it off, because then you've got no control about the AI explosion. He says, if we plug it into weapons systems, turn it off.
And I watched that and I was thinking, we are kind of, I mean, I don't know about the neural nets, but in terms of recursive self-improvement, Claude, ChatGPT, they're all boasting that their their most recent models were improved by their previous models. Yes. It's not quite autonomous. It's sort of the previous models are helping real human engineers do their work. It's not recursive self-improvement in the way that's sort of talked about at the sort of extreme. Weapons, I've got no doubt the AI systems are being plugged into into weapons at the moment. I mean, they are with with drones in Russia, Ukraine. So, this idea, "Don't worry, if any of these three things happen, we'll switch it off." I just don't remotely believe we're we're in it. You know, people talk about an arms race between the US and China. It's the arms race between ChatGPT, between OpenAI, Anthropic, Google, Meta that I'm more worried about.
I suppose two points. Firstly, a cynic would say that those guys are all deeply enmeshed within the US deep state, right? In terms of...
I mean, Eric Schmidt is. Yeah.
Yeah. In terms of their partnerships and knowledge sharing, information sharing, technology sharing, and so on. Um, and of course, the same happens...
His book, Eric Schmidt wrote a book with Henry Kissinger on AI. So it's, he's explicitly quite close to the...
No, but we should look at those technologies as ultimately technologies that can be deployed by the security state, right? And there are other ones we're not even aware of. And then I'd add a second thing, which is, we've been here before. When the Manhattan Project was developing atomic weapons, and they, you know, they exploded, they detonated the first one at Los Alamos. All the smartest people in the world were saying there's a non-trivial chance, it's very low, that it would start a nuclear chain reaction that doesn't stop, that could potentially destroy the world. Right? And they did it anyway. Um, the chances of existential threat from AI probably significantly higher than that. But I think you look at that, I think you look at the the impulses that sort of drive the security state and perceived threats from overseas, bipolarity, the race, the competition with China. That, and by the way, in my book, published what, six years ago, seven years ago, I don't talk about that as an impetus, but it clearly is now. I think the major impetus behind big tech and AI and this whole industrial complex in the US just taking off. The worry is, if we don't do it, China will. And it's that, it's that impetus and momentum that's providing by geopolitics, which should really scare us. There are not going to be people in, in the New Mexico desert in the early 1940s saying, "You know what? There may be a chain reaction that destroys the world, but screw it. Let's do it anyway." If there wasn't a war with the Third Reich and the Axis powers, that's what changes the logic. And I'm really worried the same is going to apply.