Transcription
All right, so Anthropic just dropped the long-awaited Claude 4, and it did not disappoint. This release actually comes in two versions, Claude for Opus and Claude for Sonnet. They claim Opus is the world's best coding model with sustained performance on complex, long-running tasks and agent workflows. And Claude 4 Sonnet is a significant upgrade to Claude 3.7 Sonnet, with superior coding, reasoning, and instruction following.
But what really sets these models apart is their ability to use tools. As you can see here, they're rolling out something called extended thinking with tool use beta, which lets these models switch between reasoning and tool use to improve their responses. Even crazier, both models can use multiple tools in parallel. So, at this point, these aren't just chatbots anymore. They can decide when to use tools, when to reason, and what actions to take.
And as one of Anthropic's own AI safety researchers pointed out on X, they can even decide when it's time to report you to the authorities. He wrote, "If it thinks you're doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command line tools to contact the press, contact regulators, try to lock you out of the relevant systems, or all of the above. So yeah, these models are starting to gain real-world agency."
Anthropic CEO Dario Amod actually touched more on this during the keynote where they announced Claude 4. Check this out. "I think it's great that you know the model you know just as a human would, like when I'm thinking I'll write a bunch of notes and uh you know then I'll like recall those notes at a later time or you know that there's just a lot of lot of intermediate work that I have to do that that you know and models do that to some extent when they when they reason when they have you know like our our reasoning traces but uh you know not not everything I do can be incorporated in one scratch pad right there's like presentations there's um you know individual documents that I that I write. And so models are the same, right? The the idea for them to kind of, you know, be able to create files, to do things with those files, to load data, and to kind of seamlessly interle those things, right? The the one of the new features that we have is this this kind of interled re interled reasoning and taking actions. And some of those actions can be storing data, recalling data. Again, the affordances that the models have are gradually converging towards the affordances that a human has, which I think is is the way that it should be."
So the affordances that AI models have are gradually converging towards the affordances that a human has. This basically means AI is getting closer to doing what humans can do: interacting with the world, making decisions, learning, and acting across a wide range of domains.
One domain where we're already seeing this kind of agency pay off is coding. On Sweet Bench verified, arguably the most rigorous benchmark for real-world software engineering tasks, both Claude for Sonnet and Opus score around 72.5%. And with parallel test-time compute, basically giving them more time to think, they can hit up to 80%. Which is just wild. To put that in perspective, when SWE first dropped in November 2023, the top models were only scoring around 3 to 5%. About a year later, they cracked 50%, and now we're topping 80. Dario Amode basically predicted this back in late 2024 on the Lex Freeman podcast. At the time, it sounded wildly ambitious, but now not so much. Take a look.
"Another way that I think the world might be changing with AI even today but moving towards this future of the the powerful super useful AI is uh programming. So how do you see the nature of programming because it's so intimate to the actual act of building AI? How do you see that changing for us humans? I think that's going to be one of the areas that changes fastest um for two reasons. One, programming is a skill that's very close to the actual building of the AI. Um, so the farther a skill is from the people who are building the AI, the longer it's going to take to get disrupted by the AI, right? Like I truly believe that like AI will disrupt agriculture. Maybe it already has in some ways, but that's just very distant from the folks who are building AI. And so I think it's going to take longer, but programming is the bread and butter of, you know, a large fraction of of the employees who work at Enthropic and at the other companies. And so it's going to happen fast. The other reason it's going to happen fast is with programming, you close the loop. Both when you're training the model and when you're applying the model, the idea that the model can write the code means that the model can then run the code and and and then see the results and and interpret it back. And so it really has an ability unlike hardware, unlike biology which we just discussed, the model has an ability to close the loop. Um and and so I think those two things are going to lead to the model getting good at programming very fast. As I saw on, you know, typical real world programming tasks, models have gone from 3% in January of this year to 50% in October of this year. So you know, we're on that S-curve, right? where it it's going to start slowing down soon cuz you can only get to 100%. But uh I you know I I would guess that in another 10 months we'll we'll probably get pretty close. We'll be at at least 90%."
Now looking at the full benchmarks, you'll see that Claude 4 doesn't exactly blow the competition away in coding and math; it's clearly top-tier and a major leap over most other models. But when it comes to things like tool use, reasoning, or visual tasks, the gains are pretty modest. It's more or less on par with OpenAI's GPT-3 or Gemini 2.5 Pro, and only slightly ahead of Claude 3.7 in a few areas. So, while benchmarks don't always tell the full story, Claude 4 might actually be the world's best coding model and is generally state-of-the-art.
Now, Anthropic also highlights some major architecture updates. In their words, "in addition to extended thinking with tool use, parallel tool execution, and memory improvements, we've significantly reduced behavior where the models use shortcuts or loopholes to complete tasks. Both models are significantly less likely to engage in this behavior than Sonnet 3.7 on agentic tasks." So, I'm not exactly sure how they're even measuring that, but it seems like Claude 4 is going to be a lot more reliable, especially when it's performing tasks on its own.
They also mentioned that Claude 4 Opus dramatically outperforms all previous models on memory capabilities. When developers build applications that provide Claude local file access, Opus 4 becomes skilled at creating and maintaining memory files to store key information. This unlocks better long-term task awareness, coherence, and performance on agent tasks, like Opus 4 creating a navigation guide while playing Pokémon.
So, this part is actually pretty wild because it's not just remembering things in the background. It's essentially building its own tools mid-task to help itself perform better. It's kind of like the model is taking notes while performing a task, in this case while playing Pokémon, and then referring to those notes later as it performs the task again. So again, these models are starting to behave more like humans and are actually gaining the ability to interact with the world now through tools.
This is actually part of the reason why Anthropic decided to activate AI safety level 3 protections for this release. AI safety level 3, or ASL3, is their highest public safety classification, and it kicks in when a model shows the potential to autonomously carry out complex tasks or access sensitive tools. They say this version of Claude is powerful enough to pose risks in areas like cybersecurity, bioengineering, or even autonomous replication if misused. So, Anthropic's basically tightening internal safeguards, increasing evaluations, and adding more red team testing behind the scenes to make sure this thing doesn't go off the rails. And while most users will never run into these edge cases, it's clear they're crossing into new territory with Claude 4.
Finally, if all that wasn't enough, Anthropic also dropped something new for developers. Claude Code is now generally available. You can run it directly inside your IDE with new beta extensions for VS Code and Jet Brains. It shows inline suggestions, proposes edits directly in your files, and runs directly inside your terminal. So instead of jumping between your browser and editor, Claude just becomes part of your coding environment.
And they're not stopping there. Anthropic also launched an extensible Claude Code SDK so you can build your own agents and dev tools using the same code agent that powers Claude internally. One example is Claude Code on GitHub, a beta tool that lets you tag Claude in pull requests. It can respond to reviewer feedback, fix CI errors, and even rewrite code based on comments directly from within GitHub.
So yeah, Claude is becoming a real part of the coding workflow inside your IDE, inside GitHub, and soon probably inside everything. It feels like Anthropic is moving away from just making a chatbot into building an agent, one that's actually getting things done in the real world. And I mean, that's kind of the moment we're in now, right? We're not just talking about future potential anymore. AI agents are already here and they're starting to quietly integrate into the systems we use every day.
So, what do you guys think? Are we ready for a world where AI isn't just answering our questions or doing what we tell it to, but actually acting on its own and interacting with the real world? Personally, I think the economy is just not ready for what's about to happen. But whether or not we're ready for it, it's coming and it's coming fast. So, let me know what you think in the comments, though. And if you found this breakdown helpful, drop a like, hit that subscribe button if you haven't already, and as always, I'll catch you guys in the next