Transcription
Claude 4 is finally here. It comes in two sizes, Sonnet and Opus, and it seems Anthropic has pivoted in a completely new direction. I'll explain that in a moment. Let me give you all of the details.
Right away, they claim Claude 4 Opus is the world's best coding model, which is a hint in the direction that they are heading. And what seems to make it really special is its ability to complete long horizon tasks; that is, tasks over tens of minutes up to hours without losing the thread and actually being able to complete real-world tasks.
All right, so a few details about both of these models, and then I'm going to get into the benchmarks. First, you have extended thinking with both of them, and they are both hybrid models, which means they can give you instant responses with no thinking, or you can turn on thinking for those more complex tasks. And during the thinking, you have tool use, which is of course really nice, but kind of table stakes at this point.
And now I've already been playing around with it and hit my rate limit until 2 p.m. today, which is a few hours away. And really, I only submitted a few prompts. So I think I'm going to have to subscribe to Max and put together a thorough test for you all.
So you can see right here we have Claude 4 Opus, Claude 4 Sonnet. If you click right here on search and tools, you can see the different tools available. You can select the style; you can turn on and off extended thinking. It has web search, drive search, Gmail search, and calendar search. Those are the available tools for now. But they have more deeply integrated the MCP framework into their API. And remember, Anthropic is the company that created the MCP framework that now OpenAI, Microsoft, Google, and so many other companies have adopted.
One unique thing that I really haven't seen elsewhere is that both models can use tools in parallel, which means it can send off requests to multiple tools at the same time. That seems really cool and much more efficient than doing everything sequentially. And it also seems to be much better at handling its own memory. All of this stuff is available in Claude Code, which is also now generally available and has the Claude 4 models available.
During the keynote that live-streamed this morning, the chief product officer of Anthropic spent a lot of time talking about long horizon tasks and how they were able to accomplish this, even giving an example of a company that was using Claude 4 that was able to do a task over 7 hours.
And as part of Claude's new API, they have four new features, including code execution tool, MCP connector, a files API, and the ability to cache prompts for up to 1 hour. Here's what the code execution tool looks like. You simply type in a prompt. Claude will start thinking and write code and of course execute that code. And I believe it needs to be Python for it to execute.
The MCP connector allows you to connect any MCP server to the Claude API. So now your Claude API has access to all of the MCP tools throughout the world. They also have the files API. So giving access to Claude to your local files, specifically your code files, your repositories, just became a lot easier. And then prompt caching. So of course you want to get the most efficient usage; you want to get the cheapest price, and caching is the way to go.
Now, with all of these, you probably can guess where this is going. Claude has basically given up on the chatbots race. It is clear that OpenAI and the major tech companies, Google, Microsoft, and unfortunately not Apple, have all won the chatbot race, the personal assistant race. So now Anthropic has transitioned into being an infrastructure company. They are providing the tools necessary to have the best coding agent. They are building the best agents; they're building the best coding agents, and they are plugging it in to everyone.
Thomas Dohmke, the CEO of GitHub, announced Claude for Sonnet is here. So, it's available in GitHub Copilot, and it's their default option. By the way, I interviewed Thomas at Microsoft Build. I'll drop that interview soon. So be sure to subscribe to this channel so you can get updated when that video drops. It is incredible.
But look at this. In early evaluations, the model soared in agentic scenarios. That's the key. That is what we keep hearing: memory, tools, long horizon tasks, all done by these agents powered by Claude 4, delivering up to a 10% improvement over the previous generation, driven by sharper tool use, tighter instruction following, and stronger coding instincts. And of course, it's also available in Cursor and Windscribe and basically all of the major coding platforms out there.
Now that Claude 4 is especially good at long horizon tasks, has excellent memory, built-in parallel tool usage, it's going to be especially good at pairing with Box AI, and that's the sponsor of today's video. I'm really excited to tell you about them. You're going to be able to build on Box AI using the new Claude 4 models soon.
With Box AI, you can use artificial intelligence to extract key metadata fields from contracts, invoices, financial documents, resumes, and more. And you can automate workflows super easily, and not just metadata. You can ask questions about it; you can really do deep dives into your company's own data. And again, if you're a developer, building on Box AI is easy. It handles the entire RAG pipeline for you. So you don't need to think about vector databases; you don't need to think about chunking. It's just done, and it works. And of course, because it's Box, they have enterprise-level security, governance, and compliance. And with the launch of Claude Code, if you want to use Claude Code with Box SDKs, it could not be easier. Simply give Claude Code links to the Box developer docs, and it just knows how to build with it. Check out Box's blog post about the Claude Code launch to see a demo of them building a back-end contract generation tool using Box Docgen and Claude Code. I'll drop all of the links in the description below. So unlock the power of your documents and data with Box and Box AI. Thanks again to Box for sponsoring this video.
All right, so back to the announcement blog post. Claude Opus 4 and Sonnet 4. By the way, they kind of switched the name, right? It was Claude 3.5 Opus, Claude 3.5 Sonnet, and now it's the opposite way: Claude Opus 4 and Sonnet 4. Anyways, are hybrid models offering two modes: near instant responses and extended thinking for deeper reasoning.
All right, I know you want to see the benchmarks. Benchmarks only mean so much, so take it with a grain of salt, but here they are. So, software engineering bench verified. Yep, Claude 4 is the by far winner. So here's OpenAI Codex 1, which was just announced about a week ago at 72% on the Sweetbench verified, compared to Sonnet 3.7, which was at 62.3%, and with parallel test time compute, 70.3%. But now we have a big jump all the way up to 80.2% with parallel test time compute for Sonnet 4 and 72.5% and 79.4% with parallel test time compute for Opus 4. And by the way, for those of you who weren't sure what parallel test time compute is, it basically just means they sampled a few test time compute solutions to a prompt and chose the best one.
Now, if you're looking at this, you're probably thinking the same thing I am: Did Sonnet just score better than Opus? Well, yeah, it did. And with my initial usage, I actually found Opus to be faster than Sonnet. Now, that's just anecdotal, me using it a couple of times. So I'm going to need to test it a lot more, but it does seem to output code much faster.
Now, here are some more benchmarks. Here's Terminal Bench, Claude Opus 4 winning at 43.2% compared to Sonnet 4, 35%. Here is the 03 model at 30%, GPT-4.1 at 30%, Gemini 2.5 Pro at 25%, which to date, Gemini 2.5 Pro is still my favorite coding model. Here's GPQA Diamond, which is Graduate Level Reasoning. We have Agentic Tool Use doing quite well compared to the other models.
Now you're probably noticing one other thing: Sonnet 3.7 is still doing quite well. I'm going to show you that in a second. We have multilingual Q&A, again getting a nice bump; visual reasoning getting about the same score; and then high school math competition Amy 2025 getting a very nice bump over Claude 3.7.
Now I'm going to pause for a second and show you something. This is a post by John Shonith, and he actually points out the green boxes are around benchmarks which Claude Sonnet 4 did better than Claude Sonnet 3.7. The yellow ones are where it did about the same, and red is where it actually got a decrease in performance, which is kind of nuts. So of all of these benchmarks that they submitted, half actually went down. So I don't really know what to think about that. They're saying it was a huge bump, but the benchmarks don't actually reflect that. And the benchmarks tend to be the nicest view of these models until people start doing the vibe checks of them. So very interesting. And of course, I'm going to be testing it thoroughly. We'll see.
Now, one other thing that they called out during the keynote today is that when Claude 3 came out, it was kind of lazy with coding, and then Claude 3.5 and 3.7 kind of went the other way; it tried too hard and did things it shouldn't and outputted way too much code. And they think they really dialed it in with Claude 4. They, being Anthropic, focused a lot on safety. So we've significantly reduced behavior where the models use shortcuts or loopholes to complete tasks. And of course, they're using the Pokemon example here. Both models are 65% less likely to engage in this behavior than Sonnet 3.7 on agentic tasks that are particularly susceptible to shortcuts or loopholes. Claude Opus 4 also dramatically outperforms all previous models on memory capabilities, which I've mentioned already. But I have said memory for agents is really the key ingredient to making them hyperpersonal. And they called out in the keynote today the hundredth time you use Claude 4 should be much better, much more efficient, and much more concise than the first time you use Claude 4. That's because it's learning and it's understanding what you want; it's developing a shorthand with you as the user. Opus 4 becomes skilled at creating and maintaining memory files to store key information. This unlocks better long-term task awareness, coherence, and performance on agent tasks. And here's the example of the Pokemon benchmark.
They've also introduced thinking summaries for Claude form models that use a smaller model to condense lengthy thought processes. I would love to see the thought process, but you basically see nothing. Now, now here's the key: Users requiring raw chains of thought for advanced prompt engineering can contact sales. So if you want to see the raw chains of thought, you're probably going to have to pay up.
All right, the next big announcement; I touched on it. Let's get into more detail. Claude Code is now generally available. They have new extensions for VS Code and JetBrains that integrate Claude Code directly into your IDE, which is nice. This is a direct competition to all of the coding tools out there. Claude's proposed edits appear in line in your files, streamlining review and tracking with the familiar editor interface, and they're releasing a Claude Code SDK so you can build your own coding agent. So again, they're really building out the infrastructure layer of agentic coding. So Claude Code on GitHub now available, and that's an example of what's possible with the SDK: tag Claude Code on PRs to respond to reviewer feedback, fix CI errors, or modify code. So here's an example. Here's a PR right here. You're going to come into a comment; you're going to tag Claude, "Could you please address this feedback comment," and it's going to jump in and start doing it right away: gather issue and comment context, address the feedback, create a pull request, verify lint, make the tests, and so on. And then you have a PR ready to review.
Now, the chief science officer at Anthropic has said, according to Techmeme, Anthropic's Jared Kaplan says the company stopped investing in chatbots at the end of 2024 and instead focused on improving Claude's ability to do complex tasks. And this makes sense. Claude is just not achieving the mind share necessary to win at the chatbot game. That's ChatGPT; that's Gemini; hopefully Siri in the future. So they gave up on that and went in and focused on agentic capabilities. And you know what? Good for them. Focus is what is required to win.
And how about the pricing? Let's check it out. So Claude 4 Opus, the most intelligent model for complex tasks. It has a 200k context window, which is still relatively small, and you get a 50% discount with batch processing. $15 per million tokens input and $75 per million tokens output. So that's it. I'm going to be testing it out. Expect a testing video soon. If you enjoyed this video, please consider giving a like and subscribe.