NEW Claude 3.7 Sonnet is SUPER EFFECTIVE!

But apparently, they've tested the various Claude models playing Pokémon, which is just phenomenal. Thank you, Anthropic! So you can see the great progress that 3.7 Sonnet makes. 3.5 Sonnet, the two different models, even though good, it seems like they do plateau over time. 3.7 Sonnet keeps going. I have to know more about how they did this; this seems like a fascinating little experiment. And it looks like the 3.0 Sonnet fails to even leave the house in the morning, but don't judge it too harshly; I have those days sometimes myself. We can't all be 3.7 Sonnet.

It looks like 3.7 successfully battled three Pokémon Gym Leaders, the game's bosses, and winning their badges. They're saying here that Claude 3.7 Sonnet is super effective at multiple strategies and questioning previous assumptions. Super effective—did you catch that? Phenomenal job, Anthropic!

So Anthropic announces and releases a couple of new, very exciting features. First of all, it's the Claude 3.7 Sonnet that's available on all Cloud plans, and Claude Coder, a research preview now. Claude, by a lot of people's standards, is the GOAT when it comes to coding, even against competitors like OpenAI's 03 mini High and the new Grok 3 with the think mode. You still will hear a lot of people saying that Claude is still number one when it comes to coding. So while I haven't tested it yet, this certainly seems like either it will put them back in the lead or cement their advantage they've already had. But I'm not going to bury the lead here; let's take a look at what Claude Coder is.

First of all, here's their release video. Should we be doing like a big smile, or uh… no, a big smile is creepy. That's Boris, I'm an engineer; I'm Cat, I'm a product manager. We love seeing what people build with Claude, especially with coding, and we want to make Claude better at coding for everyone. We built some tools, one of which we're sharing today. We're watching Claude Coder as a research preview. Claude Coder is an agented coding tool that lets you work with Claude directly in your terminal. We're going to show you an example of it in action. So we have a project here; it's a Next.js app. Let's open it up in an instance of Claude Code. Now that we've done this, Claude Code has access to all the files in this repository. We don't know much about this codebase; it looks like an app for chatting with a customer support agent. Let's get Claude to help explain this codebase to us. Claude starts by reading the higher-level files, and then it dives in deeper. Now it's going through all the components in the project. Cool. Here's its final analysis. So, say I was asked to replace this left sidebar with a chat history, and I'm also going to add a new chat button. I'm going to ask Claude to help me out here. We haven't specified any files or paths, and Claude's already finding the right files to update by itself. Claude can also show its thinking, and we can see how it's decided to tackle this problem. Claude's asking me if I want to accept these changes. I'll say yeah. Now Claude's updating the navbar, adding a button and icons as well. Next, it's updating the logic to ensure the saving state works correctly. After a bit, Claude completes the task. Here's a summary of what it's done. Let's take a look at the app. So we're seeing a new chat button and new chat history section on the left. Let's check if I can start a new chat while keeping the previous one saved. I'll try out the new chat button too. Great, that's all working now. Let's ask Claude to add some tests to make sure that the features we just added work. Claude's asking for permission to run commands. We'll say yes. Claude is making some changes to run these tests. After getting the results, it continues with its plan until all tests pass. After a few minutes, it looks like we're good to go. Now I'm going to ask Claude to compile the app and see if we get any build errors. Let's see what it finds. Claude identified the build errors and is now fixing them. Then it tries to build again; it'll keep going until it works. Now let's finish everything up by asking Claude to commit its changes and push them to GitHub. Claude creates a summary and a description of our changes, and it'll push the changes to GitHub. That's it; that's an example of what Claude Coder can do. We can't wait for people to start building with it.

So obviously, this is pretty exciting because a lot of these features are stuff that we've kind of talked about in the past. So as coding AIs get better, as they have more and more functionality, so they'll be able to handle more and more of sort of the software engineering tasks—from coding, troubleshooting, actually implementing, launching everything. So I haven't had a chance to check it out yet, but I'm kind of excited about testing it out and seeing what it can do. One thing that I've noticed is since this is their first sort of reasoning model; before that, they have their ant thinking, which would hide sort of the chain of thought from the actual outputs. This is their first sort of real reasoning model, similar to what GPT-4 was, what DeepSeek R1 was, what Grok 3 is, etc. But as you see it thinking, it has all these little words that it uses to kind of signify that—like cooking, schlepping, hurting, clotting, conjuring, ring, and tons more that I that I've noticed. So kind of interesting; definitely has a little bit of a sense of humor there, which is nice.

Interestingly, they're saying that Claude 3.7 Sonnet is has a different philosophy than other reasoning models on the market. Just as humans use a single brain for both quick responses and deep reflection, we believe reasoning should be an integrated capability of frontier models rather than a separate model entirely. So this model is both an ordinary LLM and a reasoning model in one. You can pick when you want the model to answer normally and when you want it to think longer before answering. So interestingly, with OpenAI, you have either one or the other; you don't really kind of get a choice. With Grok 3, it kind of seems to work this way, but from what I understand, they're they're separate models, right, with separate benchmarks; they're called different things. So when you hit "think" on the Grok 3 app, that's when you're engaging the Grok 3 reasoning app, and when you don't engage it, you're using the Grok 3 sort of like the regular base LLM, or the ordinary LLM, as Anthropic puts it. And so in extended thinking mode, that's when it self-reflects before answering, which is why you should use it on things like math, physics, instruction following, coding, and stuff like that—like the harder prompts that require reasoning.

And this, I think, is where things get a little bit interesting. Playing around with Grok 3 over this last weekend, one thing I noticed is if you take the same exact prompt, open three different sort of Grok windows and run them in both, you sometimes get slightly different outputs, or most of the time you get different outputs. Sometimes it'll think for 60 seconds, sometimes it'll think for like 3 minutes. And in a few of the sort of testing things that I've done coding, the three-minute thing, the three-minute version where it thinks much, much longer, is obviously a lot, a lot better, which kind of brings up the question: why wouldn't you be able to kind of choose how long you wanted it to think for? Why don't we have the option of choosing how hard we wanted it to think about any given particular problem? And as far as I'm aware, correct me if I'm wrong in the comments, but as far as I know, this is the first model that will allow you to choose.

So when using Claude 3.7 Sonnet through the API—so this is not for the sort of the app, but through the API—users can also control the budget for thinking. So you can specify how long, get sort of uh, how many tokens it can use to think about the prompt that you gave it, up to the output limit of 128,000 tokens. This allows you to trade off speed and cost for quality of answer. So early testing is showing that Claude is very good at coding. Cursor noted that Claude is once again best-in-class for real-world coding tasks, with significant improvements in areas ranging from handling complex codebases to advanced tool use cognition. The uh, makers of Devon AI found it far better than any other model at planning code changes and handling full-stack updates. Vercel highlighted that it's really good for agentic workflows, and Replit successfully deployed Claude to build sophisticated web apps in Canvas. Avos Claude consistently produced production-ready code with superior design taste and drastically reduced errors.

So here's its results for Software Engineering benchmarks verified. So as you can see, as all the other ones, including OpenAI's 03 mini High, but they're all just right under 50%—so 49%, 48%, etc.—Claude 3.7 Sonnet sits at 62.3%. And then with custom scaffolding—so some sort of software additions to help it run stuff—it sits at 70.3%. Agentic tool use, again, very, very strong, better than the competition. And here we have some benchmarks. So the leftmost is sort of this new Claude model with 64,000 extended thinking. Here's the OpenAI's 03 mini High—so the sort of the competition—as well as Grok 3 Beta with extended thinking. So these are probably the two sort of best competitors um to kind of do an apples-to-apples comparison. So surprisingly, at the graduate-level reasoning, Grok 3 Beta looks like has a slight edge, but I would say all three are kind of just neck and neck, very close. So let's let's say they're at the same level. So very strong on multilingual, competitive on visual reasoning, very high on instruction following, competitive on math problem solving.

So the AIM 2024 definitely seems lower than, for example, Grok and OpenAI's 03 mini High, but it's important to note that they're actually talking about this; they're saying we've optimized less for math and computer science competition problems and focused more towards real-world tasks, which certainly sounds like a good thing. I mean, a lot of people just try to get the high scores on the sort of the benchmarks to kind of show off how good the model is. At the end of the day, what we're really looking forward to is the actual real-world use cases. And they're mentioning Claude Coder, the first agentic coding tool in a limited research preview. Claude Code is an active collaborator that can search and read code, edit files, write and run tests, commit and push code to GitHub, and use command-line tools, keeping you in a loop at every step. Out of the corner of my eye, I saw them—I'm talking about Pokémon gameplay tests. You got to love Anthropic because they have, you know, kind of the standard benchmarks, but also they're like, yeah, but uh, this new model, it's good at some other, you know, perhaps more unexpected tasks—specifically playing Pokémon. Playing Pokémon Red. Pokémon Red is that older game, kind of simple. I believe it also had a Game Boy version, right? So it was a Game Boy version, but there was also some other—there's a couple of different versions, right? So I'm sure you will correct me if I'm wrong, but apparently they've tested the various Claude models playing Pokémon, which is just phenomenal. Thank you, Anthropic! So you can see the great progress that 3.7 Sonnet makes. 3.5 Sonnet, the two different models, even though good, it seems like they do plateau over time. 3.7 Sonnet keeps going. I have to know more about how they did this; this seems like a fascinating little experiment. And it looks like the 3.0 Sonnet fails to even leave the house in the morning, but don't judge it too harshly; I have those days sometimes myself. We can't all be 3.7 Sonnet.

They equipped Claude with basic memory, screen pixel input, and function calls to press buttons and navigate around the screen, allowing it to play Pokémon continuously beyond its usual context limits, sustaining gameplay through tens of thousands of interactions. There better be video or or some sort of a GitHub project we can download. There's got to be. Come on, this can't be it; I want to see this thing play Pokémon. Claude 3.7 Sonnet's improved agentic capabilities helped it advance much further. They're comparing it to poor 3.0 Sonnet, and that failed to leave the house. They're saying here that Claude 3.7 Sonnet is super effective at multiple strategies and questioning previous assumptions. Super effective—did you catch that? Phenomenal job, Anthropic! So it's super effective at trying multiple strategies and questioning previous assumptions, which allows it to improve its own capabilities as it progresses. Pokémon is a fun way to appreciate Claude 3.7 Sonnet's capabilities, but we expect these capabilities to have a real-world impact far beyond playing games. The model's ability to maintain focus and accomplish open-ended goals will help developers build a wide range of state-of-the-art AI agents.

So this, for me, is like one of the sort of the exciting things that I'm waiting for, where you can take one of these um sort of models, give it some, you know, scaffolding or some some function call ability, right, connected to a game—doesn't have to be a complicated game, right? Pokémon Red is is pretty pretty simple—and just you kind of watch it go out there and try to reason through stuff, try to beat the game. I don't know about you; something about that just really kind of fascinates me. Like I can't wait till we have more of that happening. Now there's this video, training AI to play Pokémon with reinforcement learning, and that was just absolutely fascinating, right? But with reinforcement learning, you're basically launching a million of these little things out there, and they just threw sort of trial and error—like try to figure out how to progress through the game—and that, of course, is a very interesting, fascinating process, but it it's different from something that's like a very smart reasoning large language model. It's not going to randomly try every single thing before it does even one thing; it's going to reason through what to do next. And you can see that sort of thought process; it's going to come up with a plan, a mental model, and then execute on it. There's already a thread on the Singularity subreddit, you know, complaining about not having footage of this, right? The benchmark we deserve. Agreed. Okay, Anthropic, we need that Pokémon thing. Please help us out. Although it's smart that they didn't release on the same day because they probably are aware that people like me would get distracted by it and not be testing the actual new release of Claude, but rather just wanted to play Pokémon. So well played, Anthropic, well played.

So here they're talking about serial and parallel test-time compute scaling, which again, as far as I know, this is the first model that allows us to choose how much it's going to think about stuff. So we can actually kind of like run our own experiments and maybe even find sort of that sweet spot for uh how long it takes to think about something. So here's the AIM 2024, and this model's accuracy on that test. So this is a high-level sort of um mathematical competition aimed at sort of the high school students in America, but like kind of like like the top of the sort of the math competitions in that area. And again, as they mentioned, they're not trying to like nail these sort of competitive tests; they're more going for like real-world use cases. But I think interestingly, this kind of gives you a very great visual representation of what it means to give it more of a thinking budget, right? So that test-time compute, as you can see here, if you limit it to 2,000, I mean it does rather poorly—like 32%, right—and as we scale it up, right, at 64,000 budget, it's over there at what is that, 56%, 57%, something like that. So if you have a hard, let's say, coding problem that it needs to like really think through it to make sure that there doesn't have some sort of like logical errors, just give it a little bit more budget, crank up that sort of a test-time compute, and see how well it performs. Some people in the previous video sometimes complain that some of these charts are a little bit maybe misleading, so all of these charts in general are going to be log scale. So when we're talking about sort of the the quality of the output versus the sort of the the the compute—either test-time compute or you know, even when we're talking about like when we're training the models, the compute used to for training budgets—like usually you're going to see those in in log scale. So you're getting better answers, better quality for exponentially more cost in compute.

Getting back to serial and parallel test-time compute: when Claude 3.7 Sonnet is using its extended thinking capability, it could be described as benefiting from serial test-time compute; that is, it uses multiple sequential reasoning steps before producing its final output, adding more computational resources as it goes. In general, this improves its performance in a predictable way; its accuracy on, for example, math questions improves logarithmically with the number of thinking tokens that it's allowed to sample. Another way of improving the accuracy is using parallel test-time compute, right? So we do this by sort of running, let's say, the same prompt multiple times, or as they say here, sampling multiple independent thought processes and then trying to figure out which one is the best one, right? There's a number of ways that we can select, you know, the best one; you know, a simple one is just uh majority or consensus voting. I always describe it as if you're in a foreign city, right, you're asking for directions to get somewhere, right? So if you ask one person, their answer might be wrong. If you ask 10 different people, right, and you notice that like eight of them say the same thing and then two of them, you know, kind of make up some sort of other directions for you to follow, well, it's very likely that the the eight people that independently kind of confirmed that these are the sort of directions to get where you're going, they're probably right, and those two other ones, those are probably like the wrong answers. So we can kind of apply the same thinking here, right? So and of course, as you can imagine, so if we're testing these coding models, you know, there's in various benchmarks you have these ideas of like, for example, pass-at-one or pass-at-ten. Pass-at-one means you ask it once, you sample it once for the answer, and this number, like let's say this 17%—that's how often it gets the right answer, you know, on one try. Pass-at-ten is where we do 10 separate samples, 10 separate prompts, let it kind of answer 10 separate times, and then see if it gets the best answer in that case. So obviously, right, this usually is going to go be much higher, right, because it's more likely to get the correct answer given 10 sort of independent tries. And this is important for coding because sometimes we need the thing that's pass-at-one, right? So if we want a code that does something, we don't want it giving us like, oh, here's 20 different ways it might work, right? Now we want one that will work. However, and we've seen sort of a lot of research papers in the past kind of written about this, is that there are some examples, certain situations where having what you can think of as like multiple models kind of uh all kind of suggesting different things or sampling several different sort of independent thought processes from these models does help. For example, with troubleshooting, where you you're trying to fix a specific thing where you know whether it fixed it or not, right, it might be okay for the thing to try 10 different things—all right, thing A didn't work, let's try thing B, thing B didn't work, let's try thing, let's try thing C. Okay, C fixed it; that's the solution.

And here they're mentioning sort of another way to kind of take advantage of this sort of parallel test-time compute; they're saying another is using another language model. So, for example, it could just be a second copy of Claude and asked to check its work, or a learned scoring function and pick what it thinks is best. Strategies like this, along with similar work, have been reported in the evaluation results of several other AI models. And it's funny because here each word is its own sort of link that they provide to different uh to different studies by different companies, which is interesting. This always feels to me like like it's like clapping when you're making a point—like it's the results in several other models, you know, each one being its own paper—is that just me? And they link Google Gemini, this is the recent paper we covered from OpenAI, competitive programming with large reasoning models, and literally kind of like what I was referring to. So here's its performance when you're able to just when the model is able to do 50 random submissions versus 50 handcrafted submissions that they use certain sort of a certain system to kind of like handpick it versus just give it a chance to do 10,000 submissions, right? So as you can see here, if we just let it run loose and provide 10,000 different answers, the sort of the performance just skyrockets. They also mention this paper, Grok 3 Beta, the age of reasoning agents, the OpenAI 03 mini. On this channel, while back I covered ChatDev, where you have a software engineering sort of like company, right? Each one of these little agents, they at the time they were their own version of ChatGPT 3.5, and you would go in, prompt to create a game or some tool that you wanted them to use, and so these three, again, separate sort of instances of ChatGPT 3.5 will go to work designing how to do it, then, you know, three or four different versions of them would start coding based on this design. There would also be this Chief Technical Officer that would like kind of push them into line, make sure they don't get distracted, right? Then it would go to testing where they would like iron out the bugs and stuff, and I saw this process was one of the things they asked where it would go from coding to testing and back and forth, back and forth multiple times through multiple iterations until they were able to fix a bug, and then they would like ship the product, right? They would kick it over to documenting that would write up a manual, and then it would get published or you know, they would tell you it's done. But this kind of blew my mind because again, one ChatGPT 3.5, if you had asked it to design that thing, there's no way it would have been able to do that; it would put out some garbage code that just would not work. But you copy and paste them 12 times, right? You you tell three of them they're in design and three of them are in coding, and then you tell one of them they're like the Chief Technical Officer, where you give them roles, and then all of a sudden it emerges with something better. That was so weird to see cuz this was a while ago; this this was before this was more like common knowledge at the time. I remember sitting there thinking like, how does this make sense? Like you have one model that can't do it, but you just string a bunch of them together, and all of a sudden it's like smarter. How's that possible? So I feel like now a lot of this has been sort of replicated with the reasoning model. So you can think of it as sort of designing, right? So when it's thinking about how to code up something, right, it's this department sort of designing how to do it, then the sort of when it starts to actually code, that's kind of like the coding, but we we didn't really have testing yet. But now with, you know, the coding agent with Anthropic's Claude for coding as well as some of the the other software that's on the market, like we're we're kind of like replicating this system. And again, we had prototypes like ChatDev, you know, we had it like a year plus ago, but the idea of parallel test-time compute to me seems like this, the idea of this, it's something similar, right? So when you're designing the game or you're coding the game, you don't have one one ChatGPT 3.5 doing it; you have three. Why? Well, we're seeing much better accuracy; their ability to reason and get through stuff improves drastically when either they're working together and talking back and forth—we've seen studies on that—or just by, you know, sampling the answer from 10 different ones and going with that majority consensus, whatever.

So Anthropic continues: we've achieved striking improvements using parallel test-time compute scaling on the GPT-3 evaluations, a commonly used set of challenging questions on biology, chemistry, and physics. Using the equivalent compute of 256 independent samples, a learned scoring model, and a maximum 64k token thinking budget, Claude 3.7 achieved a GPT-3 QA score of 84.7%, including a physics subscore of 96.5%, and benefits from continued scaling beyond the limits of majority vote. This might uh need its own video to cover not just what they're saying here but kind of like all the research leading up to this. Okay, so here they're talking about that sort of the physics, biology, chemistry—like the expert-level questions, right, the reasoning on those tests—and so here you have sort of the accuracy uh from from 0 to 100, right, and N is sort of the number of samples or the number of tries that the model gets, right? And this purple line, that's pass-at-N, so meaning that, you know, if N is, you know, 32, we ask it to answer the question 32 times; if even once it gets it right somewhere in those 32, then it passes. So as long as it gets it right at least once, it passes. So it's kind of like throw stuff at the wall, see what sticks. So not the best way of scoring it, but just kind of gives us an idea. Now the gray, or whatever color this is here, that's the majority-at-N, right? So we ask it, you know, for let's say 10 to answer 10 times; if six of those answers are one thing and the other ones are something else, so we vote and we say so we say, Okay, so the six times that answer this, that's the majority; let's go with that answer. And as you can see here, that tends to work, and that might work better than um just giving it one shot, but but there's there's obviously plateaus, there's problems. And then they're talking about a this new thing that they're talking about, the the scoring model. So they came up with some sort of a system to try to like score the answers somehow to see which ones might be the correct ones. And as you can see here, the blue line in general sort of is better than the than the majority-at-N, right? So they found a scoring model that tends to work better, and seemingly for every single one of these sort of subcategories, I mean it's a little bit funky here, but then uh after a certain point, still gets better. And they're saying that this parallel test-time compute scaling, it's not available in our newly deployed model, but we're continuing to research these methods for the future. Obviously, something like this would be very expensive because you're giving it a high thinking budget, and then you just basically copy-pasting that prompt, like you're running it 256 independent times, right? So it cost you a dollar to run a prompt; this would just cost you $256. What's interesting is that the whole point—not the whole point, but a kind of large point—of this paper by OpenAI, competitive programming with large reasoning models, the big idea was like if you create a custom model—like they they took the GPT-4, and they took the GPT-4—II was specifically made to sort of like be good at the IOI benchmark, and then they would sample 10,000 solutions from that model for each task, right? 10,000. And they they would use a lot of these like really complicated, fancy sort of statistical approaches to try to pull the the 50 best ones, and that was how much you were allowed to submit for the competition—50, right? Here you would be able to like improve the results of that model by adding that sort of scaffolding, adding that sort of like GPT-10,000 results, filter them down to the best 50, then submit those 50, right? And yeah, of course, you can like even improve that even further by, you know, submitting the whole 10,000 submissions, right? But what a lot of people kind of took out of this paper is that, you know, yes, you can take this GPT-4 model and do all this complicated sort of statistical stuff and scaffolding and and prove it, or you can just train a bigger model, like the next sort of the next sort of a larger model with more data, etc., and and that's going to be much, much better. And somebody—I forgot who, I apologize—so posted this little meme, right? Statistical learning, right? So you do all this like super complicated stuff, right? Or like or like in a neural network, just stack more layers, right? So you know, green arrow goes up.

So we can start using Claude 3.7 Sonnet at cloud.anthropic.com or on the API. Remember the sort of where you can pick the compute budget for test-time compute; that's only through the API. So check it out; let me know what you think. I'm going to be doing a lot of testing to see how well it works, but I do want to encourage Anthropic to be on the right side of history for this one. I encourage them to do the right thing. I think I speak for the masses when I say we want to see this thing play. You can't just post a chart and then nothing else; we need more. With that said, if you made it this far, thank you so much for watching. My name is Wes Rth, and I'll see you next time. And don't forget, you gotta catch them all!