Transcription
GPT-4.1 was just announced and released. We have a brand new model. It is better in almost every single way than GPT-4.0, and it's significantly cheaper. It is only available via the API, and there's a reason for that. This model was specifically tailored for developers to use. GPT-4.5 is actually getting deprecated in favor of 4.1. More on that later.
And so there's a bunch of exciting announcements. I'm going to go over them all. I'm going to show you some demos, and let's get into it. Here's the blog post introducing GPT-4.1 in the API, and I found it very interesting that it's literally only available in the API. Obviously, you can wrap it yourself and treat it as a chatbot, but it is not available in the ChatGPT interface, at least not yet.
So GPT-4.1 is a family of three models: we have 4.1, 4.1 mini, and, for the first time in OpenAI's history, 4.1 nano. Extremely fast, extremely inexpensive, and 1 million tokens of context. This is huge. One of the biggest drawbacks of OpenAI's models has been their limited context window as compared to their frontier competitors, but now they have at least taken a big leap in the direction of large context: 1 million tokens.
Now, these 4.1 models, as it says right here, outperform GPT-4.0 and 4.0 mini across the board, with major gains in coding and instruction following. And it's not just that they gave them very large context windows; they're actually really good at utilizing those very large context windows, as I will show you. They also mentioned in this blog post they have a refresh knowledge cutoff of June 2024. It's always interesting to see when those knowledge cut-offs are because it gives you a hint as to when the training basically ended.
Let's take a look at some benchmarks for coding. GPT-4.1 scores a 54.6 on the SWE Verified, improving 21.4% over GPT-4.0 and 26.6% over GPT-4.5, which is crazy, and continues their absolutely horrific naming scheme. 4.1 is now better than 4.5, and I'm kind of sad what's going on with 4.5, but again, I'll talk more about that in a minute. It is, according to them, the leading model for coding. And they actually brought in a special guest during the live stream, Verun, the CEO of Windscribe, to talk about that, and I'll show that video in a little bit.
Then, on instruction following scales, Multi-Challenge Benchmark, a measure of instruction following by ability, GPT-4.1 scores 38.3, a 10.5% increase over GPT-4.0. And not only that, OpenAI is actually releasing their own instruction following benchmark, and I'll give you the scores on that as well. But first, long context on VideoMME, which is a benchmark for multimodal long context understanding—essentially loading up videos—and yes, this is a multi-modal model. GPT-4.1 sets a new state-of-the-art result, scoring 72% on the long no-subtitles category, a 6.7% improvement over GPT-4.0.
So this is, in fact, the natural successor of GPT-4.0. It wasn't GPT-4.5, and I have some thoughts about that. I've already mentioned that they're going to be deprecating GPT-4.5. And so why? What is actually happening? Well, according to Kevin Wheel, the Chief Product Officer of OpenAI, they simply need the GPUs. They have all of these GPUs that were required to power GPT-4.5, which is a massive model; it was slow, and now they need it for this API-based model that is much more—I think the best word is—usable. And so soon GPT-4.5 will be deprecated. And I don't actually think that's going to be forever. They have this incredible model, this large model, and I actually suspect it's a model that they're going to continue to build on; they're going to continue to fine-tune; they're going to bring the prices down, make it more efficient. I just think they shipped it too early, and it was too expensive and too slow. And there is a GPU shortage still, years into this generative AI boom. We still have GPU shortages, which is absolutely crazy to think about because the amount of demand that these companies must be experiencing to have GPU shortages, even though so much money has gone into purchasing and producing GPUs, is just mind-blowing to think about.
Now, back to 4.1. What's interesting is they specifically say they trained this model to have real-world utility, and they worked with the development community. They worked with Windscribe, obviously. I'm not sure if they worked with Cursor and Replit and other companies that also provide really cool vibe coding tools, but they definitely worked with the developer community to create the best possible models.
First, let's talk about latency, the speed at which these models deliver results. Now, one of my pet peeves is graphs with no labels whatsoever, and that's essentially what we have here, but I'm still going to go over it briefly. On the y-axis, we have intelligence on the multilingual MMLU. On the x-axis, we have latency. We don't know what the latency is because it's not labeled here, but fine, relatively speaking. GPT-4.1 nano has lower intelligence than the other models but is also much faster. But they even said this in the live stream: GPT-4.1 mini is the true standout of this family of models. Look at that: a huge improvement in intelligence versus GPT-4 mini, yet the latency is about the same. And we have GPT-4.1 up here above GPT-4.0, so definitely higher intelligence, same latency. And listen to this: GPT-4.1 mini is a significant leap in small model performance, even beating GPT-4.0 in many benchmarks. It matches or exceeds 4.0 in intelligence eval while reducing latency by nearly half and reducing cost by 83%.
Now, the cost is especially important when you're talking about giving an API to a developer because then you are not limited by how fast a human can type out a prompt and get the response; it is programmatic, and so the potential cost of using these models is much, much higher—exponentially higher. Now, the cool thing is every one of these models has a 1 million token context window. And speaking of price, they're not charging extra for that long context. A lot of their competitors, which they pointed out, will charge you extra to use that really long context, but that's not the case with GPT-4.1. You simply pay for the tokens you use consistently across the token context window.
And so, with long context, with multimodal understanding, with its incredible ability to read charts and understand documents, of course, the enterprise use case is going to benefit the most from this. And that's why I want to show you some of the benchmarks that Box, the sponsor of today's video, put together, and some of the information that they were able to gather as an early partner on 4.1. So here's the blog post; I'll drop this down below: First look: GPT-4.1 now available with Box AI Studio. All right, so check out this benchmark. This is the Box AI Enterprise Eval. We have in purple GPT-4.1 versus GPT-4.0 in yellow. So this benchmark tests the model's ability to extract certain pieces of data from different types of documents. So on the left, we have secure data escrow. 4.1 nearly doubles at a perfect score. We have insurance documentation, a huge leap: warranty duration, liquidated damages, non-disparagement clauses. So the ability to extract the right thing from these complex documents, and a lot of documents all at once, really seems to be 4.1's best feature.
All right, so let me show you that in action. Here's a demo. So we have Nvidia's earnings; this is Box AI right here. We're going to ask it, "Please list the key trends from the earnings call." So key trends include record revenue growth; it gives you specific numbers; it gives you comparisons to previous quarters, previous years, data center dominance, and Blackwell ramp AI scaling laws and compute demand. So very successfully pulling information out of this complex document. Check out GPT-4.1; give it a try in Box AI Studio. They've been a fantastic partner, so please check them out. I'll drop a link to them in the description below. And if you want to try out 4.1, email them at aabs@box.com. Thanks again to Box.
So GPT-4.1 nano: 1 million token context window, scores 80.1% on MMLU, 50.3% on GPT-QA, and 9.8% on ADER Polyglot coding—even higher than GPT-4.0 mini. So this is the nano version, the smaller version. It's ideal for tasks like classification or autocompletion. This is the workhorse of this new 4.1 family of models. And here's the part that I love: these improvements in instruction following reliability and long context comprehension also make the GPT-4.1 models considerably more effective at powering agents. And so, of course, there are two things that come to mind with that: anything where you're building agentic systems, like with Crew AI or what I've been doing more recently, vibe coding—that is essentially taking a coding model like this and wrapping it in a gentic behavior through Windscribe, through Cursor, through Replit—and it is very exciting to see this field emerge.
Now, here's something interesting which I didn't really understand: 4.1 will only be available via the API. In ChatGPT, many of the improvements in instruction following, coding, and intelligence have been gradually incorporated into the latest version of 4, and we will continue to incorporate more with future releases. So they really are naming things in the most complex way and just making it difficult for themselves, but whatever.
Now, here's the crazy part: we will begin deprecating GPT-4.5 preview in the API as 4.1 offers improved or similar performance on many key capabilities at a much lower cost and latency. 4.5 preview will be turned off in 3 months, on July 14th, 2025, to allow time for developers to transition. Now, that is, in my opinion, really bad. Imagine you're a developer, and you get 4.5 what—a month ago, six weeks ago—and then you are already notified that you're going to have to migrate off of it. That is a huge pain in the butt. But they go on to remind us: GPT-4.5 was introduced as a research preview to explore and experiment with a large compute-intensive model, and we've learned a lot from developer feedback. Now, again, I don't really think 4.5 is going away. It is a massive model; it'll probably be used to distill smaller models, and it probably was used to distill 4.1. And so, as they learn how to get more out of 4.5, I suspect we will see it again in the future.
So let's look at some specific benchmarks. Here is SWEBench Verified Accuracy: GPT-4.1 beating 03 mini high. That is crazy. That is a full-thinking model, GPT-03 mini high, using the maximum amount of thinking and getting beat by a non-thinking model, and it's much faster and it's much cheaper. And look down here: GPT-4.1 mini as compared to 4.0 mini, a substantial improvement: 9% on SWEBench Verified compared to 24%. That is a multiple-time improvement. And just for those of you who may have forgotten, SWEBench Verified is a benchmark in which the model is given a GitHub repo, a description of a problem, and must generate a patch for the code. So this is the most real-world coding benchmark there really is. And they also focused on code diffs, and so what that means is the model will not just rewrite an entire file, but it's actually really good at rewriting only certain portions—so basically editing smaller portions of it—which, of course, saves time and saves money because instead of having to output the entire file again, which the other models tended to do more often, it will just edit the specific lines of code that need editing and leave everything else the same—aka the diff.
So here's the ADER Polyglot benchmark: 4.1 whole and diff, GPT-4.0 whole and diff, 01 high whole and diff. And so you can see across the board there is a specific accuracy reduction when doing a diff instead of the whole file. However, with GPT-4.1, there's actually a slight improvement; it's basically the same, but a 1% improvement on creating a diff versus creating a whole file. And they also point out that GPT-4.1 is actually much better at front-end coding. And so, rather than me talking about it, let me just show you the demo from the live stream. But there's also kind of the intangibles of when you're using a model—you know, when you're creating a front end—is it functional? Is it beautiful? Does it nail the mark? And so for that, we have a little example of a flashcard app I've been making; you're learning Hindi, yeah, working on it. Uh, and so I've got, you know, a prompt here; it's pretty complicated. I'm asking for this app pretty specifically; I want a nice 3D animation when you click on the flashcard. Um, and so when I give this prompt to GPT-4.0, this is what I get. Um, it follows some of the instructions, and and some of the app is functional, um, but you know, we've really trained GPT-4.1 to do better, and that model, you can see it looks way better; it's discovered colors; uh, it can also do the 3D animation. Um, so we think you're really going to like this improvement to front-end coding. Now, obviously, they still have a lot of work to do in terms of beautifification of the front end, but I think this is a big improvement.
All right, so during the live stream, they brought out the CEO of Windscribe to talk about the improvement of 4.1; they've had it for a little while, and so here it is: GPT-4.1 scores 60% higher than GPT-4.0 on Windscribe's internal coding benchmark, which correlates strongly with how often code changes are accepted on the first review. Their users noted that it was 30% more efficient in tool calling and about 50% less likely to repeat unnecessary edits or read code in overly narrow incremental steps. This means faster iteration and smoother workflows. That is great. We also have Kodo. They tested 4.1 head-to-head against other leading models, and what they found across 200 meaningful real-world pull requests with the same prompts and conditions: 4.1 produced the better suggestion in 55% of cases. Notably, 4.1 excels at both precision—knowing when not to make suggestions—and comprehensiveness—providing thorough analysis when warranted while maintaining focus on truly critical issues. And Verun specifically mentioned that 4.1 is much less verbose than other models. He said, "Other models tend to get very blabby when they're giving you their output in vibe coding scenarios, but 4.1 doesn't do that." That is very nice to hear. And they also gave a demo of instruction following during the live stream, and this is incredibly, incredibly important. Let me just play that video for you now.
So just like coding, we have made the model way better at instruction following. It now strictly follows all the instructions that you provided. So using all the feedback that we received, we created this internal instruction following eval where it mimics all how an API developer uses our model. So each sample in the eval contains a complex set of instructions where each instruction belongs to one of several categories, like formatting, uh, ranking ordered instructions, overconfidence, and so on. And collectively, that sample is given a difficulty rating from like easy, medium, and hard. And we see that this model does really well across all those axes and on difficulty levels as well. So here you can see the hard subset eval results, and this model is so much better than the previous 4.0 model. So what's an example of like a really hard set of of instructions to follow? Yeah, so let's say you're building a trip uh planning application, and you give it instructions like, "Make sure you receive all the info from the user before answering them, and when you answer with the trip itinerary, make sure it's in a table format; it contains five rows, three columns; the columns are formatted in a certain way." Yeah, I I don't know about you all; I remember all the times that you have to—you learn these tricks in prompting where you're like, "No, no, no, you really need to make this a table, not a list; trust me, my boss is going to be super mad at me if you don't get this right." So hopefully no more of that. Yeah, and people actually were doing that just so that they could get the model to follow an instruction. None of that is eaten out; the model follows all your instructions to the tea, and it's yeah, it does really well. Okay. And now look at the benchmark. This is internal OpenAI instructions following eval accuracy, hard subset. We have 4.1 at 49% as compared to 4.0 at 29%. And 49% is comparable, although trailing by a bit of 01, 03 mini, and 4.5. And we can see 4.1 mini is actually substantially better than 4.1 nano and even 4.0 mini. And for multi-turn instructions, we also got a substantial improvement. Check this out: we have GPT-4.1 here, which is a substantial improvement over 4.0, but it obviously does trail behind the thinking models.
Now here is needle in a haystack. If you're going to have a massive context window—a million token context window—being able to actually utilize that context window effectively is just as important as having it to begin with. And what do we see here? We have a perfect score: successful retrieval up to a million tokens, and 100% successful retrieval. And so they actually did a really cool demo during the live stream in which they placed a single line of non-log code into a massive log file and said, "Find the line that's different from the others." They didn't even specify what the line was. Let me show you that demo now.
To test the log file that I'm about to upload, um, that file is NASA's server request response log file from 1995 August. So in this log file, you can see the client name on the left that made the request to NASA servers; you see the timestamp, the resource that was accessed, and the HTTP response code. This is a long file that contains a lot of log lines, and you can see on the left that this is about 450,000 tokens of uh content here. Nice. So you just couldn't use this with our past models. Yeah, this wasn't possible. Now what I've done is I've snuck in a line that is not actually an HTTP request response. Let's see if it can find it. Okay, so it's a little needle in the haystack, except in this case you don't even tell it what the needle looks like; it's just figure out what's different and tell me exactly. So it's going to sift through the whole file, do some pattern matching to see how all the log lines look like, and then try to see if there's one that does not look like the others. Okay, there is a line that it has spit out that does not look like an HTTP request line to me; it does not. Okay, uh, let's see if this line is indeed in the log file that we uploaded. So I'm going to copy this keyword. Nice. There it is. Great. So it was able to find this line that has been snuck in into this 450,000 token log file. That is like very hard to find, so did pretty well.
All right, so let me show you just a couple more benchmarks. We have MMLU accuracy, which is answering questions on models, charts, multimodal, certainly. So check this out: 4.1 beats 4.0 by about 6%; it's still trailing 01 high, and it's about equal with 4.5, but of course much less expensive. And we have 4.1 mini right here, just dominating 4.1 nano and 4.0 mini. We have MathVista here; it is again 4.1 beating 4.0, matching 01 and 4.5. That is the theme we're seeing; it's basically much more efficient than the thinking models in 4.5 but with similar quality. So they were really able to do a lot with this model. Here's a chart-naive reasoning accuracy: 4.1, same thing, on par with 01 and 4.5.
All right, so let's talk about pricing. Here is GPT-4.1: $2 input per million tokens, cached input 50, and output $8 for a blended total price of $1.84 per million. Now, 4.1 mini: 40 cents for input, 10 cents for cached input, $1.60 for output, blended total 42. I really think 4.1 mini is going to be their true winner of the announcements today. And we have 4.1 nano: 10 cents input, 2.5 cents cached input, and 40 cents for output for a blended total of 12 cents. These are extremely cheap models. So that's it. There's a bunch more benchmarks that they have in the blog post; I'll drop the link to that in the description below.