Transcription
OpenAI just dropped 03 mini. Just when I thought I would be able to take just a little break, OpenAI said no way. This is the Deep Seek effect. OpenAI most likely pulled up the release date of 03 mini, and you know what? We all benefit.
Let me show you what it's all about because it is pretty incredible. They are releasing 03 mini today to everybody, including free users, and it comes in three different flavors: low, medium, and high, depending on how much thinking you want the model to do based on your task.
It's already available in the API, and as it says here, it already supports function calling, structured outputs, and developer messages. So, it's production-ready right out of the gate. You know what? I already have access to it, Chat GPT 03 mini, but we'll do a couple of tests in a moment.
Let me tell you a little bit more about it. Remember, these thinking models are especially good at STEM: Science, Technology, Engineering, and Math. The reason is because of what we've been talking about on this channel: reinforcement learning. That is when you can have a very well-defined reward function for these models.
To do that, you need to know the answer to the question. So, if it's an open-ended prompt like "write me a poem," there's no definitive answer to that. But if it's "what's 2 plus 2?" well, that only has one answer.
I also think it's really cool that they're giving three levels of thinking: low, medium, and high. So, you don't have to make it think for a long time, which means you can actually save money, make it more efficient, and lower latency.
03 mini is rolling out in the chat completions API, assistance API, and batch API starting today to select developers in API usage tiers 3 through 5. Chat GPT Plus, Team, and Pro users can access 03 mini starting today. Enterprise access is coming in a week.
03 mini will replace OpenAI 01 mini in the model picker. We're tripling the rate limit for Plus and Team users from 50 messages per day with 01 mini to 150 messages per day with 03 mini. If you're a free plan user, you can use it too. Starting today, free plan users can also try OpenAI 03 mini by selecting reason in the message composer or by regenerating a response.
Another cool thing: it comes with search right out of the box. All you have to do is select this little search the web button right here in the composer, and you can have search plus incredible reasoning capabilities.
03 mini does not support vision capabilities yet, so developers should continue using 01 for visual reasoning tasks. They keep re-emphasizing that 01 remains our broader general knowledge reasoning model, but 03 mini provides a specialized alternative for technical domains requiring precision and speed.
All right, let's look at some of the benchmarks. 03 mini has been optimized for STEM. 03 mini with medium reasoning effort matches 01's performance in math.
So, here's the A224 competition math, and as we can see in yellow, these are the 03 mini scores: low, medium, and high. As you can see, selecting low, medium, and high makes a huge difference on these complex math problems.
Here's 01 and 03 mini. Low doesn't perform anywhere near it. 03 mini medium is quite close, and 03 mini high beats it. So, you really should think about which setting you're going to use as a developer with 03 mini.
Now, what I think would be really cool is if it kind of just decided for you whether to use low, medium, or high. There should be a way to categorize prompts based on how complex the model thinks it's going to be.
Here's GPT QA PhD level science questions, and again, not as much of a difference between the three scores, but still a pretty substantial difference. An 01 preview actually does beat all three of the 03 mini scores here, as does 01, but 03 mini high does come close: 77.2 versus 78 and 78.3.
Here's Frontier math, some of the hardest math questions in the entire world—math questions so hard that the best human mathematicians in the world might hope just to get a couple of them correct.
So, here's 01 with a 5.5% at pass one, 01 mini 5.8, and 03 high 9.2. Now, moving down the chain, we have pass at 04 and pass in 08. At the very highest setting, at the eight pass, we have 03 mini high 20%. Really, really impressive.
Here's a competition coding benchmark, Codeforces. This is an ELO rating, so this seems to be where 03 mini really excels. Here's 01 preview at 1258, here's 01 at 11891, and the 03 mini low comes in just below the 01 score.
But look at this: 03 mini high beats all of the other ones—01 preview, 01, and 01 mini. Now, what's not listed here is 03 Pro or just 03 standard, but it's clear that 03 mini is meant for coding.
Just imagine how much better your coding experience with AI is going to be with this model. Here's Sweet Bench verified, again another coding benchmark. These are real-world coding challenges.
Here we go: 01 preview, 01, 41, and 48 respectively, then 03 mini at 40, all the way to 03 mini high at nearly 50% accuracy. So, really, really good, beating 01.
Now, here's some human preference evaluations. We have the win rate on the y-axis here. Here's the 01 mini Chat GPT baseline right there, and here we have 03 mini up here, much better.
Now, this is for STEM. Now, here's for non-STEM. As you could tell, 03 mini is really excelling at STEM questions, and the reason for that is exactly what I mentioned at the beginning of the video.
STEM questions have defined answers, and that way they can use reinforcement learning to make the model better versus open-ended questions, which they can't.
03 mini and all the mini models are really about speed. Listen to this: 03 mini delivers faster performance and improved efficiency. Beyond STEM evaluations highlighted above, it demonstrates superior results in additional math and factuality evaluations.
With medium reasoning effort in A/B testing, 03 mini delivered responses 24% faster than 01 mini, with an average response time of 7.7 seconds compared to 10.16.
Here's the latency on the bottom in yellow: average time to first token, 03 mini in milliseconds, 75, 500 right there, 01 mini all the way up above 10,000. They had to be very aggressive on their pricing—another benefit of the Deep Seek phenomenon.
Let's take a look at the pricing. So, for 03 mini, we have $1.10 per million input tokens, 55 cents for a million cached input tokens, and $4.40 per million output tokens.
But how does that actually compare with Deep Seek? So, this is Deep Seek. If you want to use the actual hosted Deep Seek version, Deep Seek Reasoner right there: 55 cents per million input tokens with a cache miss, 14 with a cache hit.
So, cached a million output tokens, $2.19. So, basically, extremely comparable between 03 mini and Deep Seek R1.
All right, so enough talk. Let's try a few things. The first thing I want to do is try coding, and I'm just going to test a few things right now, but I'm going to make a whole video doing a full test with my usual rubric.
Now, the first thing I want to do is write the game Snake in Python. What I'm going to be looking for and what I want to show you is the speed. So, I'm not going to touch it. I'm not going to cut away. I'm just going to play it in full. Here we go, reasoning.
So, I'm going to click, and we can see some of the output here. Obviously, and unfortunately, we don't actually get the internal monologue; we just get a summary. Now there it is. Look how fast it output that!
So, to build the game Snake with thinking, 6 seconds. Now, let's test it really quick. So, I pasted the code in. Let's see if it works. All right, here we go. We have a snake going. Let's see if it can eat the food, and it does! It seems to work just fine.
So, that was 6 seconds just like that. In this version, it can actually go through the wall, so there's no real right answer to that. Some games can go through the wall; some games can't. Either way, it's fine, but it does quit if we run into ourselves.
So now, let me show you 03 mini high. Same question, right? The game Snake in Python. I'm not going to touch it, not going to edit around it. Let's see how fast it is.
Okay, so here we go. We get the summary of the internal monologue here. The thinking is definitely already taking a little bit longer, but still lightning fast. Look at that!
Now, it looks to be about the same amount of code. So, this one was 9 seconds, and for the previous version, it was 57 lines of code. Let's see how many lines of code it is for this. This is nearly 103 lines of code.
Let's see if the game's any better. So, let's hit play, and there we go. So, it's definitely moving faster. If we run to the wall this time, it actually does end the game, and we get the "Press Q to quit, press C to play again."
So, a little bit of a different game, but overall basically the same. As we saw, 03 mini is especially good at coding. Just when 03 mini was announced, Wind Surf and Cursor both back-to-back in my timelines have 03 mini available in these AI coding platforms.
So, coding is just going to get that much easier, that much more productive. All right, so really, really impressive. Congrats to OpenAI for the great release. I'm going to be testing it like crazy.
I'll make another video just to show you the full testing using my rubric. And you know what? Congratulations to all of us because we have just had so many AI goodies over these last couple of weeks.
It's such an exciting time to be alive right now. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.