Transcription
Today's the day! It's finally here: O3 Mini, the first model from OpenAI that's actually reasonably priced and also reasonably good. There's a reason that this all came out, and yes, I'm using "reason" a bunch because it's a reasoning model. I can't help myself.
But the thing that made this release as interesting as it is isn't actually OpenAI's work at all; it's another company, Deep Seek, out of China. It really seems like they built this model to compete with what's going on over in China, and it shows with both the pricing and the performance you can get, but also with some of the weird quirks in how they implemented it.
This is going to be a real deep dive where I'm going to go from showing the performance of the models, the price of the models, how these things compare, as well as the strengths and weaknesses when you're using them for real. I spent the whole day playing with O3 Mini, and it's been a wild ride. I'm super excited to show you guys everything I've learned.
But first, a quick word from today's sponsor. Today's sponsor is Raggy, and you AI bros definitely want to hear this one because they made it way easier than it's ever been to connect basically any service to your AI apps. So if you have data in something like, I don't know, Google Drive, Slack, Salesforce, Confluence, or even Notion, they'll handle everything kind of Plaid style, where you can sign in with any of those apps in your app.
Now you can connect that data and index it in your AI applications. So if you're building a chat and you want to be able to chat with your Notion data or your Google Drive data or someone else's, this is the easiest way to do it. Everything from auth to indexing is handled for you, and the code you have to write is so comically simple. You can just curl their endpoint with your API key and tell them what you want to index. You can hand them a PDF directly if you want; they'll chunk it, index it all for you, and then you can retrieve it with a simple fetch request.
This stuff was not easy for us to set up in T3 Chat. There's a reason we're still adding things like PDF reading. This will make it super easy for you to build great Raggy experiences in your existing AI apps, or if you have a new app idea, this is probably the right way to get started with it too. Huge shout out to Raggy for sponsoring today's episode. Check them out today at soy.link/raggy.
Diving straight in, we should probably start with the pricing. I went and took all the most common things that people tend to use and put them in this quick chart so we can compare CHPT 4.0. The input and output is per 1 million tokens. I'll do that to fix formatting. So whenever you have 1 million input tokens, the token is roughly a word, so to speak.
1 million input tokens on standard ChatGBT 4.0 API pricing is $2.50; outputs are $10. For O3 Mini, it's 15 cents input and 60 cents output. That's why we offer O3 Mini as a free option in T3 Chat. Speaking of free tiers, by the way, we're actually temporarily offering O3 Mini on our free tier. So if you do want to try it out and use the best of the best current models, it's worth a shot.
And if you do want to try all the other models too, it's only eight bucks a month for very, very high message rate limits on all of the models that we offer, which right now is pretty much everything you'd ever reasonably want to use. So yeah, O3 Mini is very, very cheap.
O1, not so cheap: $15 per million input tokens, $60 per million output tokens. That's why O1 is rough to use and why you don't see it in too many services. There's one particular price I do want to call out here, though: CLA. CLA is insanely expensive, especially when you consider the fact that it's not even a reasoning model. I am floored at the price, and to get away with this, it's by far our biggest expense for T3 Chat, just paying Anthropic for cloud usage.
Deep Seek V3: 27 cents in, $110 out. V3 old, because they're changing the pricing. This is the pricing that I got so excited about. V3 was not their reasoning model; it's their—funny enough—it's actually really close to CLA in its performance and characteristics from my experience. But their old version of the pricing was hilariously cheap: 14 cents in, 28 cents out. It's actually cheaper than O3 Mini, with quality levels comparable to Sonet.
It was unbelievable. It's going to be a bit more expensive coming up soon, and I understand why. Their APIs have been hammered because of how cheap they made things and also because R1 blew up. Speaking of R1, 55 cents in, $219 out puts it at cheaper than O3 or Sonet while working closer to how O1 works.
But we are missing something in here; we don't have O3. So what is the price for O3 Mini? O3 Mini's input price is $1.10, and the output price is $4.40. That is incredibly competitive. This effectively, in my opinion, makes O1 not really worth touching. It makes the value pitch for 3.5 Sonet incredibly weird and arguably quite weak, and it also positions O3 Mini really, really well against Deep Seek R1.
All we're measuring here is the price. There's a lot of other things we need to measure. There are incredible services like Artificial Analysis; they do a great job of comparing different models and their pricing, but they don't have O3 on here yet. They're also much more focused on traditional benchmarks, things like the MLU, the GPQ.
I have my own things I like to use to test these models out, though. One in particular that I've actually enjoyed a lot is a little programming challenge I enjoy: Advent of Code. Advent of Code is a set of Christmas-themed problems that come out at the end of every year in December. You get one at around 9:00 p.m. Pacific time every night. It's a super competitive, really fun set of programming challenges. I compete actively every year, and last year I actually had a really good year.
You see my numbers here; I was in the top 1,000 almost every day. I should thank Cursor for a good bit of this; it helped a lot with writing things. I didn't want to, you know, the classic like traveling salesman type stuff. I don't want to ever write Dijkstra's again. Thank you, Cursor, for making that something I didn't have to do.
Some of these problems were brutal, though, in particular the ones that hurt me the most, which you can see by the hour plus they took: Day 17 specifically, Part Two, Day 21, and Day 24. So what I have done is I took all of these problems—Days 17, 21, and 24—and I prompted Claude O1 Pro, O3 Mini, and R1 to see how they would perform. I thought this would be a fun test and ended up being more fun than I expected, not because the answers that they gave code-wise were great, but I learned so much more about how broken all of these UIs are.
See this ChatGPT thread? See that funny hourglass thing there? That's because this O3 Mini high prompt has just been sitting here for probably like 30 minutes now. This is all just one prompt, by the way. It just kept reasoning, kept reasoning, and then just silently died. Never even updated the title. It's bad. It's so bad.
And I resubscribed to Pro again, so I'm on the $200 tier so I could prompt not just O1 but O1 Pro, the super fancy, super expensive model they don't even let you hit via API. Oh God, their UI is so laggy. You see how long it took for that tab to change? I'm clicking now; it takes like half a second and then does a terrible scroll to the bottom once you—oh God, that's so bad.
How do they build it like this? We don't have any of these problems with T3 Chat. You click, and it immediately goes where you are clicking. No weird scroll movement, no weird rendering. So I used T3 Chat as much as I could. You can see that because when you hover, now it says what model you generated with. Nice addition; makes this a lot easier for us to figure out and test.
So what were the results? Let's start with looking at Day 17 so you can understand the problem. This is a fun one. Part One was an interesting three-bit computer problem. Let me try and remember exactly what it was. Oh yeah, so you have these three registers, and you have the program. You have to determine what the program is trying to output, which is going to be a set of numbers. You're given these operands: 0 through 3 represent literal values; 4 is A, 5 is B, 6 is C, 7 does not appear in valid programs.
This is so that you're building a three-bit computer, basically. It's a fun problem; they love doing things like this. Let's see how the answers came out, though. Part One: this is my answer. O1 got it, O3 got it, R1 got it. Claude made a real fun hallucination. I'm going to Command Z till I have it back; it was hilarious. Here it is: it hallucinated Windows, believe it or not. Your window does not have a node-compatible file system on it, so I had to fix that.
But once I fixed that hilarious hallucination, we were good. Part Two: nothing got it. R1 ran it indefinitely, and Claude had the same hallucination, but it failed because it set a cap on how many times it should run because it assumed for Part Two that it could eventually just find it under a certain reasonable range. Clearly does not understand Advent of Code very well because Part Two here was what's the lowest positive initial value for register A that causes the program to output a copy of itself.
This was a brutal Part Two that took a very, very long time to solve with a ton of caching, and I don't remember how I solved this one; it was a while ago. But yeah, none of them could solve that. There's a reason it was such a hard problem. Makes sense.
How about Day 21? Day 21 was a really interesting problem. It has this keypad, and instead of the keypad being a thing you just press as a human, there's a robot that is pressing the buttons. You tell the robot which button to press by giving instructions: up, down, left, right to move its finger up, down, left, or right, and then press to press.
There are a couple catches, though. If it ever hovers its finger over this empty spot, it fails, so you have to make sure all the instructions you give never put it here. Catch two is you don't control it directly; you control it via another one of these pads. This pad is on the back of the robot. Then there's another robot behind it controlling that, then another behind that.
So you have to expand the instruction set from whatever the numbers are. Like if you wanted to hit 852, then that is to start with the robot at A, so that has to go up, left, and then hit 8. But then the robot behind that has to hit up on the robot, and hopefully you're getting the idea. This is a really weird problem, and it's a very fun one, but it's also a very hard one, which is why none of them got the answer.
One interesting thing I noticed is that Claude just straight up ignored my prompt. My prompt specifies in it to grab the input from a file named input.txt. Claude just ignored it, which is interesting. I was surprised. I don't know if that's because it's not a reasoning model or if it's just so much context it loses track of that instruction at the top, but Claude just straight up ignored that instruction.
The others at least wrote code that processed the right data, but they didn't get the right answer, and nothing even came close on Part Two. What even was Part Two on this one? Oh yeah, they changed it from like three robots to 25 for Part Two. It was brutal. If I recall, my code actually handled that okay, and my Part Two, yeah, I just had to bump a number for Part Two, and it just worked for me. That's how I got such a good score there.
Day 24 was nuts because this was on Christmas Eve, and they gave us a really hard problem. Part One wasn't too bad; it's combining binary values with ANDs, ORs, and XORs. Not too bad. You have your gates, and you have these instructions like x00 and y00 goes to z0, and you then combine one and zero to the value that you put there.
They gave a larger example down here; you have to calculate all the values that would result in through this wire system where things connect to other things. Then once you've done that, you solve the system. You have z00 through z12; use this to create a binary number and then convert it to decimal, which produces an actual number.
How did these do with that? Part One: O1 did it right, O3 also. I should note O1 is not just standard; I used O1 Pro for all these, which meant it took a while. It took like half an hour to solve this one just to give me the output code, and then it took even longer to solve the problem after. In fact, Part Two, it just hung forever.
The interesting thing with O3 Mini is that when I ran it a second time—which, funny enough, was because the UI on the ChatGPT website failed when I did the first part—so I made a new prompt to do it again so I could pass it Part Two. The second time I gave it, it failed Part One, so I didn't follow up. Instead, I ended up combining Part One and Part Two into a single prompt, saying log both the solutions for Part One and Part Two.
Seems totally fine. When I gave up Part One and Part Two in the same prompt, that's when I hit this hang. Wait, did it finish? Wait, no, it didn't because this is the—yeah, this is the combo that never finished. Yeah, it just kind of sat there. It's hilarious. So I didn't like doing that much work at once, especially on the high reasoning.
But this is where the differences actually start to show between the O series from OpenAI and R1. R1 did not solve this problem at all. I was very surprised. I thought the reasoning model would totally smoke this. Nope, it did not. That said, this means the difference between the O models and the R1 model from Deep Seek is just one of these problems.
All of the reasoning models got Part One right. Claude got Part One right after me helping. Nothing got Part Two right or any of Day 21 right, and only the reasoning models by OpenAI got 24 right, and they had a pretty high failure rate too. This isn't obviously a particularly great analysis; I'm not doing crazy scientific tests or anything here, but it's a thing that I care about because I don't want these models to get good enough at these problems that the leaderboards mean nothing.
So I always try to test this stuff. I would have written a script to test it against the APIs, and I also would have done this in T3 Chat if the R1 API worked right now. We've been to hell and back with this. With T3 Chat, we do currently offer Hyperbolic as an option. The problem with that is that Hyperbolic is not a particularly fast way to run R1.
We have the R1 distilled model that is hosted on Grok, which flies, but it's not the full R1 reasoning model. It is a version of Llama that was distilled from the R1 reasoning model so that it can behave very similarly but also works like a traditional LM. It doesn't have the extra steps; it just outputs a ton of text. But this time, the text is based on the text from traditional R1. It's actually really good.
I've been impressed with it overall. When you see the performance you can get from it, you'll understand. Like, let's give it one of these harder prompts. We'll give it Day 21, Part One and Two, and see how fast the reasoning is coming out. Like, it's faster than I can scroll; it's pretty nuts.
Yeah, this is actually insane how fast it can go. We'll see if it ever finishes reasoning because this one—this is also the same one that got stuck for O3. But wait, the example says total steps is 68, which is four digits. 68 over 4 is 17 steps per digit. My calculation here is 12 steps for the first digit; it doesn't add up.
Oh boy, it tried; it went really fast, but it did not succeed. Let's give just Part One then. Now it has just Part One; still flies. But again, like you can see the speed here. If I was to go do the same thing but with the R1 Hyperbolic model, do you see the difference here?
And yes, with T3 Chat, you can actually have multiple things streaming in at once, and it's totally fine. Unlike in the ChatGPT app, where if you have something running on Pro and you go make another query somewhere else—hell, even if you just leave the tab alone for too long, it will fail, which is obnoxious because these queries can take like 20 minutes to run sometimes. I don't know how they get away with it; it's obnoxious.
We're putting a lot of work in on T3 Chat to make this better. Oh, looks like it failed there still too. This will eventually fail; it's just taking its time.
So how does this compare to O3 Mini, the model that we're all here for today? Well, if I switch this over to O3 Mini in our UI, it's going to sit here for a while, but that's not the experience you expect. If we go do it over here, you get something different. When we use this in the ChatGBT UI, it gives you some of the info about the reasoning here, which is actually really cool to see.
There is a catch, though: they only give you this info in their UI; it's not exposed via the API at all. Why would they do that? Why would they just not give this data out via the API? They don't want us to have it. On one hand, it's because they want people like me to provide a UI experience that's even better than what they can do, but I don't think that's the main reason.
I think the main reason is reasoning. They don't want to give this reasoning data to companies like Deep Seek because if Deep Seek had access to all the reasons that ChatGPT was doing something, it'd be a lot easier for them to train a model on top of that data.
Wow, these models are really struggling with this problem. Let's give them Day 17 because that's a little less bad. Yeah, literally all of the models are failing or just looping indefinitely when given this problem. That's real fun.
Okay, 17, Part One's easy; we'll give it that. One other interesting thing with the new model: you have different levels of reasoning effort: low, medium, and high. Some other quirks I noticed with O3 Mini: since it's not streaming in or since you don't get the reasoning over the API, you also don't get to stream the response. It just kind of sends the whole thing as soon as it's done, which I don't like the user experience of.
I've grown to really like seeing the model spit things out piece by piece, and if I see it not doing what I want it to, I can't stop it because we have to add a stop button to T3 Chat coming soon. But I'll just go back up and edit and change the message, and hopefully the second time it'll do a bit better.
The other thing is you might notice it from the formatting here. I don't know why it's like this, but for some reason, O3 has an entirely different format it likes to spit out. It's very strange. If we copy this output, it's not doing markdown formatting, even though every other model I've ever worked with leans into markdown formatting by default.
It loves adding these weird bars everywhere; it doesn't indicate what language files are almost ever. I added a system prompt to help with it, and it's not helping enough. I just straight up don't know why they're doing this like this. I've never seen a model that just came up with its own formatting like this, and the result sucks for us to build around.
I don't know how much of this is intentional or a weird consequence of the things they trained on or is an attempt that they're doing to try and keep us from being able to compete with them as much as I would like to, but it feels weird. I don't like it. There's something strange going on with the formatting here.
We're doing our best to work around it in our UI. I might have to make changes to how we're actually generating the data; I might have to give it some weird tooling to handle this all. But know that we're working around the clock to get it to not do these weird blocks.
Seeing a handful of people on Twitter, especially PHP devs, getting these horrifying formats? Not acceptable. Know that we're on it. I was going to look for a benchmark, but I remembered this tweet, so I have to show this. OpenAI just straight up throwing shots at the Deep Seek stuff: "Explain AI distillation like I'm five."
This is them making fun of the fact that people are all really excited over distillation models and also pointing out that the model that Deep Seek did, in their opinion, might be a distilled model. I don't think standard R1 would qualify as distilled. Usually, distilled means there's an existing model they're trying to make better by having it prompt another model to get more data that it can train against.
That's more what Deep Seek did to their Llama distilled version that they distilled using R1. But if you read their white paper, they had some genuinely really novel techniques that made their model as good as it is for as cheap as they did. But it is nice to see them throwing shots here and there.
This is from McK Wrigley, who has been doing a lot of fun AI stuff recently. I'll give him a follow; his data has been nice to see. He has an app that he built that uses O1 right now because the prompts are difficult enough that he's found a lot of benefit from it. His agents that were using O1 have now been moved to O3 Mini, and they're all working at least as well as they were before, but some are even better, and that's for 9x cheaper and also significantly faster.
If you look at the code forces scores for O3 Mini, the low score for the low usage, low power solution is right up there with O1. The medium power is beating O1, and high is smoking O1. When you compare the pricing, you see why this is such a crazy launch.
It's crazy to think the model pricing has gotten this cheap this fast that they're actually sniping out the prices of their own non-reasoning models. This is still the craziest thing to take away from this all: a reasoning model that is by an American company is now cheaper than their non-reasoning model because they're so scared of what Deep Seek is doing.
It's still almost exactly—actually, it is literally exactly double the price. That's hilarious. That was done by choice. They realized if they went more than 2x on the price for O3 Mini, everyone would just keep showing Deep Seek. But by pricing it this way, it's a lot harder for people hosting Deep Seek themselves to get it to a price that is comparable, especially for the performance you'll get from them because getting R1 to perform on your own is hellish.
Lex had a good tweet about this that I wanted to include as well. O3 Mini is a good model, but Deep Seek R1 is similar performance, still cheaper, and it reveals its reasoning. This is the biggest thing: Deep Seek doesn't hide the reasoning; it just dumps it straight out there for you if you're using it via API. If you're self-hosting or doing anything else, it doesn't matter; you always have that data.
OpenAI, despite being OpenAI, loves to hide these things from you. As Lex says here, better models will come. He still can't wait for O3 Pro, but the Deep Seek moment is real. I think it will still be remembered five years from now as a pivotal event in history due in part to the geopolitical implications, but for many other reasons too.
Apparently, there's a podcast coming out about this; very excited for that. But the core point being O3 is great and keeps OpenAI from falling too far behind, but it doesn't bring us a lot of the things that we're so excited about with Deep Seek. It really does feel like an attempt to keep up with what happened here.
Not in the sense that their model is worse than this; it's actually, I would say, O3 is quite a bit better than R1 in terms of its raw performance, especially when you consider the speed. But R1 shows you what it's thinking. R1 is an open-source model that you can host wherever else. R1 is way, way cheaper.
O3 Mini is an attempt to catch up to where Deep Seek has put us in terms of the marketplace, and it's a very exciting one. It's so exciting that we're giving it out for free for a bit, and I have a lot of fun things I want to try with it.
My personal project: is it a revolution? No, but it is the result of what I would argue is a revolution, which is the crazy stuff that Deep Seek has been working on. I still can't get over how bad their UI is, though. I have been fighting it all day because I was trying to do these O1 Pro prompts. It takes up to 10 seconds when you scroll to the bottom for the rest of your tabs to show.
God, how is it like this? How is their pagination so bad? Their scroll containers are so broken. Like, guys, we're like two devs in a trench coat here; we don't have these problems at all. Everything just works when you click it. It's not that hard. Seriously, if you need help, my phone number is in Bookface.
This is a really exciting release. Even if it's not revolutionary, it is showing a revolution did happen. I'm curious what you guys think. Is O3 Mini exciting to you, or is this a release you're going to pass? Until next time, peace nerds!