Transcription
Another day, another model.
So today, we're going to compare O3 Mini, just released from OpenAI, with Deep Seek R1. Here you can see the tests I want to conduct today.
Of course, this will be kind of my first impressions, but I want to go over a few coding tests. We're going to try some 3D animations, as we have done in previous videos. I want to make both models write some code that can edit a small video clip, and I want to do some text extraction. I will explain that a bit more when we get into the tests.
I want to do some AI agent orchestration. Basically, I want O3 Mini to plan out how it wants to assign tasks to AI agents. I want to do the same with Deep Seek R1 and see what results we get.
I want to run the "Breaking Free" from the training data again. That is basically the river crossing puzzle, a variant of this, as we did in previous videos. I want to run my handcrafted "read between the lines" question.
Finally, I saw O3 Mini has a huge output token window, which is pretty interesting. We're going to test that out more, but let's do an initial test.
So, you can see it has 100,000 tokens for output. I want to see how far we can push that. R1 has 8K token outputs, which is a bit limited, but let's see how far we can push it to the limit.
Basically, what that means is that we can theoretically get 100,000 tokens in one response from O3 Mini. That is quite interesting and something we have to test more.
But yeah, let's just get into it and start with some coding tests, compare the results, and see what O3 Mini can do versus Deep Seek R1.
Also, just a couple of things before we start. If you look at the pricing for O3 Mini, we're going to pay $1.10 for the input tokens and $440 per million output tokens. So, it's pretty cheap. I would say it's cheaper than GPT-4, basically half the price of GPT-4, so that's a big slash in prices.
Again, I guess they kind of have to do that. If you look at the pricing on Deep Seek, that is very cheap. I think the price is going to come up a bit.
Also, I am running Deep Seek R1 using Together AI in this video, so that's kind of third-party hosting this model for us. O3 Mini has three reasoning efforts we can select from: low, medium, and high. I think we're going to select high for our tests now.
So, basically, there's a parameter here called reasoning effort, and we're going to set this to high. That's going to be for all the tests we are running.
Okay, so let's run a few coding tests here. You can see we have the Deep Seek R1 function and the O3 Mini function. We have selected the reasoning effort to high here, and that is basically it.
We're just going to read the code from "code one.txt." The assignment is going to be the context, and we're going to write code to solve the problem.
So, code one is going to be: create a 3D animated browser wind tunnel simulation with wind speed and direction adjustments and transparency settings for the wind particles.
In the middle, there is an adjustable wing that can be rotated and tilted. The wind particles should be visible to show how the wing affects the airflow. Also, include controls so we can look at all angles. That is going to be the input.
Okay, and we're going to run that through Deep Seek R1 and O3 Mini to see if either of them comes close to what we are requesting here.
So, we are going to run this. Let me find the code here. First, we are going to run this on Deep Seek R1. You can see it's starting here now. The user wants a 3D animated browser wind tunnel simulation.
Yeah, I'm just going to let the thinking tokens run, and I'll come back when we actually have the final code that we can test.
Okay, so I lost my sound here, so I just have to do a voiceover. But you can see we completed our code here.
So, we're just going to copy this and paste it into our HTML file. That's going to be the Deep Seek HTML that we're going to test later.
After that, of course, we just want to run O3 Mini with the same prompt. We're going to get the HTML code from O3 Mini, do the same procedure, copy the code, and just paste it into our HTML file.
Then we want to compare them to see if they work or kind of what the results are. So, that is what I did here.
Now we're going to open up the results from Deep Seek. You can see we have something here. As I probably stated, this was not perfect, but you can see the wind speed is working. We can align the particles inside the wing, but as you notice, it's not affecting the airflow.
At least it's something. If we open up the one from O3 Mini, it's kind of disappointing; we don't see anything.
So, yeah, I got to give this one to Deep Seek R1 because at least we got something, even though it wasn't perfect. When we got the blank from O3 Mini, yeah, that was it, I guess.
The next part is I wanted to run the second test, and I need Python code. I can take an MP3 file and an MP4 file from the CV. The code should cut the video into the following intervals: from 5 seconds to 10 seconds and from 25 seconds to 35 seconds.
So basically, we're going to cut away 15 seconds. The code should merge the MP3 sound with the video. We're going to adjust the sound to cover the length of the video, and we're going to save the output as final_video.mp4.
So basically, a small video editing test using Python. Here, I want to run both Deep Seek R1 and O3 Mini to see if both can solve this problem.
We have the MP4 file and the MP3 file in our working directory, so it should be pretty easy to access this.
So here we're going to run, I think it was O3 Mini first, just to grab the code and save this into, I think we saved it just into O3_video.py or something. Then we're going to do that for both, and then we're going to run these files.
So, yeah, you can see me here pasting the code into O3_video.py, ready for testing. Then we're going to probably do the same with R1 here.
Yeah, you can see I'm firing up R1. We grab the code, and we should now be able to test both of these and compare the results.
I think I had to move this over to a second window because I had some issues with... yeah, you can see me here.
You can see we are running the O3 video. This is working perfectly. We are actually processing, and I think I just did the same with the R1 video.
When I looked at the code, the setup was pretty much the same, so I hoped that this was going to work.
So when this is complete, now we're going to first play the O3 video. Let's hear it if there is any background sound. You can see we have cut it down.
So let's play. [Music]
Yes, that was pretty good. You can see we cut the video clip down to 15 seconds, and we included the background sound. That was from O3 Mini. That was perfect.
Now let's run the version we got from R1. Yes, we have cut the video. [Music]
And we have the background sound. So I got to say that was pretty good from both models. Both passed.
The final thing I wanted to do was to run this: I need an HTML code that can extract all URLs from a PDF file. The user should be able to upload a PDF file, and the code should extract all the URLs from the file and return them in a structured clickable list.
So that was basically the assignment we gave both O3 Mini and Deep Seek R1 in this case.
So, yeah, I just ran it through. Let's see how that worked out. We grabbed the code from, I think this was from Deep Seek R1. Can't really remember now, but let's see where I saved this.
So I'm going to save this into PDF_R1.html. Yes, that was from Deep Seek. I basically ran this again using O3 Mini, and we grabbed the code.
Now we can compare them, right? Both of these are going to be opened up in the browser.
So, yeah, you can see I pasted it in here. Probably next, I'm just going to open up both these small apps. Here is the one from R1.
So what we can do now is just pick a file. We find a Voyager paper from PDF, and boom, we got two URLs. Can we click on this? Yes, and they are working. So that's pretty good.
So the R1 code is working well. Let's look at the one from the O3 Mini model. Pretty much the same setup. We grab the Voyager paper, and boom, we got the links.
So, yeah, I would say these were a pass. But if you look at this overall, I got to give this slightly to Deep Seek because it produced a more correct code on the first test.
The second test, both passed with good grace, so I was happy with that, but a slight advantage to Deep Seek.
Okay, so the next thing we wanted to do was to look at the AI agent orchestration. We're not going to spend loads of time on this, but I have this orchestration here that has a Deep Seek R1 function that can pick the agents we want to use.
We also set up the same using the O3 Mini orchestrator. These orchestrators are basically overarching.
Yeah, you can see we have control of these three agents: the Bitcoin agent, the stock agent, and the search agent.
The main goal now for O3 Mini is to give instructions to these agents to get the most out of them.
So we're going to assign a task to each agent we think is best for completing the goal. I'm going to show you what the goal is going to be.
The goal is: I want to understand the current state of Bitcoin and how it relates to Nvidia stock. Also, find recent news and changes in the tech stock market.
So it's not too much to think about. The goal now is just going to be pretty standard. The idea is to see if the models can give instructions to the agents that can actually complete the task.
Because if we look at the pricing tools here, like the stock price tool, we have to give a stock symbol, so this could be Apple, Nvidia, Google, to make this tool work.
In the BTC tool, we have to give the number of days to fetch the price history, and so on. So it's important that the models know this and can give good instructions or orchestrate the agents by using the correct instructions so we can actually complete the tasks.
I'm just going to run this. We're going to start off by running this, selecting... so let's do O3 Mini first, and then we're going to switch over to Deep Seek R1.
Right, so let's do Python the Tinker Dop. When we run this now, you can kind of see, I can zoom in a bit. You can see starting O3 Mini orchestrator go with goal, and here we kind of have the goal and creating completion requests.
If we zoom in a bit here, you can see here are the tasks O3 Mini wants to provide: detailed analysis of the current state of Bitcoin, including the latest price.
The stock agent is going to retrieve the current Nvidia stock information, including the price performance. I don't think we can do that, but the search agent, that's using Brave Search, is going to look to find some news articles covering the latest trends in the tech stock market.
So it kind of wants to assign this to the BTC agent. That's good. The stock agent is going to do the Nvidia, and that is perfect.
Executing the task, you can see the BTC agent is working on finding the price, the stock agent is working on finding the Nvidia stock information, and yeah, we got a synthesis here.
So you can see as of February 1st, 2025, Bitcoin is trading at $102,000. Over the past five days, it has ranged from $101,000 to $104,000, with moderate volatility considerations above.
So the synthesizing here in the final summary is done by O3 Mini. You can see Nvidia stock and its context: stock declined between 11% and 16.9%, corresponding to a loss of $600 billion in market value.
Wow! And you can also see tech market trends and recent news, particularly stocks linked to AI.
Yeah, and we can kind of read the overall here: while Bitcoin shows stability and consideration at high levels, Nvidia and, more broadly, many tech stocks are dealing with market headwinds due to largely competitive pressures and broader economic factors highlighting different dynamics in crypto versus traditional stocks.
So, yeah, the orchestration was completed successfully, and we got kind of a summary at the end here.
So, yeah, pretty good job by O3 Mini here. I got to say we fetched the Bitcoin price, we got the Nvidia price changes, and that was pretty good. We got some news from the stock market. Pretty good.
Yeah, I was happy with this. I got to say this was more than I expected.
But let's now run the same on Deep Seek and compare the results to see if Deep Seek can execute this as well as O3 Mini did.
The difference that I thought was pretty cool is that when we run this on Deep Seek R1, we kind of get the thinking process.
So, okay, let's tackle this problem. The user wants to understand. So that's pretty cool. We are still missing that from the OpenAI models.
So the task would be, you can see we are assigning the Bitcoin agent, the stock agent, and so on.
I'm going to let this run out and see if Deep Seek can also complete this task.
Okay, so we completed all the different tasks. Both the BTC agent, the stock agent, and the search agent did their tasks completely successfully. That's good.
So let's take a look at the synthesizing here. They're going to do some thinking tokens first. That's pretty cool.
Okay, so the Bitcoin's current state is bullish. Nvidia, three-plus percent drop.
Yeah, I think I got to give this to O3 Mini because we are missing some of the tools here. I think O3 Mini did a better job at actually calling the tools and giving the right information to actually get some results back from the agents.
Yeah, so I think I got to give this to O3 Mini. I think it performed a bit better, even though both models actually got the agents to successfully complete the operations.
I will give this to OpenAI O3 Mini.
Okay, so next, we're going to do the "Breaking Free" from the training data question. We have done this in other videos, but basically what we want to do is ask this prompt here.
So let me just clear this up.
The following is a famous puzzle: a man and a goat are on one side of the river. There's a wolf and a cabbage on the other side. The man has a boat. The man wants to get the goat to the other side of the river. How does the man get the goat to the other side of the river? Give a final answer with a confidence score.
The idea here is that since this is a very famous puzzle, we are changing some of the parameters of the puzzle. Usually, you have to shuffle back and forth because the man, the goat, the wolf, and the cabbage start on the same side, and you have to shuffle back and forth to solve this.
But here we are doing a small variant of this, and we have seen that this kind of confuses the models. Because it has so much training data on that shuffle, it always wants to go into that.
But with this, we want to see if they can reason and just solve the puzzle as it is now and not get too focused on the previous training data.
So we're going to run this on Deep Seek R1, and then we're going to run O3 Mini and compare the results to see if both models, or one of the models, or none of the models can actually break free from this and solve the puzzle as easily as it is.
Because the answer is just take the boat and get the goat to the other side. That's the answer.
So let's run R1 first and then run O3 Mini and compare the results.
Okay, so we have the answer back, but if you scroll up a bit here, you can see it did a lot of thinking on this.
So it states in the standard puzzle the man starts with a goat, wolf, and cabbage on one side. So it is referring to the original problem.
But you can see, according to the user's problem statement, the man and goat are on the same side, the cabbage and the wolf on the other side. Wait, that's different.
So the main goal is to get the goat to the other side where the wolf and cabbage are.
You can see it's already kind of fighting with the original training data here. I'm not going to go too much into the thinking tokens, so I think we just going to skip to the answer because it kind of came to the conclusion: the man can safely get the goat to the other side by doing the following steps: take the goat across in the boat.
Since the wolf and cabbage are already on the destination side, the man stays with them to prevent conflicts. No return trip is needed because the goal is to only transport a goat. The man can remain on the destination side to supervise.
Final answer: the man takes the goat directly across the river in the boat. Perfect!
So that was the answer we were looking for. Deep Seek R1 did a good job here.
Now let's run this on O3 Mini to see what result we get there.
Okay, perfect. So this was very simple. Final answer: the man simply loads the goat into the boat, rows straight across the river to join the wolf and the cabbage. Confidence score: 100%.
Perfect! So good job by both here. We solved the "Breaking Free from the training data" problem and had no issues with that.
So, yeah, happy days! Let's move on to look at my handcrafted "between the lines" question. I've done this a few times now, and we're just going to quickly go through that before we look at the output token test.
Okay, so this prompt is a bit different. I'm going to read it out. We have done it a few times on the channel now, but basically the idea here is that I have something in mind that's kind of written between the lines.
We put in some distractions, but I want to see if the models can at least touch on what my idea was here.
So I'm going to read this out: I was walking home from the hardware store with a bucket of blue paint in my hand. It's going to be nice to get this paint on the renovated room upstairs. The weather was sunny and warm, almost too warm for me.
Also, I just read in the paper that the New York Rangers won the Stanley Cup. Suddenly, I get a message from my phone: get to the hospital now.
Put the clues together and give your best guess of what is happening in one sentence.
So we have some clues here: the bucket of blue paint is a clue, the renovated room upstairs is a clue, the weather is a distraction, the New York Rangers winning the Stanley Cup is a distraction of sorts, and the "get to the hospital now" is a clue.
What I want to see here is some kind of mention that the model thinks that the partner of the guy walking home with the bucket of blue paint has gone into labor because they are renovating a nursery for their upcoming baby, right?
That is my thinking around this: that she suddenly had to get to the hospital.
So we're going to compare the results from Deep Seek R1 and O3 Mini and see if either of them mentions some kind of rush into labor or something like this around a newborn baby.
So let's clear this and run main five to see what the models think about this.
So let's start with Deep Seek R1 and then do O3 Mini.
Okay, so we got the results here. You can see Deep Seek R1 did a good job here. It kind of went through some of the clues.
So you can see another angle: if the person is pregnant, maybe the renovating the room is a nursery, blue paint for the baby boy. Perfect!
The message is going into labor. This is exactly what I was looking for.
I'm not going to read too much into it, but basically the final answer from R1 here is: the person is rushing to the hospital because their partner, who was likely painting the nursery blue for their upcoming baby, went into labor. Had a pregnancy-related emergency.
Yes, yes, yes! Perfect! So that is what I was looking for, and I'm happy to see that again from Deep Seek R1, as we've seen in previous tests.
So let's switch this up to O3 Mini now and run this to see what we can get back from O3 Mini here.
Okay, this was not too good. So we can read it: it sounds like on what should have been a perfect day, returning from the hardware store, ready to finish painting the room, celebrating the New York Rangers' Stanley Cup, has had an accident falling down from a ladder. Urgently, they need medical attention.
Okay, so yeah, nothing like what I was looking for. It's really annoying that we can't get the reasoning tokens because then we could have kind of looked into if it had this labor emergency in the thoughts here, or like the thinking part, not the thoughts.
So, yeah, we just get the final answer, and that was not what I was looking for.
So again, I have to give this to Deep Seek R1.
I think finally, let's do our output token test to see how many tokens we can actually get out from the models.
So here you can kind of see the prompt that Claude designed for me to generate maximum tokens: please provide an extremely detailed and comprehensive analysis of the following topics: the complete history of computer science, a detailed explanation of how AI and machine learning work, and a comprehensive guide to software development.
So a big prompt here, and it's going to be interesting to see how many output tokens we can get.
Remember, O3 Mini has 100,000 output tokens. I think that's pretty cool, but we also have to remember that the output tokens are also affected by reasoning tokens.
So we're not going to expect like 100,000 tokens on the output or the final answer, but combined we could kind of see 100,000 tokens.
But I doubt we're going to see that many, so it's going to be interesting to see how much we can get out in one shot.
Of course, Deep Seek R1 is very limited on this part because it only has a max output token of 8K, and that's including the thinking tokens.
So I expect a much less comprehensive answer, and we might even get cut short during the answer.
So I want to start running this on Deep Seek R1, then we're going to switch to O3 Mini and count how many tokens we got from this token test here.
So, yeah, let's run it: python token_test.py. Let's go!
Okay, so that was disappointing. I got to say we only got 2,200 tokens total, including the prompt tokens.
2,400? I even checked it on the tokenizer here from OpenAI: 2,200. So, yeah, that was not too good.
But I want to compare it to O3 Mini, so let's switch it up. I kind of have high expectations, I got to say, because I want to see a large output here now from O3 Mini.
Hopefully, I at least want to see like 10 to 20,000, but in my experience, I kind of doubt it.
But I heard Google Gemini has a very good output token, so up to like 25,000. That is something we're going to test in an upcoming video.
But let's see what we get out from O3 Mini now.
So you can see we generated 20,300 tokens, and yeah, that was pretty good.
But you can see we kind of got cut out here in the printout. Not quite sure what happened there, but yeah, I might look into that another time.
But remember, this is not 20,000 tokens in the final answer; this is including the reasoning tokens that we can't see.
So let's say, yeah, not as much as I was hoping for, to be honest.
But I think I have to maybe explore a bit more on the prompt side here because we might... I guess maybe this is designed to take in a lot of these reasoning tokens, and yeah, it's kind of a balance there between the reasoning tokens and the output tokens that we actually see, right?
But we will explore this more in the future.
But pretty interesting, I guess 20,000 isn't too bad, but I was kind of hoping we could see like 60,000 here, and maybe 30 or 20,000 was the actual output tokens.
But it's fine; it's an okay test.
So, yeah, I just want to quickly summarize this: O3 Mini, I guess it performed pretty well. I was really happy with how it performed in the AI agent orchestration. It did a very good job there, so we're going to definitely dive into that more.
On the coding part, it did pretty well. It kind of failed the 3D animations, and "Breaking Free from the training data" it did good on.
The handcrafted "read between the lines" question it did not complete, but we couldn't see the reasoning tokens.
Overall, pretty good. But is it better than Deep Seek R1? I wouldn't say that.
In a few areas here, it was better, at least in the AI agent orchestration. That is positive for me.
But there might be something we can do with Deep Seek R1 to get that even better too.
So overall, first impression: pretty good, but I got to explore it more to give my full conclusion on this model.
I got to try it out a bit more, maybe on cursor and stuff to see what it can do.
But, yeah, I enjoyed playing around with it. It was pretty fast. It was much quicker than R1, so that is also something that is positive if we think about O3.
And remember, I ran this on high reasoning, so if you put that even lower, we could have maybe gotten even more speed out of it.
But yeah, pretty cool.
If you liked this comparison, give this video a like. Maybe leave a comment about your thoughts.
So, yeah, thank you for tuning in. Hope you enjoyed it, and I'll see you again very soon!