Transcription
OpenAI just dropped a paper showing what it's going to take for artificial intelligence to become the best coder in the entire world.
Here's the thing: the strategies that OpenAI revealed in this paper are not specific to coding. It proves that using reinforcement learning, scaled up, is the path to AGI and beyond.
So, I'm going to break down this paper to show you that with reinforcement learning plus test time compute, the sky is the limit on intelligence.
But before we dive in, I want to show you one clip. Just a couple of weeks ago, Sam Altman did an interview in Japan where he literally said, "03, which we talked about publicly in December, is the 175th best competitive programmer in the world. I think our internal benchmark is now around 50, and maybe we'll hit number one by the end of this year."
So now you see where the bar is set, and you also see where OpenAI is confident they're going to be by the end of this year.
Here's the paper: "Competitive Programming with Large Reasoning Models." It was published by OpenAI just a couple of weeks ago. If you've watched this channel over the last couple of weeks since the Deep Seek model was released and the Deep Seek R1 paper was published, you already know where we're going to go with this.
Reinforcement learning with verifiable rewards plus test time compute are the two scaling levers necessary to reach insane intelligence in these models. Removing the human from the equation seems to be the key.
Now let me pause for a second and talk about Deep Seek. Deep Seek R1 just a few weeks ago seemingly took over the world, and a lot of people focused on the cost and the efficiency of this model. It only cost $5 million to train, even though that's not really an apples-to-apples comparison, or it was really efficient.
Look, what they did was incredible, but really the key thing that that paper illustrated to the world was that reinforcement learning was the major unlock to get that model to perform well.
Since then, we've had examples of reinforcement learning with verifiable rewards being able to elicit the thinking behavior from very small models. Just a week ago, I published a video about a Berkeley PhD student who spent $30 and was able to get that thinking behavior from a 1.5 billion parameter model.
So this is clearly the strategy that is going to take us to the next level of artificial intelligence.
Let me talk about what reinforcement learning with verifiable rewards actually is. It's the same method that was used to get AlphaGo to be the best Go player in the world and, in fact, to discover strategies in the game Go that no human had ever thought of—the famous move 37.
Essentially, what it is, is you can kind of think of it as a way for AI to self-play. It's going to try a lot of different things, and you can scale that up as much as you want. Each time, if it gets the answer right, it gets a reward; if it gets the answer wrong, it doesn't get that reward.
Obviously, that's a huge oversimplification of how it's actually working, but let's again talk about the game Go. You can imagine two AIs playing the game Go against each other. Whichever one wins had the better strategy, and then we can actually give a reward to AI because it won that game.
Now imagine doing that tens, hundreds, thousands, millions of times. Eventually, the AI is going to learn the optimal strategy to play Go in any situation.
There are a few reasons why that approach is so special. Number one, we can essentially scale that up in an unlimited way. We could just continue to let these AI systems play against each other over and over again.
The reason we're able to do that is because there's no human in the loop. AlphaGo did not actually provide the AI with any existing games. It basically just said, "Here are the rules. Here's what you get if you win. Here's what you get if you lose."
Along the way, no human was actually grading the performance, and that allowed AlphaGo to beat the best human Go player in the world.
This is all verifiable rewards, and what that part means is there's actually a defined solution in the game Go. You know whether you win; you know whether you lose. That's the verifiable part, but that applies to other domains as well.
Basically, all STEM—science, technology, engineering, and math—have verifiable rewards. 1 + 1 always equals 2, and so if the model says 1 + 1 equals 3, you know that was wrong.
With coding, you also have verifiable rewards. When you have a coding problem, you know what the answer is. Even if the code can be a little bit different here and there, the final output of the code has to be the same; it has to operate the same.
So if the problem says, "Give me an array of numbers between 1 and 10," the output you already know. But how that array of numbers is actually generated can be different, but the answer is still right.
You can also execute code, so if you execute the code and there's an error, then of course you know the answer is wrong.
Now keep that in mind as we go through this paper.
Thanks to the sponsor of this segment, LangTrace. They have been awesome partners to us, so excited to tell you about them again today.
LangTrace is a leading AI software development consulting company that builds AI products to propel your business forward. Those products include an open-source and open telemetry-based observability and evaluations platform that helps you evaluate and improve your LLM usage in your application.
It's trusted by thousands of developers, from early-stage companies all the way up to Fortune 500 companies. LangTrace helps developers collect and analyze traces, collect data sets, and run evaluations, resulting in highly reliable and secure AI systems.
Again, LangTrace is open-source, and if you want to use their hosted version, you can get 20% off right now if you use the link in my description.
If you want to learn more, join one of their upcoming webinars where they go over everything. So check them out; they've been a great partner. Go see their GitHub, and thanks again to LangTrace.
Now back to the video.
In this paper, they compare a few different approaches, but really two main approaches. First, as a baseline, they use GPT-4 in a coding competition, and it performs pretty well.
But then the next step after that is a reasoning model, the 01 and the 03 models—models that leverage test time compute, the ability for the model to think during inference time. This greatly improved the coding quality of AI.
But then they also wanted to compare using handwritten inference time strategies to scaling up a more generalized model.
Now take a second to think about this. Once again, they're taking a reasoning model and they're adding humans into the loop—human beings creating those inference time strategies. When I say that, I just mean sophisticated prompts and selection criteria for the actual answers.
But the point is, humans are actually writing these, and then they compared it against scaling up the model using reinforcement learning with verifiable rewards up to 03 with no humans.
The results are very interesting.
Let me read a little bit from this paper: "Solving complex algorithmic problems demands advanced computational thinking and problem-solving skills. These problems are also objectively gradable." That's the key—verifiable rewards, objectively gradable—making it an ideal test bed to assess the reasoning capabilities of AI models.
Now they go on to talk about how initially they had kind of the standard non-thinking models, and the models ranged from 244 million parameters all the way up to 137 billion parameters. They can generate code pretty darn well, and we've seen that performance improves log-linearly with model size, and fine-tuning significantly boosts accuracy.
Then they talk about AlphaCode, and anytime you hear "Alpha," think reinforcement learning. That's the technique that they're using in AlphaCode here, and it tackled competitive programming tasks using large-scale code generation and heuristics at inference.
AlphaCode 2's improvements nearly doubled AlphaCode, solved problems, and placed it in the 85th percentile of Codeforces. Both of the AlphaCode systems use large-scale sampling of up to a million candidate solutions per problem before selecting their top 10 submissions with, here's the key, hand-engineered test time strategy—hand-engineered, human in the loop.
Then they had large reasoning models, 01 and 03. These models use chain of thought reasoning to tackle intricate tasks such as mathematics and coding work by Deep Seek, R1, and Kimmy K 1.5, which, by the way, really flew under the radar but accomplished much of what Deep Seek also accomplished independently.
This illustrates how learning chain of thought boosts performance on both mathematical and programming challenges.
So this chain of thought is really powerful, but there's still an open question: are these hand-engineered inference strategies the best approach? Are they really even necessary to reach the number one coding AI?
They say they have three systems available that can shed light on this question. They have 01, which is just the reasoning model; 01 II, which is that reasoning model but with these more sophisticated inference and selection criteria; and early checkpoints of 03, generally not 01 III.
You probably already know where I'm going with this, but I want to give another analogy. Think about Tesla's full self-driving. Just a few years ago, Tesla's full self-driving was a hybrid approach between a neural network and handwritten rules by humans, and they hit a ceiling in performance.
Again, just a few years ago, they basically ripped that system out and decided to go with a full end-to-end neural network using reinforcement learning. That means there is no human in the loop, and since then, they have blown through that ceiling, and the performance is really incredible now.
But the point is, the human in the loop not only wasn't necessary but was a limiting factor on their performance. Guess what? That's what OpenAI sees as well with coding AI.
All right, so let's start with OpenAI 01. This is the reasoning model but without sophisticated inference time prompting strategies. By generating an extended internal chain of thought before answering, 01 resembles a human who methodically works through a challenging problem step by step.
Reinforcement learning refines this chain of thought process, helping the model identify and correct errors, breaking down complex tasks into manageable parts, and exploring alternate paths when an approach fails.
We've all seen how good chain of thought is; that's basically how it works. Not only that, 01 is trained to use external tools like writing and executing code in a secure environment, and that allows the 01 model to verify whether the code it generated is right or wrong.
All right, so first we have the Codeforces benchmark. Let's see how these models actually perform. Codeforces is a programming competition website that hosts live contests. It is internationally competitive and frequented by some of the best competitive programmers in the world.
Now scrolling down, let's look at this. This is the Codeforces rating on the y-axis and the different models on the x-axis. Here we have GPT-4 with an ELO of 808. Then we have 01 preview at 1258 and 01 at 1673.
You can see a drastic improvement at each of these model upgrades, and this shows how reinforcement learning is really effective for complex reasoning.
Now here is something really important to know: this is 01 with IOI, and I'm going to explain exactly how IOI works in a moment.
During our development and evaluation of 01, we found that increasing both the amount of reinforcement learning compute and test time inference compute consistently improved model performance.
So not only letting it do that self-play with reinforcement learning, but also at the inference time, allowing it to think longer and longer using more tokens, doing that chain of thought which I just described.
As you can see here, as they scaled up train time compute, the performance improved. On the right side, same thing but with test time compute.
So this is reinforcement learning on the left and this is test time post-training on the right. Now, IOI incorporates specialized test time inference strategies engineered for competitive programming, but remember those are human-written, handwritten.
Here's how it works: they divide each II problem into its constituent subtasks, sampled 10,000 solutions from 01 II for each subtask, and then employed clustering and reranking-based approaches to decide which solution from this set to submit.
Now, if you're already thinking, "Dang, that sounds really complicated," well, you're right, and you're going to see what happens when we remove that and just scale up reinforcement learning and test time compute.
They go on to describe clustering and reranking. These are again just the very complicated handwritten approaches or strategies to choosing the best answer from this 01 model.
The gist of clustering is they took a bunch of outputs, and the ones that were similar, they clustered together, and then they would choose one from that. The reranking is again another sophisticated approach, and they would score each solution on the following: quality, errors in model-generated test inputs, and failing the provided public test cases—again, handwritten.
Then they submitted up to 50, which is the maximum allowed for humans as well, so they're following the same rules as humans would, and they did so in a round-robin fashion over a subtask, starting from the hardest.
It actually performed really well. Don't get me wrong, really well. So 01 II reached a Codeforces rating of 1807, outperforming 93% of competitors.
Then when they added in that test time strategy, that handwritten strategy, it went up to the 998th percentile at 2214.
So here you can see Codeforces 01 right here, 1673; 01 II, so handwritten strategy, 1807; 01 II with public test filtering, even better; and then with the full test time strategy, 2214.
Now let me show you 03. This is the model without any handwritten strategy.
While 01 II achieved strong results by combining additional reinforcement learning fine-tuning with carefully designed test time inference pipelines—a fancy way of saying a lot of reinforcement learning and a lot of custom-written inference strategies—its success hinged on human intervention to define and implement these strategies.
That's the key right there: human intervention. Exactly how Tesla removed humans from their end-to-end neural network, and then they saw massive improvements.
The same way that AlphaGo works: they removed humans and just let AI play with itself. This is what you're going to see right now.
So here it is: 03 on the Codeforces rating jumped up to a 99.8 percentile, 2724 ELO score, and that is a massive jump from 01 II.
Now, if you're thinking 98th to 99.8 is not much, it is. Look at the ELO difference: 2200 versus 2700.
Now here is something important to note: check this out. Whereas 01 II relied on a complex human-defined test time strategy, 03 followed a much simpler approach.
03 greatly outperformed 01. So just scaling up the reinforcement learning and test time compute without the need for subtask-specific prompts, manual partitioning, or intricate submission strategies—basically scale up AI, let it do its thing, and it will be better.
So 01 II scored 213 points with 50 submissions, and that is just above the gold medal cutoff. 03, again, 03 just raw, 395.8 four points surpassing the gold threshold and went under the 50 submission limit.
So again, it seems like just scaling up these models is what it's going to take to get us to the best programmer in the world being AI, and really beyond that, the best reasoner, the best at math, the best at science, the best at technology—AGI—and then finally from there, ASI.
What this paper illustrates is what we already know if you've watched this channel at all: reinforcement learning and test time compute is the clear path to AGI.
Sam Altman already said it. He said, "We have the clear path; we just need to build it." This is what he's talking about. They have the algorithms, they have the approach, but we just need to scale it up.
Now, if you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.