Transcription
Let's talk about reinforcement learning. It is the technique used to elicit thinking behavior from these Frontier models: 01 and 03 by OpenAI, R1 by DeepSeek, Claude 3.7—thinking. They all have this incredible thing that they do where they think, and reinforcement learning is the thing that got them to do that. Let me explain everything you need to know about reinforcement learning.
So, the very basics: what is reinforcement learning? It is when an AI agent interacts with an environment. That environment gives the agent feedback, and that feedback is the mechanism in which the agent learns. That feedback is also known as a reward. That reward tells the agent how good or how bad the action they took is, and that's all relative to whatever the goal of the agent is. And so, over time, this AI agent learns to maximize the rewards they get through their actions. They don't really understand the actions they're taking; they just know this action results in the maximum reward.
All right, so what if the reward is wrong? In this situation, we might choose the wrong reward, or we think we're choosing the right one, but it isn't actually exactly what we want. And in this case, the agent will literally learn undesirable behavior—behavior that is not in line with what we want it to do. This is called misspecified rewards, and the agent doesn't know any better; it is just trying to optimize for whatever reward we set up for it.
So here's an example of that: researchers at OpenAI set up this game, and it's a boat racing game, and they were trying to teach AI how to be the best boat racer in the game. And so the intended goal is for the human to finish the race as quickly as possible. However, the game's point system rewarded the player for hitting bonus targets on the track on the way to finishing the race. But the AI discovered something interesting: if it just drove around in circles and hit the bonus objects along the way and didn't actually finish the race, it would get a lot more points; thus, it would get a lot more rewards. But that's not what we wanted it to do; we wanted it to finish the race. And so this was an unintended behavior. This is also known as reward hacking—basically, the AI figured out a loophole or a way to cheat, and it doesn't know what's cheating or not; it's just trying to figure out how to get that reward, and it figured out a loophole in the game. And so this is where reward verification really matters. The key question is: how do we ensure that the reward truly represents the desired outcome? Reward verification allows us to confirm that the rewards we give are reliable indicators of success.
And now we come to the next topic: verifiable rewards, which is incredibly important in the world of thinking models. This is the key aspect of what makes thinking models scale to be able to think. Verifiable rewards are reward signals that come from objectives that can be verified.
So here's a very simple example: 2 + 2 = 4. If an AI predicts that 2 + 2 = 4, we know that's true, so we can verify that. That is a verifiable reward. But there are many areas of knowledge that are not verifiable at all. Think about anything creative: writing a poem, writing a story, creating a song—all of these things cannot be verified. If I say, "Write me a poem," how am I supposed to tell the AI whether it was right or wrong? Now, obviously, I can have human preference; I can say, "Yes, that's a good one," "No, that's a bad one," but that's not scalable, and it's also not necessarily verifiable. And so, for verifiable rewards, there is the requirement that it needs to be verifiably correct using automation or some kind of ground truth.
So let's go back to the boat game one more time. That's not technically a verifiable reward; that is what is called a proxy reward. The proxy is the score of the game. If you get a really high score, we have to assume you're doing well in the game. With verifiable rewards, we don't need a proxy; there is a direct link between the question or the problem and the answer or solution. And there's this concept of binary or scaled rewards. For binary, the answer is either correct or incorrect, success or failure. But sometimes they can be scaled, and you can get partially right, and as long as the partial credit is objectively determined, that can still work. And actually, a decision many model makers have to decide on is whether they're using process rewards or outcome rewards. So with an outcome reward model, you might have multiple steps to get to the final solution. So let's say we give it a really hard math problem that requires multiple steps, and it gets five of six of those steps right until the very final step, in which it got it wrong. With an outcome reward model, the entire thing would be wrong, and it didn't really learn anything about how it got stuff right along the way. Now, with a process reward model, it can get five of six right, then the final answer is wrong, but it can actually learn that those first five steps it got right, and maybe it tries something different on step six.
So why are verifiable rewards so important? Verifiable rewards ensure alignment with desired behavior. It's a really clean way and scalable way of doing things. First, it is aligning to a true goal, and so different from the boat game when we're using a proxy reward; when it's a verifiable reward, we know we are only giving credit to the model when it is due. And so, in this instance, the agent can't figure out a loophole; it either achieves the goal or it doesn't. Verifiable rewards are also very resistant to reward hacking. And so, because we're defining something where the result has to be exactly matching the actual ground truth, it's really hard to hack that kind of reward. This type of reward—verifiable rewards—also tends to be bias-free and objective, unlike other reward schemes like human preference. There is no human bias whatsoever in it; it is programmatically determined whether the model got something right or wrong. And because of this, the model learns predictable and reliable behavior. Because these verifiable rewards also leave no wiggle room for interpretation, the agent's behavior tends to be safer and more trustworthy. It's less likely to produce bizarre or unintended behavior or harmful strategies because those would just fail the verification check.
Let's talk about some real-world examples. Now, where do we see reinforcement learning in the real world? Verifiable rewards, as I mentioned, work best when there is a clear definition of what the answer is, and a lot of real-world tasks have this property. Think basically anything with STEM: science, technology, engineering, and math. These are four categories that generally have one input and one output. So if you're solving a math problem, 2 + 2 is always equal to 4; 8 + 8 always equals 16. Or if you're writing a computer program, there's a single output you're looking for, and so you can actually run the program and see, "Does it match this output?" And so the main place that we're seeing reinforcement learning with verifiable rewards come into play is large language models. So these GPT-based models are originally trained on just tons and tons of data, and we can fine-tune them further, but now we're adding in reinforcement learning with verifiable rewards to elicit this thinking behavior. So let's say you have a good base model; you can say, "Okay, you're going to be really good at solving multiplication tables, and I'm going to give you reinforcement learning with verifiable rewards because I know multiplication tables always have a verifiable reward." And after a while, it'll start answering the multiplication tables better and better and maybe eventually start thinking longer about it, especially when you have more challenging questions like advanced logic reasoning, hardcore math, coding challenges, and so on. And this technique is what's allowing these models to start thinking. DeepSeek R1 showed, independently from OpenAI's 01 and 03 models, that yes, that is how reinforcement learning with verifiable rewards can really benefit these models and start eliciting that thinking behavior from them.
And so that's the overview of reinforcement learning with verifiable rewards. It is one of the most interesting things in artificial intelligence today, and I hope you enjoyed this video. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.