Transcription
A PhD student at UC Berkeley was able to reproduce the "aha moment" of Deep Seek on his own model, and it turns out you can do it for just $30. So what does that actually mean? What did this person accomplish? I'll break everything down. Let's get into it.
Let me talk a little bit more about the "aha moment" before I show you how the Berkeley student was actually able to reproduce it. A particularly intriguing phenomenon observed during the training of Deep SR10—remember, this is the non-instruction-tuned version—is the occurrence of an "aha moment." This moment, as illustrated in Table 3, occurs in an intermediate version of the model. During this phase, it learns to allocate more thinking time to a problem by re-evaluating its initial approach.
This behavior is not only a testament to the model's growing reasoning abilities but also a captivating example of how reinforcement learning can lead to unexpected and sophisticated outcomes. And that's the key. That is really the thing that I think we're going to see much more of in the future of these LLMs. You're going to have the base LLM trained as usual, and then the second layer, which Dario, the Anthropic CEO, just talked about, which is this reinforcement learning layer, basically giving it the right rewards for getting an answer right or wrong.
Now here's the thing: this reward can only be well-defined when we actually know the answer to something—math, logic, reasoning, coding. But if you just ask it to do creative writing, there's no right answer to an open question like that. I'll touch more on that in a bit.
So this is Ja Pan—hopefully, I'm pronouncing your name right—and he's a PhD student at UC Berkeley. Just a few days ago, he put out this tweet: "We reproduced Deep Seek R10 in the Countdown game, and it just works through RL (reinforcement learning). The 3B (3 billion) based language model develops self-verification and search abilities all on its own. You can experience the 'aha moment' for yourself for less than $30. Here's the code. Here's what we learned."
Okay, let me tell you what all of this means. One of the major breakthroughs as part of the Deep Seek paper was this thing that they called the "aha moment," where this emergent capability came from the model to do this deep thinking. What they learned is that through reinforcement learning, the model started to develop this internal monologue and thinking ability all on its own. What they needed was a really good dataset.
Reinforcement learning is really special. This is the same method that allowed AlphaGo out of DeepMind to beat the best Go players in the world without actually having seen any human-annotated games. It basically just learned how to play Go by itself, and that is what they were able to show in the Deep Seek paper. By having a very well-defined reward function, the model was able to learn to think.
What is meant by reward function is an easy way to tell the model whether it got an answer right or wrong. Now, if you have a very open-ended question, then that's really difficult to do. An example of an open-ended question would be something like, "Which color is best?" There's no right answer; this is just completely opinion. However, for math, for logic, for reasoning, for programming, these are all areas in which there is generally a very definitive answer. When you have a definitive answer, you can tell the model, "Yes, you got this right," or "No, you didn't." Then the model can just try things over and over again until it gets it right more times than not.
All right, so now that you understand that, imagine taking that concept and applying it in a very narrow way to achieve that "aha moment." They did that with something called the Countdown game. The Countdown game is a game where players combine numbers with basic arithmetic to reach a target number. There is a definitive right answer to this, and so with that, they're able to create that very well-defined reward signal for the model.
So let's look at an example of what that actually looks like in practice. Here it is: a user using the numbers 19, 36, 55, and 7 creates an equation that equals 65. The assistant says, "Let me solve this step by step," so it thinks and goes through a bunch of iterations. This is the internal monologue, and then the final answer is the correct answer.
If you want to play around with Deep Seek R1 and other awesome open-source models, you can do so at AWS's Amazon Bedrock. A special thanks to AWS for sponsoring this video. Deep SR1, both the distilled versions and the full version, are now available at AWS's Amazon Bedrock. Amazon was one of the first companies that had a clear vision of a multimodal future, and they are doubling down on it by adding Deep Seek.
You can now deploy Deep Seek R1 models on both Amazon Bedrock and Amazon SageMaker AI. The broad and deep range of models in Amazon Bedrock empowers you to choose the precise capabilities that best serve your unique needs. It's all about choice, and with Amazon Bedrock guardrails, you can independently evaluate your user interactions with your models—security, safety, control, ease of deployment. Check out Deep Seek R1 on AWS right now; I'll drop links in the description below. Thanks again to AWS.
So here's the recipe we follow: Deep Seek R1 algorithm, given a base language model, prompts, and ground truth reward, we run RL. We apply it to the Countdown game. The results? It just works. Essentially, what they found is that with a good base model and reinforcement learning with a very clear reward function, the model will start to think for itself. That is so crazy to think about, especially because they were able to achieve it on such a narrow domain.
So maybe that's the future. We've talked about test-time training in the past, and more and more people are starting to talk about that right now. That is essentially the ability for the model during inference time to adjust its own model weights based on the prompt. So imagine that combined with test-time reinforcement learning. I don't even know if this is possible; I'm just speculating here. But imagine a combination of these things in which you can have insanely small models solving really hard tasks because they are so specifically tailored to whatever that task is.
So let's walk through the process. The model started with dummy outputs, meaning it was just completely wrong or nonsensical, but gradually developed tactics such as revision and search. This happened naturally—no nudging, no need to say, "Here's how to think through it." It just arrived at this thinking. In the following sample, the model proposed a solution, self-verified, and iteratively revised until it worked.
He also provides a Weights and Biases link, in which he gives you the full experimentation. If you want to dig into that, you can see it yourself. So that's what this looks like. You can actually see the performance over time, each of the steps of the reinforcement learning.
Now, one of the findings is that the base model quality is key. He tried it on 0.5B, 1.5B, 3B, and 7B. The 0.5B guessed a solution and stopped, but once they reached 1.5B and up, the model started to learn to search, to self-verify, and revise its solutions. So that's what we're seeing right here. This maroon color line is the 0.5B model that wasn't good enough to quite reach that thinking capability.
What we see is a little bit of an uptick in performance, and then it flatlines. However, the other ones really jumped up high. As we see, the bigger the model, the better their performance overall. So the starting quality of the model is really important, and either the base model or instruct model works. The instruct model learns faster but converges to about the same performance as the base. The instruct model's outputs are more structured and readable, so extra instruction tuning isn't necessary, which supports R1's design decision.
Now remember, the R10 is not the instruction-tuned model. So with Deep Seek, you got two: you got the base unwieldy R10, and then you also got the just R1 after they kind of put the final touches on it. So that's what we see here in yellow. This is the base model that starts a little worse but pretty much ends up at the same place. The gray starts a little better and again ends up at the same place.
He also found that the actual reinforcement learning algorithm that you use doesn't actually matter all that much, which is really interesting. These are such cool findings, and Deep Seek kicked it all off. That is the power of open source.
Now, obviously, there have been people who already knew about this stuff, but once the Deep Seek paper was published, once people saw how good that model is and could actually read how to reproduce it, everybody started tinkering. The power of open source.
So here we go. He continues with the specific reinforcement learning algorithm: it doesn't matter much. We tried PPO, GRPO, and Prime. Long chain of thought all emerged, and they seem to all work well. We haven't got the time to tune the hyperparameters, so we don't want to make quantitative conclusions about which algorithm works better. The model's reasoning behavior is very task-dependent.
So for Countdown, the model learns to do search and self-verification. However, for number multiplication, the model instead learns to break down the problem using distribution rules and solve it step by step. So interesting! It really changes its behavior based on the task at hand. That's why I go back to what if the future is really, really small models doing all of this reinforcement learning, test-time compute, test-time training at inference time? It's so cool to think about.
We're going to have hundreds, thousands, millions of these tiny models that can just be hyper-tuned to very specific scenarios. Now, a caveat he leaves at the end: this is only validated on the Countdown task and not general reasoning. We are bounded by compute.
Now in this thread, the author answered a bunch of questions, and I want to go over some of those to give you even more detail into how cool this really is.
So here: any insights on why Open Llama achieves high response length but low scores? Open Llama doesn't like to generate EOS tokens, and since we haven't implemented stop after the SL answer, the model often fails to terminate. We didn't report Open Llama's results in the Twitter thread as we believe the results will improve significantly once we fix this problem.
So again, it doesn't really matter what model you're applying this technique to. It doesn't really matter which reinforcement learning technique you're using. Of course, there are going to be little optimizations here and there, but this is so exciting because it's such a generalized emergent capability from reinforcement learning.
All right, one question: what model size and algorithm do the $30 refer to? The 3B model, 3 billion parameter model, PPO reinforcement learning, and it takes 10 H100 hours, which is the $30.
So another question: do you have any observations regarding the chain of thought length changes during training? Is there a plot showcasing chain of thought length increase during training? Great question! Early results show the 3B model initially reduces output length for correct formatting, then increases chain of thought length for better performance. Here is what that looks like: it initially drops the output length down, then learns long chain of thought, and it kind of stays equal—maybe drops a little bit over time.
Now here's an interesting question: do you find the original paper's numbers about the training cost to be plausible? Yes, with mixture of experts and FP8 (floating point 8), that's expected, and that's exactly what Dario, CEO of Anthropic, just said in his paper.
So if you want to play around with this, he open-sourced everything. It's called Tiny Zero. I'll drop a link to the GitHub down below. The dataset, the Weights and Biases run—everything is available. Open source wins! If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.