Transcription
A new research paper was just dropped showing how large language models can think internally before ever outputting a single token. You're probably used to hearing about Chain of Thought with test time compute. These are the new thinking models, and the thinking happens where you can actually read the internal monologue.
However, with this new approach, the thinking actually happens in latent space, so inside the model, before it ever outputs a single token. This is very different from Chain of Thought and has the potential to tackle problems that can't be described with words alone.
So before I get into this paper, I want to show you what Yan Laon, Chief AI Scientist at Meta, has said about the limitations of large language models. He's one of the leading minds in AI and one of the loudest voices saying that large language models cannot reason, cannot plan like humans do. This is due to the limitations that describing things with language alone has.
Let me show you this first clip of an interview on The Lex Friedman Podcast, where he describes exactly this:
"If you have a model of this type, you can use it for planning. So now you can do what LLMs cannot do, which is planning what you're going to do so as to arrive at a particular outcome or satisfy a particular objective. I can predict that if I have an object like this, right, and I open my hand, it's going to fall, right? And if I push it with a particular force on the table, it's going to move. If I push the table itself, it's probably not going to move with the same force. So we have this internal model of the world in our mind, which allows us to plan sequences of actions to arrive at a particular goal."
All right, so what he's essentially saying is that to really have true reasoning models, models that can actually understand the world around us, we need more than just what language can describe. Of course, he has his own thoughts on what it'll take, but that's not the purpose of today's video. His argument is that language models alone are not enough to reach true planning and true reasoning.
Now keep in mind, his interview was recorded before the entire thinking model phenomenon really went mainstream. Now, of course, he probably already knew about thinking models, test time compute, Chain of Thought. He probably knew about all of this and still believes that's not enough.
Now for me, when I look at the Chain of Thought, when I see the actual reasoning steps that a model is going through before it gives you the final output, that's pretty convincing. But he is not convinced.
Let me show you where we're fooled by their fluency. Right? We just assume that if a system is fluent in manipulating language, then it has all the characteristics of human intelligence. But that impression is false.
So he says it right there: we are being fooled by the fact that these large language models are incredibly good at manipulating language, at telling us what we want to know. But that in itself is not enough for true reasoning and true logic.
But again, I go back to actually being able to read the Chain of Thought and see the reasoning patterns. Now, of course, there's a chance he still might be right. The Chain of Thought is really just language manipulation, so who knows?
Let me show you one more clip from that interview, and then we're going to get into the paper.
"There's ample evidence that we're not going to be able to learn good representations of the real world using generative models. So I'm telling people, everybody is talking about generative AI. If you're really interested in human-level AI, abandon the idea of generative AI."
All right, so there he said it: abandon the idea of generative AI, which is very surprising given all of the advancements as of late. He is in the minority in this thinking. All of the major AI companies out there obviously believe that we're going to be able to hit AGI and ASI, true reasoning, true logic, true representations of the real world using language models alone.
It seems to be that Chain of Thought and test time compute is the last lever necessary to reach that, but again, he doesn't.
Now with that, let me show you this paper because it actually might be the thing that he's been talking about we're missing from large language models.
So here's the paper: "Scaling Up Test Time Compute with Latent Reasoning: A Recurrent Depth Approach."
Now what the authors of this paper have figured out is that you can actually have a model that does the thinking in latent space, so inside the model, before ever outputting a token. This might be the missing piece that Yan Laon thinks is necessary to really have a true reasoning and a true planning model.
This type of architecture allows the model to think and to scale up that thinking internally, and I'm going to explain how it works and what they've discovered.
So in the abstract, this novel language model architecture is capable of scaling test time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test time.
So they have this hidden block inside the model which can infinitely think and go deeper until it comes up with the final answer, and all of this happens at test time. That's the important part, and that's in contrast to mainstream reasoning models that scale up compute by producing more tokens.
So remember, kind of the current wave of thinking models use Chain of Thought at test time, and the Chain of Thought is literally just outputting tokens, correcting itself, reflecting back on those tokens. The number of tokens output in these thinking models is far more than the traditional non-thinking models, and there are a lot of benefits with this new approach.
So it doesn't require any specialized training data, and I'll get to what that means in a minute. You don't need massive context windows as with Chain of Thought and the current thinking models that require huge context windows.
Most importantly, and most interestingly, it can capture different types of reasoning that are not easily represented with words alone. Again referencing Yan Laon's main argument against large language models is that you can't represent the real world completely using words, and so maybe this is the solution.
So they created a proof of concept 3.5 billion parameter model that shows some of these techniques in action.
So let's take a step back. Let's talk about how this all works and what it means.
So let's talk about how human thinking works. A substantial amount of thought happens through complex recurrent firing patterns in the brain before the first word of an answer is uttered.
Now if we think about current Chain of Thought, it is outputting immediately. Even with Chain of Thought, the thinking happens with output. You're thinking out loud, you're having to say these words out loud, or you're thinking them in your mind, but you're still having the language as the kind of first thing that happens in the thinking process.
But that's not always what happens. There's a lot of thinking that happens before you even use a single word. You can conceptualize different topics, you can conceptualize different situations without actually saying a single word in your head and without verbalizing any of those words.
In fact, it kind of reminds me of people who have no internal monologue. How do they think? Well, they can, and they don't use any language.
So it says more recently researchers have explored ways to enhance the reasoning capability of models by scaling test time computation. Those are the thinking models: Deep Seek, R1, 01, 03.
The mainstream approach involves post-training on long Chain of Thought examples to develop the model's ability to verbalize immediate calculations in its context window and thereby externalize thoughts.
Now what it's saying is you need a lot of examples of how to think to train the models to think.
Now one thing I'm a little confused about is when I think about the Berkeley PhD student who was able to essentially elicit the thinking behavior in a model for just $30, and he didn't have a bunch of examples of how to think. He used reinforcement learning with verifiable rewards, which doesn't require a bunch of examples of how to think.
So maybe this paper was written before that example. However, the constraint that expensive internal reasoning must always be projected down to a single verbalized next token appears wasteful.
It is plausible that models could be more competent if they were able to natively think in their continuous latent space.
So again, internal thinking, no language, just being able to think about a problem before that first token comes out. This is not a new discovery. This idea is foundational to machine learning and has been rediscovered in every decade, for example, as recurrent neural networks, diffusion models, and as universal or looped transformers.
So again, it's not new, but the authors are saying it is rediscovered with every kind of generation of AI.
All right, so what is the gist of what's happening at test time? The model can improve its performance through recurrent reasoning in latent space.
Recurrent reasoning means it can kind of think and think and think over and over again about the same problem, and in latent space means internally, without having to output those tokens, enabling it to compete with other open-source models that benefit from more parameters in training data.
So it is more efficient, this new technique.
So let's talk about some of the benefits. Recurrent layers enable a transformer model to perform arbitrarily more computations before emitting a token.
So some of the advantages: latent reasoning does not require the construction of bespoke training data.
Now I just talked about that. Traditionally, when making a model become a thinking model, you need a bunch of examples about how to think. But we've already seen smaller examples of reinforcement learning with verifiable rewards that get a model to start thinking without explicitly teaching it how to think.
And that's different than what they're saying here. We don't know if that'll scale up, but we've already seen multiple examples.
Next, latent reasoning models require less memory for training and inference than Chain of Thought reasoning models.
Now that is because Chain of Thought in the traditional thinking models take a lot of tokens, which means that the context window has to be really big, and that's computationally very expensive.
Recurrent depth networks perform more flops, which is just think about it as compute per parameter than standard transformers, significantly reducing communication cost between accelerators at scale.
Accelerators meaning GPUs. So they're basically saying that they're able to utilize a single GPU much more than they would need to multiple GPUs connected together.
And then next, by constructing an architecture that is compute-heavy and small in parameter count, we hope to set a strong prior towards models that solve problems by thinking, by learning meta strategies, logic, and abstraction instead of memorizing.
So this is an important factor as well.
Now with traditional large language models, it is memorizing information, and the ability to generalize outside of its training data is still under discussion and argument. A lot of people are saying that models cannot generalize outside of their training data, which means it's not truly artificial general intelligence.
However, with this new technique, they might be able to.
All right, so let's look at how it works. So this is at test time. You have the input "hello," then you have the recurrent block.
So this is the iterative stage, and it can go on forever where it's thinking, thinking, and thinking about what to do next. What do I put next? And then finally, you get the actual output of "world" because it's "hello world."
Now these green blocks are the thinking part of the model, and it happens before a single token is output.
So you can see that this is "world," the first token being output, and all of the thinking happens before that.
Let me show you the proof that it actually does work. What we're looking at here in this graph on the Y-axis is performance. Think about just how good the output is, and on the X-axis, we have the recurrence at test time, basically the amount of thinking it does at test time before, again, remember, before a token is actually output.
You can see on the different evals we have H-Swag, GSM, AK, Human Eval, and you can see as it thinks 4, 8, 16, 32, 64 times, the performance continues during that thinking time.
So it shows right there, there is the proof that the more thinking it does inside the model without generating a token improves the performance of the next generated token.
I should actually say the total answer in general because these are against different benchmarks, and what we're seeing here is the more training tokens going into these models, the better they perform also.
But that's kind of how all transformer models work. That is the scaling law, and we're seeing the same thing here: 100, 200, 300, all the way up to 800 billion tokens, and it continues to do better.
Here's another interesting thing: it can actually decide how much compute to use depending on the task at hand.
Just like Chain of Thought, it can think longer, it can think shorter, and that's actually a really good way to optimize for efficiency.
So that's what we're seeing here in figure 10. We have, from left to right, high school math, philosophy, logical fallacies, and moral scenarios.
So for high school math, on the X-axis, we have the amount of compute used. Just think about the amount of compute, and on the Y-axis, what we're seeing is how often the model needs that many steps to actually accomplish the correct answer.
What we're seeing here is that for high school math, it doesn't need a lot of steps. It gets to a really good answer pretty quickly.
For philosophy, it starts to need more. For logical fallacies, it needs more. For moral scenarios, it needs more.
So for simpler tasks, high school math, it doesn't need as much. For more complex tasks, it needs more.
That's exactly what you would do as a human. If you're thinking about a problem, if it's a simpler problem, you don't think as long. If it's a more complicated problem, you probably will think more.
Now here's the interesting thing that it also says: using this technique of latent space thinking does not negate the use of Chain of Thought at test time and actually using tokens.
So you can combine these different things. You can have some initial thinking without generating any tokens, and then when you finally do, you can generate those tokens to use additional thinking.
Take a second to think about how you solve really hard problems. You probably think about it in your head, you write some things down, think about it some more, iterate on what was written down, and continue that process until you solve the problem.
I think with these two techniques, latent-based thinking and actual token-based thinking, these two things together can be really powerful and mirror how humans think.
So I found this fascinating. It's a proof of concept. You can download the model, you can try it out yourself.
I hope you enjoyed this because I just find all of these papers so very fascinating. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.