Transcription
Test time compute is the most important breakthrough since Transformers changed the world in 2017.
Allowing AI models to think long term makes them much smarter. But not only that, this might be a significantly bigger market than pre-training ever was.
So, I've collected a bunch of proof to show you how big inference time scaling really will be. But first, let me explain what it exactly is.
In the most simple terms, it's allowing models to think long term. It's allowing them to use more tokens during inference time. Essentially, when you prompt the model to actually think about things rather than just giving you the first response it comes up with.
So, here's a paper from Google DeepMind from just a few months ago. Scaling LLM test time compute optimally can be more effective than scaling model parameters.
This paper sets out to prove that we can actually get much more out of scaling up the thinking ability of models. Enabling LLMs to improve their outputs by using more test time computation is a critical step towards building generally self-improving agents that can operate on open-ended natural language.
We study the scaling of inference time computation in LLMs, focusing on answering the question: if an LLM is allowed to use a fixed but non-trivial amount of inference time compute, how much can it improve its performance on a challenging prompt?
Well, it turns out a lot. This was not only shown by the 01 model, which was incredible, but now the 03 model, which scored incredibly well on many different frontier benchmarks, including the Ark AGI Benchmark.
I'll show you that in a moment. This segment is sponsored by Synf Flow AI. Synf Flow is your no-code platform for building voice AI agents. Everything from lead qualification, booking appointments, follow-ups, customer support FAQs, and more.
So, if you want to scale your company, scale the personal interaction between you and your customer. Synf Flow is a great option. Imagine having an AI receptionist that can handle the inbound call to your business with a really personal touch, but not using any of your time.
They can greet callers, direct them to the right person in your company, all while operating 24/7. They also support automated appointment booking, so connect it to your calendar and you can allow Synf Flow's AI agents to book on your behalf.
Check this out. Here's a demo of me working with one of their agents.
"Hello, I'm Emma from Southwest Mortgage Corporation. I'd love to discuss your mortgage needs. Do you have a moment to talk?"
"Yeah, I don't, but I'd like to set an appointment for 5:00 p.m. on Friday."
"Got it! I'll set up an appointment for you at 5:00 p.m. on Friday. Could you please provide your full name and the best contact details?"
"Matthew Burman. Matt at forwardfuture.com is my email address."
"Thanks, Matthew! I'll send a confirmation to Matt at forwardfuture.com."
"All right, thank you!"
All right, so it's that easy. So definitely check out Synf Flow. Use my link down below so they know I sent you and check out what it can do for your business.
Now, back to the video. So why does test time compute work so well? Well, let's think about how humans work.
Humans tend to think for longer on difficult problems to reliably improve their decisions. When somebody asks you a difficult math question, or a logic question, or really any question that requires any kind of thinking, you are not just spitting out the first thing that pops in your head.
You're thinking about it. You're thinking about different permutations of the answer, and then you're trying to come up with the best answer before stating it.
So can language models do this? Can they employ the same tactic? Well, yes, they can. Such a capability at test time also has the potential to unlock new avenues in agentic and reasoning tasks.
How does it work? Right now, there are a few techniques for doing test time compute. One is called best of n sampling. That means coming up with a bunch of candidate responses and trying to figure out which one is best.
Each of those candidate responses is different, as described in the paper, sampling and outputs in parallel from a base LLM and selecting the one that scores the highest per a learned verifier or a reward model.
And that brings us to process-based reward models. Let's talk about the difference between outcome reward models and process reward models because it is critical in understanding how these thinking models work and how the thinking works in general.
With outcome reward models, you basically reward the model for getting an answer right or wrong, but it doesn't really matter how they arrived at the answer. It just matters whether they got it right or wrong.
Now, there are some inherent problems with that, which will become obvious when I describe process reward models. In process reward models, you're actually rewarding the model for each step along the way to arriving at a solution.
So if a model gets the first four out of six steps right, it should know that it got those first four right rather than just thinking it got the whole thing wrong. Because then it could, instead of throwing the whole thing out, keep the first four steps and try to figure out what those last two steps would be.
And it gets rewarded as such, as we see here. So on the left, we see OMM gets the whole thing wrong, thus it's wrong. On the right, it does get the final answer wrong, but it got a few steps right before it started getting the final steps wrong.
And that is really important to understand. There are a number of ways to allow these models to think. Let's talk about a few of them.
So search methods against a process reward model. We have the best of n weighted, so basically generate a bunch of answers, figure out which one the model thinks is right, and then choose that one. Very simple.
Then you have beam search, so it optimizes the PRM by searching over its per-step predictions. Sample the initial predictions, then score the generated steps, then filter for the highest scored steps.
Then for each candidate, come up with next steps and then do the same process. Finally, we have look ahead search, so it modifies how beam search evaluates individual steps.
It uses look ahead rollouts to improve the accuracy of the PRM's value estimation in each step of the search process.
So you can see each one right here: best of n, generate the answers, choose the best one; beam search, generate steps, choose the best step at each step; and then look ahead search, basically a little bit of a modification of beam search, generate the steps, look ahead, come back, and it's just more iterative in its approach.
All right, so now let's zoom out a bit. We remember how important the 01 launch was. It really showed the world that inference time scaling was a thing and really could be powerful.
Then 03 came, and it really blew everybody's mind with how powerful it truly was and how much it can be scaled. More importantly, okay, so look at this benchmark.
First of all, the Ark AGI Benchmark is a set of questions that are easy for humans to understand but very difficult for AI to understand, at least so far.
The important thing to note is that even the best models prior to these thinking models were not doing very well. We're talking single-digit percentages, maybe up into the teens and the 20s, but not really much better than that.
Then 01 came along and did much, much better. What we're seeing here is in the high 20s, low 30s, and then 03 came.
Now, here's something to keep in mind. Here's 03 low, meaning it didn't use a lot of thinking, but it still thought. Then here's 03 high when it was able to use essentially unlimited thinking.
What we see is the more thinking, the better it can do, but it's still very, very expensive. That is something key to remember.
To get from 76 to 88, I believe the 88% score cost hundreds of thousands of dollars to run because it used so many tokens over such a long period of time.
So here's Lisa Su, the CEO of AMD, also by the way, cousin of Jensen Wang, the CEO of Nvidia, which is kind of crazy to think about.
But she says inference time scaling will be a significantly larger market than pre-training. What we've done with MI30 is we've built an exceptional product for inference, especially large language model inference.
So when we look going forward, much of what work has done right now is companies kind of training and deciding what their models are going to be.
But going forward, we actually think inference is going to be a larger market, and that plays well into some of what we've designed MI3004.
And she's not the only one. Her cousin Jensen just gave a keynote at CES 2025. I got to watch it; it was pretty awesome.
In it, he talks about this new scaling law, and he also talks about how much bigger of a market inference time compute will be.
In this first clip, he's going to talk about the three scaling laws, the third being test time compute. The industry is chasing and racing to scale artificial intelligence.
The scaling laws say that the more data you have, the training data that you have, the larger model that you have, and the more compute that you apply to it, therefore the more effective or the more capable your model will become.
But there are, in fact, two other scaling laws that have now emerged. The second scaling law is post-training scaling law.
Post-training scaling law uses technologies and techniques like reinforcement learning and human feedback. Basically, the AI produces and generates answers based on a human query.
It's much more complicated than that, but the reinforcement learning system, with a fair number of very high-quality prompts, causes the AI to refine its skills.
This third scaling law has to do with what's called test time scaling. Test time scaling is basically when you're using the AI. The AI has the ability to now apply a different resource allocation.
Instead of improving its parameters, now it's focused on deciding how much computation to use to produce the answers it wants to produce.
Instead of a direct inference or one-shot answer, you might reason about it. You might break down the problem into multiple steps.
You might generate multiple ideas and evaluate which one of the ideas that you generated was the best one.
In this clip, he's going to talk about how inference is such an important part of their strategy as a company and why he thinks it's going to be a massive market.
Well, the amount of computation that we need, of course, is incredible. We would like, in fact, that society has the ability to scale the amount of computation to produce more and more novel and better intelligence.
Intelligence, of course, is the most valuable asset that we have, and it can be applied to solve a lot of very challenging problems.
Scaling law is driving enormous demand for NVIDIA computing, but again, not alone. Here's a third person, Jonathan Ross, the CEO of Grok.
I'm an investor. He originally thought the market for inference would be 10 to 20 times the size of pre-training, but now he actually thinks it's much bigger, and I tend to agree.
It is going to be much bigger. Now, here's the thing about Grok chips. Grok focuses on inference; they are an inference company.
So they're well positioned to take advantage of this new scaling law. But here's the thing: it's really expensive currently to do all of this test time compute.
Tokens are relatively cheap, but when you continue to stack them on top of each other and really allow a model to think for minutes, hours, days, weeks, maybe even months, that becomes really expensive.
But just like every other technology, inference is going to get cheaper as continual investments flood into the AI market. New technologies, new innovations happen with chips; this inference will get cheaper.
But what happens when it gets cheaper? Does that mean the market is going to shrink? Because if everybody's buying the same amount of compute but it's a lot cheaper, then the total amount of spend is a lot less?
Well, no. Speaking of Jonathan Ross, he actually covered this exact point, which I thought was incredibly interesting.
When you make compute cheaper, do people buy more? Yes, it's called Jevons Paradox, and it's a big part of our business thesis.
So let's look at an example. In the 1860s, an Englishman wrote a treatise on coal, where he noted that every time steam engines got more efficient, people bought more coal.
It's a paradox because if they're more efficient, why are they buying more? Well, the answer is when you make a steam engine more efficient, it reduces the OPEX, operating expenses.
When you reduce the OPEX, it increases the number of activities that are profitable. Therefore, people will do more things using steam engines, and coal demand rises.
The same paradox applies to compute. So let me just break it down really quickly.
Let's say you have this technology, and it's really expensive, but it's useful for this one use case because you can go ahead and charge your customers a lot for it.
Well, if all of a sudden that technology became orders of magnitude less expensive, the amount of use cases that you can apply it to increases dramatically.
That means you're really going to be spending a lot more, even though each unit of that technology is a lot cheaper because you have a lot more use cases that you can go ahead and resell to your customers.
So he continues: over the last 60 years, almost like clockwork, every decade compute gets about a thousand times cheaper.
People buy 100,000 times as much of it, spending 100 times more overall. Our mission at Grok is to drive the cost of compute towards zero.
The cheaper we make the compute shoot, the more people spend. Over the next decade, we want to drive the cost down for generative AI a thousand times, making a lot more activities profitable.
We think it will cause a 100 times spend increase. So again, this market is going to be huge. These models are going to get better; they're going to be able to think much more long term.
At this point, you might be thinking, "Matt, stop! It's too much inference time." Well, I got one more thing to show you.
This is a paper that was literally just dropped by Google DeepMind one day ago, January 6, 2025: Inference Time Scaling for Diffusion Models.
So now we have another entire area of AI that just unlocked the ability to do inference time scaling. Diffusion models are the models that generate images, text to image, image to image, different things like that.
Now they're applying the same techniques, basically allowing the models to think at inference time, but this time it's thinking about what image to generate.
Looking at this paper, recent research has begun to explore inference time scaling behavior in large language models, revealing how performance can further improve with additional computation during inference.
Unlike LLMs, diffusion models inherently possess the flexibility to adjust inference time computation via the number of denoising steps.
That is like the very highest level, simplest explanation of how diffusion models work. It starts with a very rough image and then continuously denoises it, basically removes the fuzziness or the unknown parts of it.
Although the performance gains typically flatten out after a few dozen, in this work, in this paper, we explore the inference time scaling behavior of diffusion models beyond increasing denoising steps and investigate how the generation performance can further improve with increased computation.
So again, more test time compute, more thinking. Bcloud had a really good breakdown of this paper, so let me just read it to you quickly.
Diffusion models typically improve sample quality by increasing the number of denoising steps, but those gains plateau, which we just talked about.
Now this paper is proposing giving these models the ability to use more compute during this diffusion process.
The framework has two main components: verifiers, models that evaluate the quality of generated samples, very similar to everything we've talked about with traditional LLMs, and algorithms, methods to search for better noise candidates based on verifier feedback.
So it's all the same process: generate a bunch of samples, choose the best one. And what we're seeing is the results speak for themselves.
They actually generate much higher quality results when given the time to actually have all of this additional compute.
So what does all this mean? Well, if you're thinking about where to spend your time learning about AI, if you're thinking about where the next innovations in AI are going to come from, well, inference time is the place to look.
If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.