Transcription
And do we understand firmly which of the three dimensions is best for reasoning? So interconnect the flops; they don't matter as much. Is it memory? Memory, right? Context length. We're going to get into technical stuff, right?
Say there's two articles in this one that I could show. Maybe graphics that might be interesting for you to pull up. Oh, for the listeners, we're looking at the section of 01 inference architecture tokenomics.
How do you want to explain KV Cas before we talk about this? I think like it's better to... okay, yeah, we should get through a lot of specific technical things of Transformers to make this easy for people because it's incredibly important. This changes how models work.
But I think resetting, right? Why is memory so important? It's because so far we've talked about parameter counts, right? And mixture of experts. You can change how many active parameters versus total parameters to embed more data but have less flops.
But more important, you know, another aspect of this humongous revolution in the last handful of years is the Transformer, right? And the attention mechanism. The attention mechanism is that the model understands the relationships between all the words in its context, right?
And that is separate from the parameters themselves, right? And that is something that you must calculate, right? How each token, right? Each word in the context length is relatively connected to each other, right?
And I think Nathan, you should explain KV Cas better. KV Cas is one of the optimizations. So the attention operator has three core things: it's queries, keys, and values. QKV is the thing that goes into this. You'll look at the equation; you see that these matrices are multiplied together.
These words, query, key, and value, come from information retrieval backgrounds where the query is the thing you're trying to get the values for, and you access the keys. The values is... my background's not in information retrieval and things like this; it's just fun to have backlinks.
What effectively happens is that when you're doing these matrix multiplications, you're having matrices that are of the size of the context length, so the number of tokens that you put into the model. The KV cache is effectively some form of compressed representation of all the previous tokens in the model.
So when you're doing this, we talk about autoregressive models. You predict one token at a time. You start with whatever your prompt was. You ask a question like, "Who was the president in 1825?" The model then is going to generate its first token.
For each of these tokens, you're doing the same attention operator where you're multiplying these query-key-value matrices. But the math is very nice so that when you're doing this repeatedly, this KV cache, this key-value operation, you can keep appending the new values to it.
So you keep track of what your previous values you're inferring over in this autoregressive chain. You keep it in memory the whole time, and this is a really crucial thing to manage when serving inference at scale.
There are far bigger experts in this, and there are so many levels of detail that you can go into. Essentially, one of the key drawbacks of the attention operator and the Transformer is that there is a form of quadratic memory cost in proportion to the context length.
So as you put in longer questions, the memory used in order to make that computation is going up in the form of a quadratic. You'll hear about a lot of other language model architectures that are like sub-quadratic or linear attention forms, which is like state space models.
I don't... we don't need to go down all these now. And then there's innovations on attention to make this memory usage and the ability to attend over long contexts much more accurate and high performance. Those innovations are going to help you with... I mean, if you're highly memory constrained, they help with memory constraint and performance.
So if you put in a book into... I think Gemini is the model that has the longest context length that people are using. Gemini is known for 1 million and now 2 million context length. You put a whole book into Gemini, and sometimes it'll draw facts out of it.
It's not perfect; they're getting better. But so there's two things: one, to be able to serve this on the memory level, Google has magic with their TPU stack where they can serve really long contexts.
And then there's also many decisions along the way to actually make long context performance work. This implies the data; there's subtle changes to these computations in attention, and it just changes the architecture.
But serving long contexts is extremely memory constrained, especially when you're making a lot of predictions. I actually don't know why input and output tokens are more expensive, but I think essentially output tokens you have to do more computation because you have to sample from the model.
I can explain that. So today, if you use a model, like you look at an API, OpenAI charges, you know, a certain price per million tokens, right? And that price for input and output tokens is different, right?
And the reason is that when you're inputting a query to the model, right? Let's say you have a book, right? That book, you must now calculate the entire KV cache for, right? This key-value cache.
And so when you do that, that is a parallel operation. All of the tokens can be processed at one time, and therefore you can dramatically reduce how much you're spending, right? The flop requirements for generating a token and an input token are identical, right?
If I input one token or if I generate one token, it's completely identical. I have to go through the model, right? But the difference is that I can do that input, i.e., the prefill, i.e., the prompt, in a batch nature, right?
And therefore, it is all flop. I think the pricing model mostly they use is for input tokens is about 1/4 the price of the output tokens, correct? But then output tokens, the reason why it's so expensive is because I can't do it in parallel, right?
It's autoregressive. Every time I generate a token, I must not only take the entire... I must not only read the whole entire model into memory, right? And activate it, right? Go calculate it to generate the next token.
I also have to read the entire KV cache, and I generate a token, and I append that KV, that one token I generated, and its KV Cas, and then I do it again, right?
And so therefore, this is a non-parallel operation. And this is one where you have to, you know, in the case of prefill or prompt, you pull the whole model in, and you calculate 20,000 tokens at once, right?
These are features that APIs are shipping, which is like prompt caching, pre-filling, because you can drive prices down, and you can make APIs much faster. If you know you're going to keep... if you run a business and you're going to keep passing the same initial content to Cloud's API, you can load that into the Anthropic API and always keep it there.
But it's very different than we're kind of leading to the reasoning models, which we talked... we showed this example earlier and read some of this kind of mumbling stuff.
And what happens is that the output context length is so much higher. I mean, I learned a lot about this from Dylan's work, which is essentially as the output work length gets higher, you're using this... you're writing this quadratic in terms of memory used.
And then the GPUs that we have effectively, you're going to run out of memory, and they're all trying to serve multiple requests at once. So doing this batch processing where not all of the prompts are exactly the same is really complex handling.
And then as context lengths get longer, there's this, like, I think you call it critical batch size, where your ability to serve more users, so how much you can parallelize your inference, plummets because of this long context.
So your memory usage is going way up with these reasoning models, and you still have a lot of users. So effectively, the cost to serve multiplies by a ton.
And we're looking at a plot when the x-axis is sequence length, i.e., how many tokens are being generated. SL prompt, right? So if I put in a book, that's a million tokens, right?
But, you know, if I put in, you know, "The sky is blue," then that's like six tokens or whatever. We should say that what we're calling reasoning and Chain of Thought is extending this sequence length.
It's mostly output tokens. So before, you know, three months ago, whenever 01 launched, all of the use cases for long context length were like, "Let me put a ton of documents in and get an answer out," right?
And it's a single, you know, prefill, compute a lot in parallel, and then output a little bit. Now, with reasoning and agents, this is a very different idea, right?
Now instead, I might have... I might only have, like, "Hey, do this task," or I might have all these documents. But at the end of the day, the model is not just producing a little bit, right?
It's producing tons of information. This Chain of Thought just continues to go and go and go and go. And so the sequence length is effectively that, you know, if it's generated 10,000 tokens, it's 10,000 sequence, right?
And plus whatever you input in the prompt. And so what this chart is showing, and it's a logarithmic chart, right, is, you know, as you grow from 1K to 4K or 4K to 16K, the memory requirements grow so fast for your KV cache that you end up not being able to run a certain number of, you know, your sequence length is capped or the number of users you let say the model.
So this is showing for a 405B model in batch size 64, Llama 31, 405B, yeah. And batch size is crucial to essentially... you want to have a higher batch size to parallelize your throughput, 64 different users at once, right?
And therefore, your serving costs are lower, right? Because the server costs the same, right? This is 8 H100s, roughly $2 an hour per GPU. That's $6 an hour, right?
That is like somewhat of a fixed cost. You can do things to make it lower, of course, but like it's like $16 an hour. Now, how many users can you serve? How many tokens can you generate?
And then you divide the two, and that's your cost, right? And so with reasoning models, this is where a lot of the complexity comes about and why memory is so important.
Because if you have limited amounts of memory, then you can't serve so many users. If you have limited amounts of memory, your serving speeds get lower, right?
And so your costs get a lot worse. Because all of a sudden, if I was used to, "Hey, on this $16 an hour server, I'm serving Llama 405B," or if I'm serving, you know, Deep Seek V3, and it's all chat-style applications, i.e., we're just chatting, the sequence lengths are thousand, few thousand, right?
You know, when you use a language model, it's a few thousand context length most of the time. Sometimes you're dropping a big document, but then you process it, you get your answer, you throw it away, right?
You move on to the next thing, right? Whereas with reasoning, I'm now generating tens of thousands of tokens in sequence, right?
And so this memory, this KV cache has to stay resident, and you have to keep loading it. You have to keep it in memory constantly. And now this butts out other users, right?
If there's now a reasoning task, right? And the model is capable of reasoning, then all of a sudden, that memory pressure means that I can't serve as many users simultaneously.
Let's go into Deep Seek again. So we're in the post-Deep Seek R1 time, I think. And what we're... there's two sides to this market. Watching how hard it is to serve it.
On one side, we're going to talk about Deep Seek themselves. They now have a chat app that got to number one on the App Store. Disclaimer: number one on the App Store is measured by velocity, so it's not necessarily saying that more people have the Deep Seek G than GPT app.
But it is still remarkable. Claude has never hit number one in the App Store, even though everyone in San Francisco was like, "Oh my God, you got to use CLA, don't use ChatGPT."
So Deep Seek hit this. They also launched an API product recently where you can ping their API and get these super long responses for R1. Out in... at the same time as these are out, we'll get to what's happened to them.
Because the model weights for Deep Seek R1 are openly available, and the license is very friendly, the MIT license, commercially available. All of these midsize companies and big companies are trying to be first to serve R1 to their users.
We were trying to evaluate R1 because we have really similar research going on. We released the model, and we're trying to compare to it. And out of all the companies that are quote-unquote serving R1, and they're doing it at prices that are way higher than the Deep Seek API, most of them barely work, and the throughput is really low.
To give context, right? Everyone... one of the parts of, like, freaking this out was like China reached capabilities. The other aspect is they did it so cheap, right?
And so cheap. We kind of talked about on the training side why it was so cheap. Talk about why it's so cheap on the... it works well, and it's cheap. Why is R1 so damn cheap?
So I think there's a couple factors here, right? One is that they do have model architecture innovations, right? This MLA, this new attention that they've done is different than the attention from "Attention is All You Need," the Transformer attention, right?
Now, others have already innovated. There's a lot of work like MQ, AGQA, local-global, all these different innovations that try to bend the curve, right? It's still quadratic, but the constant is now smaller.
Related to our previous discussion, this multi-head lat and attention can save about 80 to 90% in memory from the attention mechanism, which helps especially along context.
It's 80 to 90% versus the original, but then versus what people are actually doing, it's still an innovation. This 80 to 90% doesn't say that the whole model is 80 to 90% cheaper; it's just one part of it.
Well, and not just that, right? Like other people have implemented techniques like local-global sliding window and GQ, MQ, that... but anyways, like Deep Seek has their attention mechanism as a true, real innovation.
They did tons of experimentation, and this dramatically reduces the memory pressure. It's still there, right? It's still quadratic; it's still attention; it's still quadratic.
It's just dramatically reduced it relative to prior forms. All right, that's the memory pressure. I should say, in case people don't know, R1 is 27 times cheaper than 01.
We think that OpenAI had a large margin built in, okay? So that's there. There are multiple factors we should break down the factors.
I think it's two bucks per million token output for R1 and $60 per million token output for 01. Yeah, let's look at this.
So I think this is very important, right? OpenAI is... you know, that drastic gap between Deep Seek in pricing, but Deep Seek is offering the same model because they open-weighted it to everyone else for a very similar, like, much lower price than what others are able to serve it for, right?
So there's two factors here, right? Their model is cheaper, right? It is 27 times cheaper. Well, I don't remember the number exactly off the top of my head.
So we're looking at a graphic that's showing different places serving V3, Deep Seek V3, which is similar to Deep Seek R1, and there's a vast difference in serving cost, right?
In serving cost, and what explains that difference? And so, like, part of it is OpenAI has a fantastic margin, right? They're serving... when they're doing inference, their gross margins are north of 75%, right?
So that's a 4 to 5x factor right there of the cost difference is that OpenAI is just making crazy amounts of money because they're the only one with a capability. Do they need that money? Are they using it for R&D?
They're losing money, obviously, as a company because they spend so much on training, right? So the inference itself is a very high margin, but it doesn't recoup the cost of everything else they're doing, okay?
So yes, they need that money because the revenue and margins pay for continuing to build the next thing, right? As long as raising more money.
So the suggestion is that Deep Seek is like really bleeding out money. Well, so here's one thing, right? We'll get to this in a second, but like Deep Seek doesn't have any capacity to actually serve the model.
They stopped signups. The ability to use it is like non-existent now, right? For most people because so many people are trying to use it, they just don't have the GPUs to serve it, right?
OpenAI has hundreds of thousands of GPUs between them and Microsoft to serve their models. Deep Seek has a factor of much lower, right?
You know, even if you believe our research, which is 50,000 GPUs, and a portion of those are for research, a portion of those are for the hedge fund, right? They still have nowhere close to the GPU volumes and capacity to serve the model, right? At scale.
So it is cheaper. A part of that is OpenAI making a ton of money. Is Deep Seek making money on their API? Unknown. I don't actually think so.
And part of that is this chart, right? Look at all the other providers, right? Together AI, Fireworks AI are very high-end companies, right? XMeta, Together AI is Trew and the inventor of, like, flash attention, right?
Which is a huge efficiency technique, right? They're very efficient, good companies, and they're serving... and I do know those companies make money, right? Not tons of money in inference, but they make money.
And so they're serving at like a 5 to 7x difference in cost, right? And so, you know, now when you equate, okay, OpenAI is making tons of money, that's like a 5x difference.
And the companies that are trying to make money for this model is like a 5x difference. There is still a gap, right? There's still a gap, and that is just Deep Seek being really freaking good, right?
The model architecture, MLA, the way they did all these things, there is like legitimate just efficiency difference. All their low-level libraries that we talked about in training, some of them probably translate to inference, and those weren't released.
So we may go a bit into conspiracy land. Is it possible the Chinese government is subsidizing Deep Seek? I actually don't think they are.
I think when you look at the Chinese labs, there's Huawei has a lab, Moonshot AI, there's a couple other labs out there that are really close with the government. And then there's labs like Alibaba and Deep Seek, which are not close with the government.
And, you know, we talked about this, this CEO, this reverent figure who's quite different, who has like... sounds awesome, very different viewpoints based on the Chinese interviews that are translated than what the CCP might necessarily want.
Now, to be clear, right? Does he have a loss leader because he can fund it through his hedge fund? Yeah, sure. So the hedge fund might be subsidizing it, yes.
I mean, they absolutely did, right? Because Deep Seek has not raised much money. They're now trying to raise a round in China, but they have not raised money historically.
It's all just been funded by the hedge fund, and he owns like over half the company, like 50-60% of the company is owned by him. Some of the interviews, there's discussion on how like doing this is a recruiting tool.
You see this at the American companies too. It's like having GPUs, recruiting tool, being at the cutting edge of AI, recruiting tool, open sourcing, open sourcing, recruiting tool.
So much talent. They were so far behind, and they got so much talent because they just open-sourced stuff.
More conspiracy thoughts: is it possible since there are hedge funds that they timed everything with this release and the pricing, and they have shorted Nvidia stock and stock of USA companies and released it with star-like just perfect timing to be able to make money?
Like they've released it on inauguration day. They know the international... what is on the international calendar. But I mean, I don't expect them to.
If you listen to their motivations for AI, it's like... they released V3 on like December 26th. Like who releases the day after Christmas? No one looks, right?
They had released the papers before this, right? The V3 paper and the R1 paper, so people had been looking at it and be like, "Wow."
And then they just released the V R1 model. I think they're just shipping as fast as they can, and like who cares about Christmas? Who cares about, you know, getting it out before Chinese New Year, right?
Obviously, which just happened. I don't think they actually were like timing the market or trying to make the biggest splash possible. I think they're just like shipping.
I think that's one of their big advantages. We know that a lot of the American companies are very invested in safety, and that is the central culture of a place like Anthropic.
And I think Anthropic sounds like a wonderful place to work, but if safety is your number one goal, it takes way longer to get artifacts out.
That's why Anthropic is not open-sourcing things; that's their claims. But there's reviews internally at Anthropic.
Ra mentions things to international governments. There's been news of how Anthropic has done pre-release testing with the UK Safety Institute.
All of these things add inertia to the process of getting things out. And we're on this trend line where progress is very high.
So if you reduce the time from when your model is done training, you run a Vows, that's good. You want to get it out as soon as possible to maximize the perceived quality of your outputs.
Deep does this so well. Dario explicitly said Claude 3.5 Sonet was trained like 9 months or 10 months ago, 9 to 10 months ago.
And I think it took them another handful of months to release it, right? So it's like there is a significant gap here, right?
And especially with reasoning models, the word in the San Francisco street is that like Anthropic has a better model than 03, right? And they won't release it.
Why? Because chains of thought are scary, right? And they are legitimately scary, right? If you look at R1, it flips back and forth between Chinese and English.
Sometimes it's gibberish, and then the right answer comes out, right? And like for you and I, it's like great, great. This is why people are infatuated, right?
You're like, "You're telling me this is a high-value thing, and it works, and it's doing this. It's amazing." I mean, you talked about that sort of like Chain of Thought for that philosophical thing, which is not something they trained to be philosophically good.
It's just sort of an artifact of the Chain of Thought training it did. But like that's super important in that like can I inspect your mind and what you're thinking right now? No.
And so I don't know if you're lying to my face. And Chain of Thought models are that way, right? Like this is a true, quote-unquote, risk between, you know, a chat application where, hey, I asked the model to say, you know, bad words or whatever, or how to make anthrax, and it tells me that's unsafe.
Sure, but that's something I can get out relatively easily. What if I tell the AI to do a task, and then it does the task all of a sudden randomly in a way that I don't want it, right?
And now that has like much more task versus like response is very different, right? So the bar for safety is much higher, at least this is Anthropic's case, right?
Like for Deep Seek, they're like, "Ship."