Transcription
Deepseek just broke, obliterated Gemini 3 Pro, GPT5 High, and Sonnet 4.5 on these three benchmarks, and it's getting pretty close or surpassing on other benchmarks as well. It seems like open-source models are not only catching up, but surpassing uh, closed-source models in many areas.
There are two models released: Deepseek V3.2 Thinking and Deepseek V3.2 Into Spatial. I'm not sure how to read this. So, this one is raw power. So, long reasoning thoughts, a lot of tokens, and it's winning everywhere. And this one is more efficient. So, shorter reasoning thoughts for higher speed and cheaper cost.
Let's actually talk about the AI research behind this immediately because other channels will be testing it, uh, doing prompts. Because on my channel, I want to make all of you AI researchers. I want to show you how to write AI papers yourselves, how to come up with stuff like DeepSeek yourself. By the way, if you want to learn how to build a large language model from scratch, check out this uh latest video. I'm going to leave the link below. And now I will presume you understand the basics of large language models so I can explain this research.
Quick update, I'm going to bring you on my channel the best AI research experience in the world from top universities. In a few weeks, I'm going to move here next to Uda Co. This place is absolutely stacked with the best researchers in the world, near Chinua, best university, uh, Peking University, Renmin. These universities are comparable to Harvard and MIT. And I will do research with my friends there who are AI PhD students. And I'm going to make full courses and tutorials and videos on my channel, uh, writing AI research papers from scratch so you can get world-class AI research PhD education on my channel. I also know some people from Moonshot Kimmy. Maybe we can even get DeepSeek people to do research with us. I don't know if they will, but top university students and PhD students and professors definitely will. So follow my channel. In a few weeks, we are setting off.
The first component is this DeepSeek sparse attention where this part of the large language model is going to choose which tokens to pay attention to. So it will not pay attention to every single previous token, every single previous word, but it will choose 2,000 most important tokens and pay attention only to them. So this is very good for long context when you have a lot of text because in traditional standard transformers, every token needs to attend to every other previous token. So you can see that how compute grows quadratically there. But here, every token just attends to a maximum of 2,000 previous tokens. So it actually grows linearly the necessary compute.
They also worked on scaling reinforcement learning reasoning training to even more compute. And nowadays, companies are having completely separate uh training at the end for agentic tool use. And here they generate a lot of uh data for training agentic tool use. So synthetic data. First, they say that even though open-source models like Minimax, Moonshot, Jerput, that's Zi, they are continuing to improve, closed-source models are actually accelerating their improvement recently. So Anthropic, DeepMind, OpenAI has accelerated at a significantly steeper rate, and the gap seems to be widening where proprietary models demonstrate increasingly superior capabilities in complex tasks. Also, we can add that maybe they mean maybe by agentic reasoning or interleaved thinking, interleaved tool usage. Because I do feel like at some areas, closed-source models are converging and uh, closing the gap, while it might be that in other areas, they are the gap is widening.
They identified three main issues that open-source models have. First of all, the vanilla attention where every token interacts with every token. Maybe there are some small efficiency tricks, but it still grows quadratically with the sequence length. And it's quite possible that closed-source companies have better solutions to this. And this is also a big issue for post-training for thinking because thinking requires a lot of tokens. Also, open-source models are not given enough compute for post-training, for reasoning training. And open-source models are lagging behind in agentic use cases where generalization and instruction following are necessary. Although Kimmy K2 Thinking might be uh, very good at this, I, as I understand.
To address these issues, they first developed DeepSeek sparse attention, which they already released in a previous model. So here, instead of tokens looking at every single previous token, they just look at a maximum of 2,000 previous most important tokens. So no matter how long the sequence, they will only look at a maximum of 2,000 previous tokens. So you see that this scales pretty well linearly. They also scale compute for the reinforcement learning reasoning training to up to 10% or more of the pre-training compute, which is a big increase in compute for the reasoning for thinking training. And thirdly, it's agentic tool use uh training where they generate massive amount of synthetic data in 1,800 distinct environments and 85,000 complex prompts.
And check this out, they have this sparse attention. So they're not paying so much for generating new tokens because it's only looking at the past 2,000 tokens or whatever the number is. And now they can use many tokens for reasoning and thinking. So it builds up on each other, cheaper uh, token generation. And so it allows for better reasoning training with a lot more tokens. And to make an even better model, they relaxed length constraints on reasoning. And this way, it reasons on par of Gemini 3 Pro.
DeepSeek sparse attention has two components. First, uh, let's say we are looking at some token in the context in the sequence. Lightning indexer will find which of the previous 2,000 tokens this token will do attention with. If there is less than 2,000 tokens previously, then it will just pay attention to every token. But the power of this comes when the sequence is very long. So the longer the sequence, we always just pay attention to uh, 2,000 previous tokens.
Now, how does this guy determine which tokens to pay attention to? Actually, this is also attention or it's similar to attention mechanism. Uh, maybe I should say part of attention mechanism where we uh, multiply this small lightning indexer query of the current token with these small lightning indexer keys of every previous token. Now, even though this seems like attention, this is actually these vectors are a lot smaller and a lot faster to compute. They are also written in FP8 and they are also using ReLU activation function, which is very quick to compute. Maybe it's a bit worse, not so powerful, but for this very simple task, we don't need anything else. So, so this is an extremely fast attention mechanism. That's still done with every single previous token, but it's way faster than standard full attention. And so this lightning indexer will give scores to previous tokens, and then 2,000 tokens that have highest scores will be attended to in the next step. And after that, they will do the full attention with the query of the token and the previous keys, but only if those keys and values belong. So if the score for them, here I score, belongs to top K, which is maybe top 2,000 scores.
And they combine this uh, with their multi-head latent attention. That's also their innovation in DeepSeek V3. So it's like two optimizations stuck together, and they have a picture diagram of how they combine them. So in multi-head latent attention, instead of generating uh, queries and keys from the token directly, they generate a lot smaller vectors that keep instructions on how to later generate the queries and keys. The trick here is that these smaller vectors are a lot better because they are saved in memory, and we don't need to save queries and keys themselves. We will generate queries and keys on the fly. Here, from the token, they generate this vector C that's later be used to generate key and value, but it's a lot smaller, and only this and this is saved in memory. So, uh, a lot less memory used. So then later, we use that vector with up key projection matrix to generate the key, and then we split key into heads. Now, as I said, this key would be a lot larger than this. So, but this, we generate this on, on the fly, just as we need them, as we need to calculate stuff with it. We don't need to save it into memory. And we also have to apply positional encodings, RoPE, to key. We don't need to apply that to value, just to key. So, we will do that directly from the uh, token embedding itself. And but it will be smaller. It will be smaller because this projection matrix is small. And later, we will concatenate uh, this RoPE, this part of the RoPE with the key, with uh, every head individually. And for the value, it's a lot simpler. We just have the up value matrix. So it will convert this smaller latent vector into the value and then split into heads. That's very simple.
You can see the same thing here. So starting from the initial token, we generate the latent vector that's later going to be used um, for generating value and the key. Where is the value here? Maybe they didn't draw value, but they did draw the key. So latent vector generates um, this part of the key, and this is a RoPE part of the key that has RoPE applied, as I said, it's concatenated, and then top K selection, the top 200 selected. And very similar for the query, query also has this latent vector that's saved into memory, and then on the fly, it's computed. Now you can see the Here it's just um, once we generate one key head, apply RoPE, but here for every head, we have separate uh, head that gets like RoPE embedding. So it's same key head, same key RoPE part for every head, and here every head has individual its own RoPE uh, encoding head part. And then up here is where we concatenate the RoPE part and this uh, part that is generated from the latent vector.
By the way, here we already see some research topics. Uh, they describe how they train this lightning indexer, what normalization they use. We can just try, for example, different normalization, different KL divergence settings, or replace it with something else and see how it works. In the beginning, they just train the lightning indexer. So they already have a working large language model, and they freeze all of the weights, everything, they just train the lightning indexer. And so, uh, they see which tokens were important, were paid strong attention to, by this current token, which token it looked at, and then they use that to train the lightning indexer to predict which tokens are going to be important for this token. After the warm-up stage where they train the indexer, they then unfreeze all of the model weights because now the model needs to learn to use that indexer. So they continue pre-training with of the model.
As I said, they look at uh, 248 most important previous tokens. Interestingly, this training of the both main model and the indexer goes on for 480 sequences of 128,000 tokens, which is a lot of tokens. It doesn't seem like a lot of sequences, but it does seem like a lot of tokens, which results in almost a trillion tokens. That's more than I expected. Wait, uh, I don't get this math. So this amount of sequences, each sequence having this amount of tokens, that would be 61 million tokens. So how did they get almost a trillion tokens? I'm not sure. Oh, wait, wait, wait. And then 15,000 steps, 15,000 of of those. Yeah, when I multiply with 15,000, then we get a huge number. So this is one batch, 480 sequences of 128k tokens. This is, this is such a huge batch. Usually I can fit maybe five sequences of 500 tokens on my GPU.
Okay, benchmarks, evaluations. I couldn't care less. So inference costs go from quadratic scaling to L * K, which is, they like usually this is seen as linear scaling where K is a lot smaller than L, the sequence length, it's just the number of selected tokens, which is 48. Although lightning indexer still has complexity of L squared, it requires much less computation compared to MLA because as I said, it's a lot faster to calculate with smaller vectors and faster activation function.
Oh, this is the scaling graphs. So you see how the orange one, our new method, scales a lot better. So cost is a lot lower. You want this line to be down. And you see it's still scaling, but the scaling is a lot smaller than this quadratic scaling here. And in the beginning, it's going to require a bit more compute because you are not only doing full attention, but there is also this lightning indexer. So this is if the amount of tokens is less than uh, 248, then you are doing like the same full attention, but there is also lightning indexer as well. So maybe an optimization would be not to do lightning indexer if the uh, amount of tokens is less than 48. Nevertheless, this is the long context thing.
In addition to writing tasks and general question answering, their framework encompasses six specialized domains: Mathematics, programming, general logic reasoning, general agentic tasks, agentic coding, and agentic search.
Okay guys, this was interesting, but I want to do our own research, teach you how to do our own research. And uh, so I'm going to be making those videos right now. I don't want to like spend too much time here. So, subscribe, follow me, and see you in the next video. We'll be doing research on linear attention, images, maybe JEPA, and a bunch of other advanced frontiers.