Transcription
Deepseek is built different. They just dropped a new model, Deepseek V4. But here's the thing. Unlike the top closed AI labs out there that are spending billions on data centers with unlimited compute, DeepSeek is incredibly constrained. They don't have nearly as much compute. Heck, they don't even have the top NVIDIA chips. And their team is like 40 times smaller than OpenAI. They're incredibly resource limited. But yet, they managed to build a model that's on par with the top closed models out there. The ridiculous thing is they even open sourced this and they even released a paper on how they built it.
Now, I spent the past few days reading this and the design is absolutely beautiful and ingenious. But here's the thing. If you're not technical, all of this looks like alien language. So, in this video, I'm going to break down everything in simple terms so you can understand how incredibly cracked this model and the DeepSeek team is. Let's jump right in.
First, let's talk specs. So, this latest V4 Pro model is massive. It has 1.6 trillion parameters. If you're not technical, parameters are basically the dials and knobs inside the model's brain that stores everything it knows. In theory, the more parameters, the smarter and more capable the model is. And 1.6 trillion parameters is among the best models out there. But this comes with a catch. It's way harder to build and train a model of that size, which we'll talk more about in a second.
Now, this new V4 also has a context length of 1 million tokens. This is basically the model's short-term memory, or how much info you can stuff into your prompt for it to remember all at once. 1 million tokens is roughly 750,000 words, which is huge. It's like feeding it the entire Harry Potter series and asking it about a very specific detail on a page, and it'll actually remember it. Or for agents, it means the model can run for hours on long tasks without losing track of what it's doing. But again, a 1 million token context window is insanely hard to actually build correctly. So, the DeepSeek team had to come up with some ingenious solutions to all of this.
In fact, let's dig deeper into why having a 1 million token context window is so hard. You see, modern AI models don't read text the way we do. Every time it reads a new word, or what's technically called a token, it asks, "How does this word relate to all the other words before it?" Take a simple example. The cat didn't cross the street because it was too tired. When it processes each word, it looks at all the words before it to see which ones are most relevant to the current word. In fact, this is the attention part first introduced by the legendary paper "Attention Is All You Need" by Google. And this is the foundation behind all large language models that we know of today. In fact, if you're interested in learning more, definitely see this video for a full explainer.
Now, for a short sentence like this, it's fine. If you're at the 10th word, that's just 10 comparisons. No big deal. But if you're at the 100,000th word, that's 100,000 comparisons. And this is the fundamental bottleneck of every large language model. Imagine pushing this to a million tokens. At that scale, the number of comparisons becomes astronomical. So large that even high-end hardware starts to choke just trying to keep up. And it's not just the compute that suffers. To make this all work, the model has to store intermediate results. Basically, a running memory of everything it has seen so far. This is called the key value cache or KV cache. You can think of it as like a massive lookup table. For every past word, it stores information about what that word meant in context. Now, at small scales, like for a short paragraph, this is pretty manageable. But at a million tokens, it becomes absurd. You're now storing a ton of data just to maintain context for a single conversation. These are like gigabytes sitting in expensive GPU memory just so the model doesn't forget what it read earlier. And if you put these two things together, the insane compute required and this exploding memory issue, you hit a wall.
So, how did DeepSk solve this? Looking at the paper, what's interesting is that the team didn't just throw more brute force compute at the problem because, well, they didn't have a lot of compute. Instead, they asked a much more elegant question. What if the model didn't have to look at everything in the first place? The key idea behind Deepseek V4 is deceptively simple. Don't treat all past information as equally important because in reality it isn't. When you're reading a book, you don't constantly reread every page you've ever seen. You skim, you summarize, and you jump back only when something is relevant. Your brain is selective, and Deepseek tries to do the same thing. They call this a hybrid attention architecture. And at its core are two complimentary strategies called CSA and HCA. Basically, compress the past and then ignore most of it.
Let's start with the compression. So in a traditional model, every token is stored individually. One word, one entry, no shortcuts. What DeepS did instead is group them. So one part of the system is called compressed sparse attention or CSA for short. It takes small chunks of tokens, say four at a time, and merges their information into a single denser representation. So instead of remembering individual tokens, it stores a compact summary of all four. So right away you've reduced the sequence length by a factor of four, which means fewer comparisons and less memory and therefore less compute. But that alone isn't enough. Even after compressing, you're still left with hundreds of thousands of these blocks at large scales. Still too many to process efficiently. Compression helps, but it doesn't solve the core problem.
The real breakthrough comes from the next step, which is sparsity. Once the past is compressed, the model doesn't treat all of it equally relevant. Instead, it uses a fast internal mechanism, kind of like a built-in search engine, to pick out only the most useful pieces. They call this the Lightning Indexer for sparse selection. When the model processes a new token, it doesn't scan the entire history. It rapidly scores all those compressed blocks and selects only a small subset, the ones that most likely matter, for the current context. Everything else is ignored, just skipped entirely. And this is a subtle but profound shift. The model isn't trying to remember everything perfectly. It's trying to remember the right things at the right time. And that changes things completely. Instead of doing massive computations over the entire past, it focuses compute only where it actually matters. And this drastically reduces the workload without losing meaningful context.
But Deepseek didn't stop there because sometimes you do want a broad high-level understanding of everything that came before even if you don't need the fine details. That's where the second system comes in. So this is called heavily compressed attention or HCA for short. Here the compression is far more aggressive. Instead of grouping four tokens, it groups something like 128 tokens or like an entire paragraph into a single representation. Now you're shrinking the sequence length by orders of magnitude. And at that point, something interesting happens. The sequence becomes so short that the model can afford to look at everything at once because everything is now small enough to handle. So you end up with a layered strategy. One pathway keeps moderately detailed chunks and selectively retrieves the most relevant ones. Another one keeps extremely compressed summaries of the entire history. And both are used together interled through the network. So this is a balance between precision and efficiency between remembering everything and remembering just enough. Instead of asking how do we process all of this now deepseek flipped the question to how little can we process and still understand everything.
But there's an obvious problem with this approach. If you compress 128 tokens or an entire paragraph into a single mathematical representation, you're throwing information away. There's no way around that. You're trading detail for efficiency. But what happens when you actually need a specific detail back? What if you ask the model about a very specific sentence buried half a million tokens ago? Not the general idea, but the exact phrasing or a specific number. If that entire paragraph had been compressed into a single block, wouldn't that information be gone? So, this is the fundamental problem in every compression system. The more you shrink something, the more you risk losing what made it precise. But the cracked team at Deepseek also solved this issue. So, their solution wasn't to avoid compression. It was to balance it. They introduced a third pathway that does the opposite. It keeps information completely uncompressed. So this is called the sliding window attention. While the model aggressively compresses the distant past, the sliding window continuously tracks the most recent tokens. Let's say the last 128 words with full exact fidelity, word for word, no compression.
So now you have three parallel views of the same document. One that's highly compressed, capturing the broad detail of everything that came before it. One that's moderately compressed, selectively retrieving relevant chunks when needed, and one that's completely uncompressed, preserving the immediate context in perfect detail. To understand how these work together, it helps to think of a student studying for an exam. You don't reread the entire textbook every time you answer a question, right? Instead, you operate on multiple levels at once. You keep the last few pages open in front of you because they're immediately relevant. That's your sliding window. You rely on summaries of earlier chapters to keep the big picture in mind. So that's the heavily compressed attention. And when you need something specific, you look through your highlighted sections. That's the Lightning Indexer for sparse retrieval. What Deepseek does is turn that intuitive human strategy into a precise mathematical system. By interle these three distinct attention strategies layer by layer through the neural network. Deepseek v4 manages to capture the best of all possible worlds. It gets deep long range retrieval and also broad understanding and also sharp immediate focus. It's able to achieve a 1 million token context window while sidest stepping the devastating compute cost of traditional attention.
And the efficiency payoff they document in the paper is pretty insane. So they benchmarked this new V4 version against the previous DeepSseek V3.2 which for context was already considered one of the most efficient large language models in the industry. But even for V4 which has a huge 1.6 trillion parameters with a massive 1 million token context window. Here you can see that DeepS v4 Pro requires 3.7 times lower flops or you can think of this as compute compared to the previous DeepSseek version 3.2. Version 4 runs on roughly 27% of the compute that was required for the previous version. That's an insane improvement in efficiency. Here's another crazy chart. This is the KV cache. Again, this is like the short-term memory of the AI model. And here you can see that Deepseek version 4 Pro is like almost 10 times smaller than the previous DeepSseek version. It only requires 10% of the KV cache memory. So in other words, they reduced the memory footprint of the model's short-term memory by 90%. That alone completely changes the hardware requirements for deploying these huge models. It makes them ultra efficient.
But here's the question. Does solving attention actually solve the whole problem? Not even close. Because the challenge isn't just memory, it's also scale. Now, again, this new V4 Pro has a whopping 1.6 trillion parameters. Again, these are the dials and knobs that make up the model's intelligence. And this is massive. This is like one of the biggest models in the industry. And all these parameters means the model is huge with many, many layers. When you stack dozens of these layers together at a trillion parameter scale, the system can become very unstable. The signals flowing through the network start to amplify like crazy. And just as a simple side effect of all this math, think of it like a microphone placed too close to a speaker. You know that high-pitched screech feedback? That's exactly the same feedback loop in neural networks, but instead of sound, it's numbers. The values explode, the loss function diverges, and it's really easy to crash a training run. This is called a signal explosion.
So, how do you stop a trillion parameter network from crashing? Well, traditionally, developers have tried to solve this using residual connections, which act as bypass lanes that let the signal skip certain layers and keep the math stable. In fact, if you're interested in learning more about residual connections, I cover it in a bit more detail in this video. But these residual connections are still not enough for you know a trillion parameters. So more recently the industry has moved toward hyperconnections which now expand the width of these residual streams to give the signal even more lanes to travel in. But the Deepseek paper explicitly mentions that when you scale to over a trillion parameters, even standard hyperconnections start experiencing these catastrophic spikes. These traditional bypass lanes just aren't enough at that scale.
So the engineering team introduced a novel architecture called manifold constrained hyperconnections or MHC for short. In fact they published a full paper on it back in January of this year. And this is where the math in the paper gets genuinely beautiful. The core idea is to take these residual connections and constrain it onto something called a manifold of doubly stochastic matrices. Now what on earth does that mean? It probably sounds like pure alien language. So, let's break it down.
On my channel, I featured so many different AI models and tools, it can be very overwhelming. What if you can use all of these models all in one platform? And that brings us to Chat LLM by Abacus AI, the sponsor of this video. Chat LLM is an all-in-one platform for you to use the best AI models out there. You can seamlessly switch between different models in your chats. Plus, you can also use all the top image generators out there, as well as the top video models out there, all in one integrated platform, and they're usually very quick to add a model to their site once it comes out. Plus, if you're coding something, they have a really useful artifacts feature so that you can preview your generation side by side. Plus, they have a deep agent feature which can do some really complex tasks all autonomously, like creating PowerPoints, websites, and research reports. It's going to supercharge your productivity. Best of all, you can access all these AI models and image and video generators and Deep Agent for only $10 a month. This is way cheaper than if you paid for each tool separately. Definitely check out Chat LLM that comes with Deep Agent in the description below.
You can think of the residual connections not as a simple pipe carrying the signal forward, but as a distribution grid. In a normal unconstrained network, as the signal flows through layer after layer, the pressure in that grid can accidentally build up. And when it does, it feeds on itself and it creates a massive feedback loop just like a microphone near a speaker, which causes the signal explosion. Well, here's the key idea. Instead of trying to dampen that explosion after it happens, Deepseek enforces a hard rule before it can ever occur. It forces the residuals to behave like what is called a doubly stochastic matrix. In other words, every row must sum up to one and every column must also sum up to one. In plain terms, that means the total signal is always conserved. It will never amplify. The signal can literally not blow up because of these constraints because the math forbids it. But here's the real challenge. Defining that rule is easy, but applying it at the trillion parameter scale is not. So to make it work they used something called the Syncorn Knop algorithm. Basically before each layer processes the signal the model runs a rapid sequence of row and column normalizations around 20 iterations until the matrix perfectly satisfies those constraints. At a first glance, this sounds terrible. We just spent all this time optimizing compute. And now you're saying we need to insert a 20step loop for every layer in a 1.6 trillion parameter model. That would require way too much compute and kill performance. But here's where the genius engineering of DeepSeek kicks in. They used very aggressive low-level optimization. So these are like programs that control how the GPU actually runs computations. Really complex stuff like fused GPU kernels, selective recomputation, etc. This is really deep and niche. Basically, think of this as like custom tweaking the GPU to be even more optimized. And basically they were able to shrink the overhead of this entire process to only 6.7% of runtime. So yes, there's a cost to adding a 20step loop to each layer, but it's only 6.7% more time. A relatively small price to pay compared to having a training run crash due to a signal explosion. So it's a very worthy insurance policy. This is how they were able to keep the signal of a trillion parameter model from exploding.
But they didn't stop there. You see, to train a massive model, you need an optimizer function. This is the algorithm that decides how the model learns from its mistakes. How should its dials and knobs be changed if its response is wrong. If you want to learn more about how a model is trained, again, see this explainer video for more details. For now, just think of this algorithm as deciding how the model should adjust its dials and knobs if its response is wrong. Now, for years, the industry standard has been an algorithm called Adam W. But Deepseek also replaced this with a custom optimizer called Muon. Here's how Muan works in simple terms. Instead of applying updates directly, you can think of Muon as like reshaping them using a two-phase process. First, aggressively pushing the system toward convergence and then carefully stabilizing it. You can think of this as like tuning a guitar. First, you make big rough adjustments to get the strings close to the right pitch. Then, you switch to tiny precise tweaks to adjust things into the perfect pitch. Rough and fast at first, then switch to subtle precise adjustments. That combination lets the model learn faster and also stay stable.
So, at this point, the picture is coming together. You've got a hybrid attention system solving the memory problem, a mathematically constrained architecture preventing signal explosions, and a highly optimized algorithm that accelerates learning. On paper, it's beautiful. But here's the reality check. There's a huge gap between elegant equations and getting thousands of GPUs in a data center to actually execute them efficiently. And at the scale of Deepseek V4 at 1.6 6 trillion parameters. The biggest bottleneck isn't even computation anymore. It's communication. This model is so huge that it can't just live on one chip or even one rack. The layers of a model have to be scattered across different racks in a data center. Now, while a GPU or even a rack of GPUs can process stuff really quickly, the problem now is that as the model processes data, that data needs to flow from rack to rack to rack. If GPUs need to wait for data to be sent to it, well, every millisecond of waiting is wasted money. Ideally, you want nothing in the data center to ever be idle.
Of course, the cracked DeepSeek team was also aware of this issue. And here's how they solved it. Instead of waiting for a massive batch of data to arrive before kicking off computation, they basically choreographed the data transfer. In simple terms, they broke it down into smaller sequential waves. As soon as the data required for the first wave arrives, the GPUs immediately spin up and start crunching the numbers. And while those GPUs are working on wave 1, the data for wave 2 is already actively traveling over the network cables in the background. Then wave three and wave 4 and so on. Computation and communication are perfectly overlapped. So the network latency essentially disappears. The network cables stay saturated with data. The compute cores stay busy crunching numbers and the whole system runs on one continuously beautifully optimized pipeline.
But to choreograph something this complex at the microscopic level of GPU computing is really hard. You can't just use standard coding languages. So the paper mentioned how they used a specific language called tilang to develop what are known as fused kernels. A fused kernel is basically taking multiple separate math operations and merging them into one single command so the GPU doesn't have to keep wasting time reading and writing intermediate results back to its main memory. It saves an enormous amount of time, but writing fused kernels by hand is notoriously errorprone and exponentially harder when you're working with over a trillion moving parts. The awesome thing is they've even published a GitHub repo where they open sourced how exactly they did this. But basically they used something called a Z3 SMT solver to 100% prove that their fused kernel code was indeed correct. Remember at this scale even a one in a billion error could happen constantly and silently corrupt the model. Instead of hoping the code works they need to mathematically guarantee it. And the remarkable thing is they were able to do this using some incredibly clever and complex math.
All right, up to now we've already covered a ton of pieces. If we zoom out, we have the parallel attention compression. We have the manifold constrained hyperconnections. We have the Muon optimizer. We even fine-tune how a data center should fit and run the model. All of this is just the architecture design. Next, we also need to feed it data to train the model so it actually becomes intelligent and performant. Well, here it says, "Deepsek v4 probe was trained on 33 trillion tokens." It's hard to imagine how huge this is, but it's basically like more text than any human could process in many lifetimes of reading. If you just throw all this data at the model, there's a risk of it just drowning in it. It's like if you give a complete noob a very advanced textbook on quantum physics, he'll just get very overwhelmed and likely fail to learn anything. So for DeepSeek, they didn't throw the entire data set at it at once. Instead, they used a curriculum. At the start, the model only sees short sequences around 4,000 tokens, just enough to learn a local structure like grammar, syntax, and basic patterns. Then, as the training stabilizes, they gradually stretch the context from 16K, and then after that, they increase it further to 64K all the way up to the full 1 million token window. It's like slowly expanding the working memory of a brain while it's still learning how to think.
But again, here's where things get messy. At this scale, with 1.6 trillion parameters and feeding it stuff with a million tokens, training can become very unstable. One of the biggest threats is something called a loss spike. In simple terms, the math basically blows up and the whole training run crashes. The usual industry fix is not really ideal. You just stop and roll back to an earlier save and try again. It's expensive and it doesn't actually fix anything. Now, Deepseek did something weird. They introduced something called anticipatory routing. It doesn't look at now, but instead it looks at slightly earlier versions of now, historical snapshots of the model's parameters and uses those to make decisions. And you might be thinking that sounds counterintuitive, right? Why deliberately use outdated information when you're training the system? And that's because the instability is in the noise. The sharp chaotic fluctuations that happen step by step. The real trend underneath moves slowly. It's like when you're looking at a stock chart, the daily movements can be quite chaotic. But if you look at a moving average, then you can see the overall trend more clearly. But using a slightly older snapshot, the system effectively ignores the noise and locks onto the underlying trend. And to make this practical, they don't even run it constantly. The system monitors its internal statistics and the moment it detects early signs of a loss spike. It activates this anticipatory routing to stabilize the signal and then gradually hands control back to realtime routing once the danger passes. So the model is like self-stabilizing during training. And combined with the earlier constraint systems, they built something genuinely remarkable. A massive model that's built by a really small team that's constrained in resources and compute. But it's even able to match the performance of the top models out there.
You can see across all these different benchmarks for knowledge and reasoning as well as agentic capabilities, it's pretty much on par with some of the top closed models out there, including Opus 4.6 Max and Gemini 3.1 Pro. And if you look at the win rate between DeepSeek and Opus 4.6 6 Max. This was Claude's previous best model. You can see that DeepS version 4 on average has a higher win rate than Opus 4.6. It also outco competes Opus 4.6 across all these tasks. Here's another crazy achievement. In the realm of frontier mathematics, we have this benchmark called the Putnham 2025. This is like famously known as one of the most difficult undergraduate math competitions in the world. But get this, Deepseek V4 achieved a perfect score, 120 out of 120. And remember its 1 million token context window. This is like its short-term memory or how much info you can stuff into your prompt at once for it to remember. When pushed to the absolute extreme limit of 1 million tokens, its retrieval accuracy even beats Google's latest Gemini 3.1 Pro, which also has a million tokens. And if you look at this independent leaderboard from Artificial Analysis, you can see that Deepseek V4 Pro is like the second best open- source model out there, just below Kimik 2.6, and it's edging very close to the top closed models out there from Google Anthropic and Open AAI.
And that sums up my deep dive on Deepseek V4. Just to summarize, here's the cracked design of this model. This is incredibly tricky to build because it has a massive 1.6 six trillion parameters with a context window of a million tokens. You've got an interled attention system solving the memory problem, a constrained architecture preventing signal explosions, a highly optimized algorithm that accelerates learning, some beautiful low-level code that makes sure it runs efficiently in a data center, and some really clever self-stabilizing training, plus a ton of other technical details I didn't even cover, otherwise the video would be way too long. But in the end, you get a beast of a model that's on par with the top closed models out there. And that's really the theme behind this entire video. It's not just one breakthrough. It's dozens of cleverly engineered solutions all working together that turns something pretty much impossible possible. And remember that the Deepseek team is very constrained in resources. They don't have a huge data center. They don't even have the top NVIDIA GPUs. Their team is way smaller than OpenAI or Google, yet they were able to optimize everything so efficiently and still achieve top tier performance. And the ridiculous thing is instead of just keeping it closed and behind a paywall, they actually open sourced the model. So it's out on Hugging Face, you can download this for free and run it offline if you have the right hardware. And they even released this paper spilling all the info on how it was designed and how they trained it, including this infrastructure stuff, which is like top secret for the closed AI labs. This is info that they never share. There are so many gold nuggets in this paper, it's hard to believe that they're revealing all of this for free. What a cracked team.
Anyways, that sums up my explainer on the new Deepseek V4. I dove way deeper this time with a ton of technical stuff, so hopefully it was easy to understand. Let me know in the comments what you think of this. As always, I will be on the lookout for the top AI news and tools to share with you. So, if you enjoyed this video, remember to like, share, subscribe, and stay tuned for more content. Also, there's just so much happening in the world of AI every week. I can't possibly cover everything on my YouTube channel. So, to really stay up to date with all that's going on in AI, be sure to subscribe to my free weekly newsletter. The link to that will be in the description below. Thanks for watching and I'll see you in the next one.