Transcription
If you've been working with AI models, especially with agents that are doing some really challenging or long tasks, they take quite a while to process and generate your answer. It sometimes takes a painfully long time to wait for it to generate a response or finish a task.
But what if that weight basically disappeared? What if instead of watching a loading spinner, you got your answer back almost instantly? And what if this speed boost didn't cost you anything in quality? No shortcuts or dumbed down answers. This seems way too good to be true, right?
But the genius team at Deepseek did it again. They found a way to push AI systems to more than six times the output capacity and speed things up by over 80% without any loss in quality. This seems impossible because usually there's a trade-off between speed and quality. You can't really have both at the same time, but surprisingly, Deepseek was able to pull this off.
In this video, we're going to go over this breakthrough and the clever engineering behind it. Now, this is quite a technical paper, but as always, I'll break it down and make it easy to understand for everyone. Let's jump right in.
First of all, if you haven't heard of DeepS, this is a small Chinese lab that's incredibly resource constrained. They have like 20 times fewer employees than OpenAI. Heck, they don't even have access to the top Nvidia GPUs out there. So, they're incredibly limited in terms of compute, but the team is simply brilliant and relentless.
In fact, they've already built one of the most intelligent and efficient open-source models out there, Deepseek V4. Not only did they release the model, but they were also generous enough to reveal all the details on how this works. In fact, if you want to learn more about this, I highly recommend this video where I do a deep dive on Deepseek V4.
Anyway, because of their situation, because they are severely limited by resources, they can't just keep building bigger and bigger models, they have to be laser focused on making them as efficient as possible. And that's why the publications from DeepSeek are so interesting. It's not just about brute force, but about smarter design.
And a few days ago, they released a new system called DSpark. This claims to speed up AI models and increase their output capacity by over 600% without any cost in quality. That's a pretty crazy result.
Now, before we go over this, it's probably helpful to first understand how current AI models work and what the bottleneck is. You see, when you prompt Chat GPT or Gemini or Deepseek, you probably need to wait at least a few seconds before you get a response. You'll see a loading spinner as you wait.
Well, this is because modern AI models generate text through a process called auto regressive generation. Basically, AI writes one word at a time. And to generate each word, it has to look at everything written so far. More technically speaking, each word has to look at how every previous word relates to it.
Now, for a short sentence, that's fine. But imagine writing a really long report that's thousands of words or even hundreds of thousands of words. Every word has to look back at every previous word to see how it relates to itself and then run this through the billions of parameters of the AI model to process and generate the next word. The number of computations required is astronomical.
Now, modern GPUs are already incredibly powerful. They're designed to do a ton of calculations at once in parallel. So, they can easily handle all this data flowing through billions of parameters of an AI model to generate the next word. This step is actually a piece of cake for the GPU.
The real slowdown isn't these calculations through a neural network. It's that the chip needs to fetch all the saved values for the relationships of every single word and how it relates to every other word. Again, for a short sentence, it's fine. But if you're handling a ton of text and each word has to look back at all the other words before it to figure out the context of everything, well, that's the slow part. It's fetching all these values from memory for each word.
And after all these values are fetched, then this data can be passed through the neural network to process. And here the GPU can process everything very quickly. It's just like a tiny burst of work to spit out the next word. But then it sits idle again waiting for the next batch of values to be fetched from memory.
And because AI models are designed to generate only one word at a time, you can't begin calculating things for the next word until the previous word has been generated and confirmed. And this actually creates a direct relationship between the length of the response and the time it takes to generate it. In other words, the longer the answer, the longer it'll take to generate because it needs to generate everything one word at a time. And for each word, it has to look at all the relationships of every other word before it.
Now, of course, the industry is aware of this massive inefficiency and there are workarounds for this. In fact, the current gold standard for speeding up generation is something called speculative decoding. And the idea here is surprisingly simple. Instead of having a large model slowly generate every word one at a time, you use a faster smaller model to guess several words ahead. Then the big model just checks those guesses. If the guesses are good, the large model can accept multiple words at once instead of producing them one at a time.
Here's an easy analogy to understand this. Imagine at work there's a brilliant senior boss. This dude's pretty smart and every final decision has to go through him. But there's just one problem. His time is very expensive. He's also really slow. He insists on writing every single word one at a time. Well, this senior boss is like the full AI model. It's intelligent and we trust it to produce a good answer, but it's just really slow and expensive to use.
So, to speed things up, the company hires an intern. The intern is like a tiny lightweight AI model. He's not as smart as the boss. He misses details and sometimes chooses the wrong words, but he works extremely fast. And here's how speculative decoding works. Instead of asking the boss to write everything, you actually get the intern to quickly draft a chunk of text in advance, maybe 5 to 10 words. The intern hands this draft to the boss. And now the boss doesn't have to check words one by one. He can just verify all the words simultaneously in parallel. In AI terms, the large model can check several guessed words from the smaller model at once because GPUs are built for parallel work. So instead of using the big model to slowly write one word at a time, we just use the small model to guess several words and then we use the big model to check its response all at once.
But what happens if the intern makes a mistake? Well, the system uses a technique called rejection sampling. The bigger model scans the intern's draft left to right and accepts all the words that align. But if the intern messes up on a certain word, the boss pulls out a red pen and rejects that word and everything after it. Then the intern starts again from there and drafts the next chunk for the boss to review again. This is basically how speculative decoding works. The small model guesses ahead. The large model checks the draft and accepts the correct words. If the word is wrong, everything else after it is rejected and the smaller model starts from there and the process repeats again and again.
The important part is that the big model always has the final say. That's why speculative decoding can be lossless. Having a smaller model doesn't mean its answer is dumber. It doesn't mean we're lowering the quality of the output because ultimately the final answer still comes from the large model's decision. We're just using a small model to help it write faster. But the output quality is the same as if the large model had written everything itself. So in a nutshell, that is speculative decoding.
And while this is one of the standard ways to speed up AI models, there's currently a huge dilemma that this idea faces. You see, the main problem is that these interns or the smaller models are deeply flawed and they fail in completely opposite ways. So let's go over this problem.
There are basically two kinds of interns. One is careful and slow and the other is fast but unreliable. You see, the first type is called an auto regressive drafter. This works almost the same way as the bake model. It predicts one word at a time. It generates word one, then word two, then word three, and it keeps going step by step. The advantage here is that each new word knows what came before it. So, it's less likely to get the draft wrong. The output quality is higher. But because it's generating this one word at a time, it's also slow. The longer the draft, the longer it takes for it to generate.
Well, the second type of intern is called a parallel drafter. This is a lot more aggressive. Instead of outputting words one at a time, it predicts an entire block of words at once. For example, you can get it to output multiple words in just a single pass. And this makes it way faster. It's guessing the first word at the same time that it guesses the last word. This also works well because well GPUs are built for parallel work. So a parallel drafter can use more of the hardware efficiently.
Now this sounds great, right? But there's actually a hidden weakness because it predicts all these words at the same time. It doesn't get to naturally build the sentence step by step. It suffers from something called suffix decay. Here's an example of what this is. Imagine the model is trying to agree with the user. Because there are many ways to express the same thing in natural language. It could output of course or it could also say no problem. Both are valid. But for a parallel model, because it's generating both words at the same time instead of sequentially, there is a chance that it could output a messed up answer like of problem or no course. And here it's just a very simple two-word example. But as the draft gets longer and longer and there are more words in it, this gets even worse. The first few words might be fine, but later words are often always messed up. So, this is the suffix decay problem.
To sum things up, here's the dilemma we're facing. There's currently no good solution for speculative decoding because one type of drafter is very careful, but very slow. Another type of drafter is very fast, but it has a lot of errors. And we currently don't really have any solution to this.
Well, that's exactly what DeepS aimed to solve in this paper. So they proposed a system called DSpark and the design is quite genius. Here's how it works. So they took the parallel intern and built on top of that. Remember this parallel intern can basically predict multiple words at once. So this is really fast, but it's also extremely prone to error.
Well, Deepseek added something on top of that. They introduced a tiny, incredibly lightweight loop that iterates one position at a time. It's like having a hyperfast editor sitting next to the intern just lightly nudging them in the right direction. For example, if it typed the word of, then it's going to bias the next word towards course. Or if it typed no, then it's going to bias the next generation towards problem.
Claude Fable is back this week. And if you want to use it to turn your ideas into production-ready content, definitely check out Higsfield, the sponsor of this video. Claude Fable 5 is already one of the smartest models out there. It can understand your idea, reason through it, and come up with really strong creative direction, but normally it's still stuck inside a chat box. It can tell you what to make, but it can't actually make the content. Well, that's where Higsfield comes in. With Higsfield MCP, you can connect Claude to some of the best image and video tools in the world. So, instead of just describing a concept, it can generate the visuals, create the videos, edit the assets, and help build the full campaign. Instead of manually jumping between tools and copying and pasting stuff, you can work directly inside Claude. Claude can plan the concept, write the creative brief, generate the videos or images through Higsfield, and even place the final assets into your folder. No switching tabs, no copying prompts back and forth, no separate creative handoff. And because Fable 5 remembers the context, your characters, brand style, and creative direction stays consistent across the whole project. And if you want to go even bigger, there's also Higsfield's supercomputer. This is basically Higsfield's full AI creative team in one chat. You can choose Fable 5 as the brain, then build everything from the first idea to the final deliverable. Instead of micromanaging every tiny step, you give it the goal and let it act more like a creative director. Whether you're making ads, launch videos, social media videos, or any other content, Higsfield is a game-changer that will supercharge your production workflow. Try it today using the link in the description below.
Now, if we dive deeper, here's how it works in technical terms. They used something called a Markoff head. In probability theory, a Markoff process assumes that the probability of the next state depends exclusively on the current state, completely ignoring any states that occurred before. This phenomenon is actually found everywhere in life. For example, when you're walking, your future position depends entirely on your current position plus which direction you step next. Everything else before that point didn't really matter.
So the markoff head in DSpark only looks at the immediately preceding word to adjust the probabilities for the next word. In other words, it looks at the previous word to guide what the next word would likely be. And it turns out that adding this Markoff mechanism fixes the suffix decay problem we had before.
All right, so it seems like a good fix, but you might be wondering, if we add this extra component and make it sequential, wouldn't it slow things down by a lot? Well, the cracked team at DeepSeek used a technique called low rank factorization. It's too technical for this video, but it basically compresses the information down and makes the computations extremely fast and efficient. And it turns out that after applying this Markoff head with low rank factorization, then the compute cost is actually quite negligible.
In fact, adding this tiny markoff editor only added an additional latency of 2 to 1.3% of the total generation speed. But the return on this tiny investment is massive. With this, it's now much less error-prone and you can boost the length of a correct draft by up to 30%. Which is incredible. In fact, for this new D-Spark model, even if we make it a shallow two layers, it actually outperforms a massive five-layer pure parallel drafter across all benchmarks. It's vastly superior and more efficient and better quality. Pretty much solving this drafting intern dilemma.
All right, so up to now, we've kind of solved this, at least in theory, at least in code. But once we deploy DSpark to a real data server, well, that unleashes a ton of new infrastructure problems. You see, this works completely fine if we have just one GPU serving exactly one user. This is fairly simple and straightforward, but in a real production setting, you have a data center with multiple GPUs, plus you'll have like thousands of concurrent users all using the same model at the same time.
You see, the full AI model, in other words, the boss model has a strict physical limitation known as the batch capacity. It can only process a set number of tokens at once across all users simultaneously. So, if user A asks a question, the system sends a draft from the intern model to the boss model to verify. And this draft consumes a percentage of the total compute pie. If the draft for user A was a bad guess and the boss rejects it halfway, well, all of this compute is essentially wasted. If the draft is long and contains a ton of errors, not only does this waste the boss model's batch capacity, but it also wastes the interns compute, plus it also adds latency while the other users' queries are held in Q. So, it goes without saying sending bad drafts to the boss model wastes compute and hurts everyone else on the server. You want to avoid sending as many bad drafts as possible.
Now, to fix this, you might be thinking, maybe we can just make the drafts shorter so that we don't waste capacity, right? And shorter drafts tend to contain fewer errors, especially near the ends. Well, actually, that would be inefficient in the opposite direction. You see, the point of making drafts in the first place is for the generations to run faster. And the longer the draft, the faster it can generate the output.
So, here's the trick that DeepS proposed. The length of the draft should actually be based on context. If the context is about content that's very predictable, for example, answering a math or coding problem, the draft length can be longer because the answer is very deterministic. It's less likely for the inter model to get the draft wrong. But if the context is open-ended, for example, if it was prompted to write a creative story, then it's really easy for this draft to contain a ton of wrong words, or in other words, a ton of words that the boss model would reject because there could be so many possibilities of answering this. So in this context, the draft length should be shorter or cut early.
Now, how do we do this? The system needs a kind of dynamic traffic cop. It needs to look at the context and dynamically decide how long the draft should be or when to end it early. In fact, we need a mechanism that can cut the draft writing early if it detects that it's likely to make a mistake at that point. That way, it won't waste compute generating more stuff that the boss model would likely reject.
So, the Deep Seek team also added a specialized lightweight module called the confidence head. This is attached directly to the drafter or the intern model. For every word that it generates, the head also outputs a confidence score ranging from zero to one. Zero would mean it's completely guessing and it's not sure if it's the right word. Whereas one would mean it's 100% certain that this is the right word. This is the word that the boss model would also write. And to make this work, they added a simple rule. If the word is above a threshold, let's say above a confidence score of 0.6, then keep going. If any word is below that threshold, then you end the draft writing immediately and you stop wasting compute. You only send that portion of the draft to the boss model for review.
And after implementing this and testing it out, the results were staggering. After using this confidence threshold, the acceptance rate from the boss model jumped up from a dismal 45.7% to a crazy 96%. In other words, now the drafts are much more likely to be accepted by the boss model. Therefore, we're wasting way less compute.
Let's trace the logic of this improvement. Let's say the AI is attempting to write a creative story because it's so open-ended. The draft it writes starts to have very low confidence early on. Let's say by the fourth or fifth word. At that point, based on the current mechanism, the system terminates the draft because its prediction is weak. The intern is not really confident if it continues writing the draft whether the boss model would actually accept it. And conversely, let's say the AI is trying to solve a math or coding problem. In this case, the answer is much more deterministic and when it's writing the draft, it's very confident throughout its prediction. Its confidence values are almost always above the threshold. So, the mechanism doesn't have to cut the draft early. The draft can be longer in this case and therefore the generation can be faster. So this is a mechanism that can automatically decide how much to speed up the generation while maintaining good quality and avoiding wasting any compute.
But that's not all. You see, the genius of this design is that it's also hardware aware. It looks at all active requests, gathers the confidence of all these drafts, and then compares them against something called an SPS curve. This is basically a chart showing how GPU speed changes depending on the batch size. You see, smaller drafts process quickly and are easier on the GPU, but larger drafts slow down the GPU, which could affect other concurrent users. Well, the mechanism also monitors current GPU load and decides in real time the length of the drafts it accepts. This ensures the best possible trade-off between speed and hardware performance.
During off-peak hours, for example, when there are relatively few users, the system notices that it has spare GPU capacity. So, it loosens up, letting the draft model guess more words at once. This uses the extra available power to make each response faster for users. But during busy hours, it senses the added strain. GPU capacity becomes low, so it shortens how many words the draft model is allowed to guess. Individual responses do get a bit slower, but this protects the system as a whole from slowing down or crashing under load.
So we have assembled the pieces. We have a fast hybrid intern generating drafts. We have a confidence head evaluating the output of the draft and deciding when it should be cut off. We also have an algorithm that looks at the GPU load in real time to automatically adjust the drafts that it receives. This is a completely organic and self-regulating engine. It's a beautifully designed optimization for the entire stack from software to hardware as expected from the DeepSec team.
And if we look at the results in production, it's extremely impressive. If you compare this new D-Spark method against DeepSseek's previous MTP system, which only predicts one word or token at a time, you can see that this new DS-park delivers a 60 to 85% increase in generation speed without any loss in quality. That's the key here. We're essentially getting faster speeds without any sacrifice in quality.
In fact, get this. If the researchers set a fixed rule saying that every user must get at least 120 tokens per second, then the old MTP system broke down fast. It could only serve a small number of users before it crashed because the software and hardware were just not optimized. On the other hand, DSpark avoided this because it's smarter about resource use. It knows when to use longer drafts or when to end a draft early to save GPU power. As a result, it could achieve an almost 700% higher total system output, which is an insane improvement. And again, all of this did not affect output quality.
In fact, because of this dynamic traffic cop system, remember this is the mechanism that allows DSpark to adjust itself automatically. It can now handle combinations of speed and user capacity that weren't possible on the same hardware before. You can see this new dspark method is able to process way more data even as the number of concurrent requests from users increases.
So in a nutshell, that is the dspark system by deepseek. When you don't have unlimited GPUs, unlimited money or unlimited people, you're forced to be clever. And DSpark is a perfect example of this clever engineering. They solved a problem that the industry assumed was not solvable. And from that they were able to significantly increase the speed and output of AI models without affecting quality. And that's what makes these publications from DeepSeek so fascinating to read. Their ingenuity is just out of this world.
And as always, they've released the code to this already. So I'll link to this GitHub repo in the description below, which contains instructions and the code to run DSpark. Plus, this is released under a very permissive MIT license. It's awesome that the DeepSeek team is willing to reveal the details to this, especially all this infrastructure stuff on how to actually deploy this to a data center and run it efficiently. This is incredibly valuable information and it's often top secret for the closed labs out there. These are like industry secrets which the closed labs definitely don't want you or any competitor to know about. But here, Deepseek is revealing all these details.
Plus, the awesome thing is this D-Spark mechanism has already been implemented into their latest DeepSseek V4 model. So, I'll link to this hugging face page in the description below where you can download the latest DeepSseek model with DSpark implemented. And in theory, this should run even faster and more efficiently compared to the previous version, which by the way is already insanely fast and cheap. This is the fastest and most efficient Frontier model out there, but they just made it even faster.
Anyway, this is one of the more technical papers I reviewed so far on my channel, so hopefully I made it easy for you to digest. In fact, the paper is jam-packed with a ton of additional technical details which I didn't have time to cover. So, if you're interested in digging deeper, I'll link to this original paper in the description below. As well, let me know in the comments what you think of this.
As always, I will be on the lookout for the top AI news and tools to share with you. So, if you enjoyed this video, remember to like, share, subscribe, and stay tuned for more content. Also, there's just so much happening in the world of AI every week. I can't possibly cover everything on my YouTube channel. So, to really stay up to date with all that's going on in AI, be sure to subscribe to my free weekly newsletter. The link to that will be in the description below. Thanks for watching and I'll see you in the next one.