Transcription
Okay, so is unified memory good for running capable models? This is Deep Seek V4 Flash. A 284 billion parameter model, one of the most capable open weight models that we have seen.
Now, this model shouldn't fit on any of these devices. Memory requirement for this, stored the normal way, is almost four times what these devices have. But, you can still run this device, but it runs on both of them at real usable speed. And how it runs is one of the most interesting pieces of system engineering that I have seen this year. This is a project called Dwarf Star from the creator of Redis. Let me draw exactly how this works.
Okay, so let's start with this problem. Small language models have gotten really good. You can run a 1 billion or an 8 billion model on basically anything now. They're fast, they're private, they're free. And for most of the tasks, they're genuinely all you need. But, here's the uncomfortable truth. Our capability still scales with size. The bigger models are simply better at reasoning, at coding, at tool usage.
Now, Deep Seek V4 Flash is really good at agentic coding tasks. This is what we would call a quasi-frontier performance from an open weight model. The problem is that the models worth running are exactly the ones you can't run. They live in data centers behind APIs, even though if they are open weight models. So, the question is how do you get a model like that onto hardware that you actually own?
Okay, but first let's talk about why exactly they don't fit. So, if you have a 284 billion parameters stored each one at 16-bit, you need 568 GB just for the weights. Now, a maxed-out MacBook Pro or DGX Spark gives you 128 GB. So, we are not close. Now, even if you cut every weight to eight bits, you still need 284 GB, which is more than double the VRAM that these devices have. And that's actually the main issue with local models. It's kind of binary. Either the model fits in memory or it doesn't run at all.
So, if you're using a hosted version of open weight models, you're still accessing it through an API. And basically, it's the same as paying for OpenAI or Anthropic. Now, the standard escape hatch is quantization. And to understand what Dwarf Star does differently, you need to see what quantization actually does.
So, every weight in the neural network is just a number. Most of them tiny values scattered around zero. At 16-bit, you have 65,000 possible values to store each one, which is plenty of precision. Now, quantization says, "Use fewer levels." At four bits, you're only looking at 16. Now, if you push it to two bits, you're just looking at four different levels. And our 568 GB model shrinks to around 80 GB. So, you could see, this fits.
So, why doesn't everyone just do this? Because precision is not free. Here's what actually happens when you try to squeeze it too hard. Some weights matter more than others. When an important weight gets snapped into the wrong level, the layer now computes a slightly wrong answer. And a transformer is deep. The slightly wrong answer feeds the next layer and the next. Which means errors are going to compound. You can see it in the quality curve. Down to about four bits, models hold up surprisingly well if they are large models. Below three bits, quality historically falls off a cliff on its own. So, native two-bit quantization gives you the model that fits in the RAM, but isn't worth running. You saved the memory, but threw away the intelligence. Unless, and this is the key insight of the whole approach, you don't quantize everything.
Okay, so this is where Dwarfstar comes in. DS4, a project by the creator of Redis, and it's built with a philosophy I find refreshing. Instead of a general engine that runs everything like llama.cpp or ollama, it runs exactly one model family, which is Deep Seek V4, and it optimizes everything around it. It's basically a self-contained C, and it ships its own custom-built model files tuned for this exact engine. And the idea is that it isn't that it seems to work, it actually is measured.
Now, this is interesting because if you look at Deep Seek V4, it's not just one giant dense network, it's a mixture of experts. And that structure is exactly what makes this whole trick possible. Okay, so let's talk about the architecture a little bit. Inside each of its 43 layers, there is a router, and behind that router sits 256 separate experts, which are basically small feed-forward networks across the whole model, and there are thousands of them. So, when a token comes through, the router picks just a handful of experts to process it. The rest are completely idle for that token. That's why MoE is usually much faster compared to dense networks. Now, there is also a small shared expert that every token passes through. Keep an eye on that one. It matters in a minute.
Now, if you add it up, you get the magic ratio. 284 billion parameters total, but only about 13 billion active for any given token. So, the model itself is huge, but per token, it is actually pretty tiny. So, now the question becomes if you have to crush some of the weights down to two bits, which one do you pick? Dwarf Star's answer is beautifully simple.
Now, to explain this, think of the model as a building. The attention layers, the routers, the shared experts, the output heads, they are the load-bearing walls. Every single token flows through them, so damage here propagates everywhere. But, the routed experts, that's actually the furniture within the building. There's a massive amount of them. It's most of the building, but each token only ever touches a few of those pieces. The routed experts get crushed or quantized to around two bits. Everything load-bearing stays at four bits, effectively untouched. And the reason this works with 256 experts per layer is that there is redundancy. Any single token only meets a few quantized experts, which are switched between high-precision layers. The error never gets the chance to compound like it did in dense cases.
Now, the results, the model drops from 268 GB to about 81 GB. Suddenly, you can see that it's under the 128 GB of MacBook Pro or DGX Spark. And all of a sudden, that thing that couldn't fit now fits.
Okay, but what about the performance? So, they actually are running coding agents. They call tools reliably. But, how would you actually know that? That brings us to the calibrations. Now, there's a second layer to this trick, and it's the difference between guessing and measuring. Now, before quantizing anything, they run the model over nearly 4,700 real prompts, which is about 3 million tokens. These include code reviews, math problems, agent tool calls, and long documents. And while it runs, they record which weight columns actually carry signal, which parts of each expert the model genuinely leans on while doing real work. Then, the quantizer uses that map. The heavily used columns get protected, placed more precisely on those four levels. The rarely used ones absorb the error. The calibration set includes tool-calling prompts in Deep Seek on format. They tune the quantization for gentle work exactly where cheap quants usually fall apart.
Okay, so the model fits, and it was quantized carefully. The obvious question is how much did it lose on performance? Now, they take 100 prompts, send them to the official Deep Seek API, and record exactly which token it produces. This is going to be used as ground truth. Then, they ask the two-bit local model, "How much probability did you assign to each one of those exact tokens?" Because we have access to those probabilities. That's basically the negative log-likelihood token by token measured of drift. And if the quants were damaged, the two curves would split apart. But, in their case, they actually track pretty closely.
Okay, so do you need a 128 GB machine to run this? What if you have 64 GB of unified memory? Now, in the normal approach, you would run into this uh cliff again. But if you're using a MacBook or even GB10, there's another pool of memory sitting right there that you can access, and that is the SSD. Which is usually multiple terabytes on modern machines. And it reads at gigabytes per second. So, here's the question. Do all those routed experts actually need to sit in RAM? Or could they live on disk? And show up only when they are called?
So, this is SSD streaming, and it's actually a pretty interesting part that you definitely want to understand. So, in streaming mode, the load-bearing weights, which are attention, router, and shared experts, sit permanently in RAM. They're needed for every token, so they never leave. Now, next to them, Dwarf Star carves out an expert cache. Which is basically a set of slots pinned in memory, each holding one complete expert. And the full set of experts, which is about 11,000 of them, stays down here on SSD inside the model file.
Now, when the token comes in, the router picks its experts. And if they're already in a cache slot, that's a hit. And it's going to be a fast path. There's no disk involved. But if one misses, the engine reads the single expert straight off of the SSD, drops it into the slot, and evicts which expert has been called for the longest. But here's the interesting thing. The usage actually follows a power law. Some experts are just popular. And this Dwarf Star ships a pre-filled hot list and preloads the popular ones at startup, so the cache starts warm.
Now, the result is long prompts still process fast and generation slows down only as much as you miss the cache, which leads to a complete reframing of that cliff. Okay, so remember we talked about the RAM cliff, whether everything fits in or nothing. Now, here is what it looks like. With streaming, there's no cliff anymore. There's just a slope. So, RAM stops being a wall and just becomes a dial. And this basically controls the cache hit rate now. So, if you have a smaller cache, there are going to be more misses. It's going to be slower, but everything will still run because you are putting the most important bits on the SSD now.
Now, the SSD streaming turns the amount of RAM from a hard cut-off, which defines whether you can run this model or not, into a continuous spectrum of speed levels. So, with this whole project, the question is no longer, "Can I run this model?" It's, "How fast can I run this model?"
Okay, so now weights are only half the story. The other monster is the KV cache, which is basically the model's working memory for your orchestration. Normally, every token you process leaves a record in every layer. At a million tokens of context on a classic architecture, that cache alone can outgrow the model itself.
Now, Deep Seek 4 has a clever layered design. Each layer keeps the most recent 128 tokens raw full resolution. That's basically your local context. Now, older history gets compressed a long time. Half the layers pull every four tokens into one row with an indexer that picks the 512 most relevant rows to attend to. The other half compacts 128 to one. So, the result of this is that a million tokens of context costs about 26 GB. And because it compacts, Dwarf Star treats it as a first-class disk citizen. The whole session gets saved as a file. So, if you just reopen that file, you can resume the model instantly. There is zero reprocessing. So, say if you had a two-hour-long session with the model, that is just a file that you can come back to.
And if you want to push past one machine entirely, Dwarf Star does distributed inference, which is pretty great for running large models. So, say you take two MacBook Pros, you just connect them via a Thunderbolt 5 cable. Now, in this case, you split the model by layers. First half layers on one machine, second half on the other machine. Now, for processing your prompt, this becomes an assembly line. So, while machine B chews on chunk one, machine A is already working on chunk two. And that pipeline genuinely pays. On a 64,000 token prompt, prefill runs 1.85 times faster than a single machine.
Okay, but the generation is actually the footnote here. So, it's one token at a time, so the pipeline collapses into ping-pong across the cables, which is going to be around 19% slower. So, this trick is for fitting larger models and faster prompt processing, not faster generation. But here's the thing, with this trick you can run the bigger 1.6 trillion parameter model at almost 11 tokens per second. Which is kind of incredible.
All right, let's look at some benchmarks. So, all of this is from the repo's published benchmarks. Same model, two-bit quantization. So, these are the generation speeds for different MacBook Pros. Now, if you're running this on DGX Spark, you get slightly lower speed of generation. Now, prefill is where these machines fly. 250 to nearly 470 tokens per second on a long prompt. These numbers are for V4 flash, but you can actually run the Pro 1.6 trillion model at a reasonable generation speed at 9.6 tokens per second at 32K context window. Which is readable speed for a model of that size on hardware you own. Which is kind of incredible.
Okay, I'm going to show you a live demo, but let me zoom out because I think this project matters beyond a single model. First, it really shows what you get when you own the whole stack. In this case, it's the engine, the quantization, the validation, even the coding agent, and tunes them for each other instead of just trying to maximize everything in general. Second, the reframing of RAM as a dial and not a wall kind of really changes what you can run on these systems locally. The SSD in your laptop is going to start becoming part of the memory hierarchy for AI. And the third is the ability to run a quasi-frontier model completely locally on your MacBook or DGX Spark. That is kind of incredible.
Okay, so let me show you a practical example of this. Right now, I'm using the DeepSeek V4 flash on DGX Spark that has 80 GB of VRAM. In order to set it up, just go through this Dwarf Star for Reaper. It has detailed instructions on how to get started. Or you can simply ask a coding agent to set this up on either your MacBook or on DGX Spark. So, in my case, I used Cloud Code to create this UI. I'm currently working on a harness specifically designed for this model, but at the moment, I just want to show you the chat version. Right now, the model is using DS4 server on my DGX Spark, and I am accessing that on the same network.
Now, to show you how this works, we can just click on one of these questions. You can actually see the the first time to first token. Then, we're going to look at the free prefill speed, decode speed, also the total number of tokens, right? In real time, you actually see the GPU use, and the beauty is that once you load the model, then, since it's using the SSD for the KV cache, you don't really see much impact on the memory that you're going to be needing. So, this was a quick response. Now, in my case, I'm getting about 13 tokens per second, which actually is pretty usable.
Okay, I'm going to ask it to create something more challenging. Uh so, I'm asking it to create an encyclopedia of uh 25 legendary Pokémon. Uh you actually see that the time to first token was about 600 ms, which is pretty good. Now, it's going to take its good time uh in order to generate the whole response, and I'm going to show you what the actual speed and the number of tokens actually look like. Now, again, the GPU utilization is somewhere around 93% peak, which is pretty good. This is a 284 billion parameter model running completely locally on consumer hardware.
Okay, so for this it generated about 8,000 tokens at 11 tokens per second. Almost 10 minutes and and the peak chance around 93%. Now, the interesting thing is that the response that it generates are very close to the officially hosted DeepSeek.
Okay, so I ran the same prompt on the DeepSeek website using the instant model. Now, here is the output that is generated by the official hosted version. Now, for the same prompt, these are the generations from our locally hosted version that is running on GB10. Again, keep in mind this is a two-bit quantized version with selective quantization, but still it preserves the behavior of the original model. So, this is pretty close to what the hosted version generates, which means you can run these large language models on consumer-grade hardware locally now.
I highly recommend to check out the Dwarf Star Four project because I think this is a step in the right direction, especially what we are seeing with the whole Anthropic fable drama. I think having local models which are actually close to the frontier are going to be very critical. Now, at the moment, they are mainly focusing on the DeepSeek models, but we could see similar projects dedicated to other models like the newly released GLM 5.2. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one.