Transcription
And some of this requires low-level engineering. It just is a giant mess and trickery.
So, as I understand, they went below CUDA; they go super low programming of GPUs. Effectively, Nvidia builds this library called Nickel, right? In which, you know, when you're training a model, you have all these communications between every single layer of the model, and you may have over 100 layers.
What does Nickel stand for? It's NCCL, Nvidia Communications Collectives Library. Nice.
When you're training a model, right, you're going to have all these all-reduces and all-gathers, right? Between each layer, between the multi-layer perceptron or feedforward network and the attention mechanism, you'll have basically the model synchronized, right? You'll have all-reduce and all-gather, and this is communication between all the GPUs in the network, whether it's in training or inference.
So, Nvidia has a standard library. This is one of the reasons why it's really difficult to use anyone else's hardware for training, because no one's really built a standard communications library. Nvidia's done this at a sort of a higher level, right? A deep seek, because they have certain limitations around the GPUs that they have access to. The interconnects are limited to some extent by the restrictions of the GPUs that were shipped into China legally—not the ones that are smuggled, but legally shipped in—that they used to train this model.
They had to figure out how to get efficiencies, right? One of those things is that instead of just calling the Nvidia library Nickel, right, they instead created their own communications, which some of the labs do, right? Meta talked about in Llama 3 how they made their own custom version of Nickel. They didn't talk about the implementation details; this is some of what they did, probably not as well as Deep Seek.
Because Deep Seek, you know, necessity is the mother of innovation, and they had to do this. Whereas, in the case of OpenAI, they have people that do this sort of stuff—Anthropic, etc. But, you know, Deep Seek certainly did it publicly, and they may have done it even better because they were gimped on a certain aspect of the chips that they have access to.
So, they scheduled communications on, you know, by scheduling specific SMS. You could think of SMS as like the core on a GPU, right? There are hundreds of cores, or there's a bit over 100 cores, SMS on a GPU. They were specifically scheduling, hey, which ones are running the model, which ones are doing all-reduce, which are doing all-gather, right? They would flip back and forth between them, and this requires extremely low-level programming.
This is what Nickel does automatically, or other Nvidia libraries handle this automatically, usually. Yeah, exactly.
So, technically, they're using, you know, PTX, which is sort of like, you could think of it as like an assembly-type language. It's not exactly that or an instruction set, right? Like coding directly to assembly or instruction set, it's not exactly that, but that's still part of technically CUDA.
But it's like, do I want to write in Python, you know, PyTorch equivalent and call Nvidia libraries? Do I want to go down to the C level, right? Or, you know, and code even lower level? Or do I want to go all the way down to the assembly or ISA level?
And there are cases where you go all the way down there at the very big labs, but most companies just do not do that, right? Because it's a waste of time, and the efficiency gains you get are not worth it.
But Deep Seek's implementation is so complex, right? Especially with their mixture of experts, right? People have done mixture of experts, but they're generally 8, 16 experts, right? And they activate two.
So, you know, one of the words to be like used is like sparsity factor, right? Or usage, right? So, you might have four, you know, 1/4 of your model activate, right? And that's what MRA's mixol model, right? Their model that really catapulted them to like, oh my God, they're really, really good.
OpenAI has also had models that are, and so have all the other labs that are major closed. But what Deep Seek did that maybe only the leading labs have only just started recently doing is have such a high sparsity factor, right? It's not 1/4 of the model, right? Two out of eight experts activating every time you go through the model; it's eight out of 256.
And there are different implementations for mixture of experts where you can have some of these experts that are always activated, which just looks like a small neural network, and all the tokens go through that. Then they also go through some that are selected by this routing mechanism.
One of the innovations in Deep Seek's architecture is that they changed the routing mechanism in mixture expert models. There's something called an auxiliary loss, which effectively means during training you want to make sure that all of these experts are used across the tasks that the model sees.
Why there can be failures in mixture of experts is that when you're doing this training, the one objective is token prediction accuracy. If you just let training go with a mixture of expert model on your own, it can be that the model learns to only use a subset of the experts.
In the literature, there's something called the auxiliary loss, which helps balance them. But if you think about the loss functions of deep learning, this even connects to the bitter lesson, is that you want to have the minimum inductive bias in your model to let the model learn maximally.
This auxiliary loss, this balancing across experts, could be seen as intention with the prediction accuracy of the tokens. So, we don't know the exact extent that the Deep Seek change, which is instead of doing an auxiliary loss, they have an extra parameter in their routing, which after the batches, they update this parameter to make sure that the next batches all have a similar use of experts.
This type of change can be big; it can be small, but they add up over time. This is the sort of thing that just points to them innovating, and I'm sure all the labs that are training big models are looking at this sort of thing, which is getting away from the auxiliary loss. Some of them might already use it, but you just keep accumulating gains.
We'll talk about the philosophy of training and how you organize these organizations, and a lot of it is just compounding small improvements over time in your data, in your architecture, and your post-training, and how they integrate with each other. Deep Seek does the same thing, and some of them are shared, or a lot.
We have to take them on face value that they share their most important details. I mean, the architecture and the weights are out there, so we're seeing what they're doing, and it adds up.
Going back to sort of the efficiency and complexity point, right? It's 32 versus a four, right? For like mix draw and other models that have been publicly released, so this ratio is extremely high.
What Nathan was getting at there was when you have such a different level of sparsity, you can't just have every GPU have the entire model, right? The model's too big; there's too much complexity there. So, you have to split up the model with different types of parallelism, right?
You might have different experts on different GPU nodes. But now, what happens when a set of data that you get, hey, all of it looks like this one way, and all of it should route to one part of my model, right?
So, when all of it routes to one part of the model, then you can have this overloading of a certain set of the GPU resources or a certain set of the GPUs, and then the rest of the training network sits idle because all of the tokens are just routing to that.
This is one of the biggest complexities with running a very, you know, sparse mixture of experts model, i.e., you know, this 32 ratio versus this four ratio, is that you end up with so many of the experts just sitting there idle.
So, how do I load balance between them? How do I schedule the communications between them? This is a lot of the extremely low-detailed work that they figured out in the public first and potentially like second or third in the world, and maybe even first in some cases.
What lesson do you, in the direction of the bitter lesson, take from all of this? Where is this going to be the direction where a lot of the gain is going to be, which is this kind of low-level optimization? Or is this a short-term thing where the biggest gain will be more on the algorithmic high-level side of like post-training?
Is this like a short-term leap because they figured out like a hack? Because constraints necessitate the mother of invention? Or is there still a lot of gains?
I think we should summarize what the bitter lesson actually is about. The bitter lesson, essentially, if you paraphrase it, is that the types of training that will win out in deep learning as we go are those methods that are scalable in learning and search, is what it calls out.
This scale word gets a lot of attention. The interpretation that I use is effectively to avoid adding the human prior to your learning process. If you read the original essay, this is what it talks about: how researchers will try to come up with clever solutions to their specific problem that might get them small gains in the short term, while simply enabling these deep learning systems to work efficiently for these bigger problems in the long term might be more likely to scale and continue to drive success.
Therefore, we were talking about relatively small implementation changes to the mixture of experts model. It's like, okay, we will need a few more years to know if one of these are actually really crucial to the bitter lesson.
But the bitter lesson is really this long-term arc of how simplicity can often win. There are a lot of sayings in the industry like the models just want to learn; you have to give them the simple loss landscape where you put compute through the model, and they will learn.
Getting barriers out of the way, that's where the power of something like Nickel comes in, where standardized code could be used by a lot of people to create sort of simple innovations that can scale.
This is why the hacks—the I imagine the code base for Deep Seek is probably a giant mess. I'm sure they have—Deep Seek definitely has code bases that are extremely messy, where they're testing these new ideas. Multi-ad lat and attention probably start in something like a Jupyter notebook, or somebody tries something on a few GPUs, and that is really messy.
But the stuff that trains Deep Seek V3 and Deep Seek R1, those libraries, if you were to present them to us, I would guess are extremely high-quality code, high-quality readable code.
I think there is one aspect to note, though, right? There is the general ability for that to transfer across different types of runs, right? You may make really, really high-quality code for one specific model architecture at one size, and then that is not transferable to, hey, when I make this architecture tweak, everything's broken again, right?
That's something that could be, you know, with their specific low-level coding of like scheduling SMS, is specific to this model architecture and size, right? Whereas Nvidia's COLLE library is more like, hey, it'll work for anything, right? You want to do an all-reduce? Great, I don't care what your model architecture is; it'll work.
You're giving up a lot of performance when you do that in many cases, but it's worthwhile for them to do the specific optimization for the specific run, given the constraints that they have regarding compute.
I wonder how stressful it is to, like, you know, these frontier models, like initiate training, like to have the code to push the button that you're not spending a large amount of money and time to train this. There must be a lot of innovation on the debugging stage of making sure there's no issues, that you're monitoring and visualizing every aspect of the training, all that kind of stuff.
When people are training, they have all these various dashboards, but like the most simple one is your loss, right? And it continues to go down. But in reality, especially with more complicated stuff, like the biggest problem with it, or FP8 training, which is another innovation, you know, going to a lower precision number format, i.e., less accurate, is that you end up with loss spikes, right?
No one knows why the loss spikes happen. For some of them, you do; some of them are bad data. I give AI's example of what blew up our earlier models is a subreddit called Microwave Gang. We love to shout this out; it's a real thing. You can pull up Microwave Gang. Essentially, it's a subreddit where everybody makes posts that are just the letter M.
So, it's like extremely long sequences of the letter M, and then the comments are like beep beep, because that's when the microwave ends. But if you pass this into a model that's trained to be a normal producing text, it's extremely high loss, because normally you see an M, you don't predict M's for a long time.
So, like this is something that causes loss spikes for us, but when you have much—like this is old, this is not recent—and when you have more mature data systems, that's not the thing that causes the loss spike.
What Dylan is saying is true, but it's like it's levels to this sort of idea with regards to the stress, right? These people are like, you know, you'll go out to dinner with like a friend that works at one of these labs, and they'll just be looking at their phone every like 10 minutes.
They're not like, you know, it's one thing if they're texting, but they're just like, is the loss, is the L tokens, tokens per second, loss not blown up? They're just watching this, and the heart rate goes up if there's a spike.
Some level of spikes is normal, right? It'll recover and be back. Sometimes a lot of the old strategy was like you just stop the run, restart from the old version, and then change the data mix, and then it keeps going.
There are even different types of spikes. So, Dirk Grenal has a theory too, that's like fast spikes and slow spikes, where there are sometimes where you're looking at the loss, and there are other parameters you can see it start to creep up and then blow up, and that's really hard to recover from.
So, you have to go back much further. You have the stressful period where it's like flat or might start going up, and you're like, what do I do? Whereas there are also loss spikes that look good, and then there's one spiky data point.
What you can do is you just skip those. You see that there's a spike, you're like, okay, I can ignore this data, don't update the model, do the next one, and it'll recover quickly.
But these are like on trickier implementations. So, as you get more complex in your architecture and you scale up to more GPUs, you have more potential for your loss blowing up.
So, there's a distribution. The whole idea of grocking also comes in, right? It's like just because it's slowed down from improving in loss doesn't mean it's not learning, because all of a sudden it could be like this, and it could just spike down in loss again because it learned, truly learned something, right?
And it took some time for it to learn that. It's not like a grad process, right? And that's what humans are like; that's what models are like.
So, it's really a stressful task, as you mentioned, and the whole time the dollar count is going up. Every company has failed runs. You need failed runs to push the envelope on your infrastructure.
So, a lot of news cycles are made of X company had Y failed run. Every company that's trying to push the frontier of AI has these. So, yes, it's noteworthy because it's a lot of money, and it can be a week to month setback, but it is part of the process.
But how do you get, if you're Deep Seek, to a place where, holy, there's a successful combination of hyperparameters? A lot of small failed runs and so rapid iteration through failed runs until successful ones.
You just build a intuition like this mixture of expert works, and then this implementation of MLA works, key hyperparameters like learning rate and regularization and things like this, and you find that works for your code base.
I've talked to people at frontier labs. There's a story that you can tell where training language models is kind of a path that you need to follow.
So, you need to unlock the ability to train a certain type of model or a certain scale, and then your codebase and your internal knowledge of which hyperparameters work for it is kind of known.
You look at the Deep Seek papers and models; they've scaled up, they've added complexity, and it's just continuing to build the capabilities that they have.
There's the concept of a YOLO run. So, YOLO, you only live once. What it is, is like, you know, there's all this experimentation you do at the small scale, right? Research ablations, right?
You have your Jupyter notebook where you're experimenting with MLA on like three GPUs or whatever, and you're doing all these different things like, hey, do I do four active experts, 128 experts? Do I arrange the experts this way? You know, all these different model architecture things you're testing at a very small scale, right?
A couple researchers, few GPUs, tens of GPUs, hundreds of GPUs, whatever it is. Then all of a sudden, you're like, okay, guys, no more screwing around. Everyone take all the resources we have, let's pick what we think will work, and just go for it, right? YOLO.
This is where that sort of stress comes in. It's like, well, I know it works here, but some things that work here don't work here, and some things that work here don't work down here, right? In terms of scale, right?
So, it's really truly a YOLO run. There is this discussion of certain researchers just have this methodical nature; they can find the whole search space and figure out all the ablations of different research and really see what is best.
There are certain researchers who just kind of like, you know, have that innate gut instinct of like, this is the YOLO run. You know, looking at the data, this is it.
This is why you want to work in post-training, because the GPU cost for training is lower, so you can make a higher percentage of your training runs YOLO runs.
Yeah, for now.
For now.
So, some of this is fundamentally luck, still luck and skill, right? In many cases, yeah. I mean, it looks lucky, right? But the hill to climb, if you're in one of these labs and you have an evaluation, you're not crushing, there's a repeated playbook of how you improve things.
There are localized improvements, which might be data improvements, and these add up into the whole model just being much better. When you zoom in really close, it can be really obvious that this model is just really bad at this thing, and we can fix it, and you just add these up.
So, like some of it feels like luck, but on the ground, especially with these new reasoning models we're talking to, there are just so many ways that we can poke around.
Normally, it's that some of them give big improvements. The search space is near infinite, right? And yet the amount of compute and time you have is very low, and you have to hit release schedules.
You have to not get blown past by everyone. Otherwise, you know what happened with Deep Seek, you know, crushing Meta and Mistr and Cohere and all these guys. They moved too slow, right? They maybe were too methodical.
I don't know; they didn't hit the YOLO run. Whatever the reason was, maybe they weren't as skilled. Whatever, you know, you can call it luck if you want, but at the end of the day, it's skill.
So, 2025 is the year of the YOLO run. It seems like all the labs are like going in. I think it's even more impressive what OpenAI did in 2022, right? At the time, no one believed in mixture of experts models, right?
At Google, who had all the researchers, OpenAI had such little compute, and they devoted all of their compute for many months, right? All of it, 100%, for many months to GPT-4 with the brand new architecture, with no belief that, hey, let me spend a couple hundred million dollars, which is all of the money I have, on this model, right? That is truly YOLO.
Right now, you know, people are like, all these training run failures that are in the media, right? It's like, okay, great, but like actually a huge chunk of my GPUs are doing inference. I still have a bunch doing research constantly, and yes, my biggest cluster is training, but on this YOLO run.
But like that YOLO run is much less risky than like what OpenAI did in 2022 or maybe what Deep Seek did now, or, you know, like sort of like, hey, we're just going to throw everything at it.
The big winners throughout human history are the ones who are willing to do YOLO at some point.