Pre-Training GPT-4.5

How many parameters is it still, or do we not care? I think we should just…

Okay, so usually when we do these, it's to talk about a new product that we're about to launch, um, but we're gonna do something a little bit different today, which is to talk about the research that went into our product. When we launched GPT-4.5, we thought people were going to like it; we were very proud of the model. But people liked it much more than we thought. People said all kinds of things like, "I never thought I was going to have this experience talking about a model; it's so different than GPT-4; it's way better in these ways that are either obvious or hard to explain, or this or that." But, uh, there was like a lot of interest about what went into making GPT-4.5. So today we have some of the key team that made GPT-4.5, and we're going to talk about it. Uh, we're going to talk about sort of like what went into it, what we learned, and just like what it takes to make a giant model like this. Um, actually, maybe we start with that: What does it take to make a giant model like this? You go first.

Uh, a lot of people, and a lot of time, and a lot of compute. Oh, maybe you guys should introduce yourselves. Alex, you want to start?

Sure. Uh, yeah, hi, I'm Alex. Uh, I work a lot on pre-tuning data; I also led a pre-tuning ML for GPT-4.5.

Uh, I'm Amin Chian. I… I'm OpenAI’s chief system architect; I oversee systems and networking broadly at OpenAI.

I'm Dan; I work on data efficiency and algorithms.

Yeah, okay. So what… what goes into it?

Um, yeah, so I think we started this project basically two years ago, or so, um, and uh, we kind of knew that we had a big new cluster coming online, uh, and we… we kind of saw this on the horizon, and we started doing a bunch of work to kind of convince ourselves, uh, of the features that we wanted to include in the run, doing a lot of large derisking runs, building out a very long plan for this, um, and kind of across the full stack, from systems, ML, everything, um, and yeah, it was… it was a long story of execution for uh, derisking it, uh, and kind of preparing for the run, um, before the run itself, which, you know, itself was a very large endeavor.

Yeah, I think it's a process that starts at inception with a collaboration between the ML side and the system side and goes all the way to the time that we know what model precisely we want to train, and then it's starting the uh, run process in itself, with the pace that we are working at, and especially trying to make use of our most recent computer that is made available to us, it becomes something that is difficult to priority plan perfectly.

Mhm. So we almost always go into a launch with a lot of unresolved issues, um, and try to make forward progress throughout the run despite all the challenges—basically, add more compute, resolve all the issues that we probably might not have anticipated, despite all the projections that we had, both on the ML side and the system side, and try to basically close the gap between what we predicted should happen and what is happening. Uh, I think that is, at a very high level, uh, and broadest stroke, is the entirety of the process, and the tail end of it is the execution, which takes a lot of people, a lot of energy, and momentum for a prolonged period of time to go through the training process.

How… how close do you feel like we were to what we expected to happen to what actually happened?

Um, on the… usually at the beginning, uh, I'm talking about the system side of it, uh, we are usually far away from where we expect it to be, and there is always a choice to delay the launch and basically defer until more and more issues are resolved, or launch early and try to basically figure it out as we go. It's always a balance of figuring out not to delay the process unreasonably, uh, but almost always there are issues that we don't necessarily know at the inception that we are going to run into, and the entirety of the process is try to uh, handle the known to the extent that we can and have a plan for what… how the run should go, and as we make progress, just deal with the unknowns, which is the variability on, let's say, if a run is successful, how long it would take, and so…

How far have they been?

Well, yeah, I think… I guess, one at the highest level, I guess with this project we set out to do GPT-4.5, which means like 10x smarter than GPT-4. So that was sort of, I think, the… the initial goal starting like two years ago that we had set our sights on, um, and then there's a lot that kind of, you know, kind of happened along the way of like, oh, we think, you know, can we do better or worse. Um, I think uh, it was a very complicated row, but in the end we got to a model that we feel hit this mark of 10x smarter than GPT-4, um, in terms of kind of the effective compute that we… we put into it.

Yeah, on this execution side of it, of course, initially it was far, far away from how uh, long we thought… it did take longer than we thought. Uh, yes, but I think the process is to try to shorten it to basically match what we would…

Two… two-part question about that: Why do… why does going from, you know, making up numbers here, 10,000 GPUs to 100,000 GPUs, why does that make the problem much harder?

Um, a lot of issues. It's… I… I do believe that issues that you observe at the scale, if you have a very keen eye, you would observe them at a smaller scale. It's not that they only manifest at larger scale, but something that is a rare occurrence becomes something that is catastrophic at a scale, uh, especially if you haven't anticipated it. Being…

What are some of the kinds of things that have become captured? I mean, among those things is that I think is quite well known is uh, issues with uh, the infrastructure, uh, the failure rates that you observe, the… the variety of failures that you observe, uh, in both in terms of the types of failures and also the… the count itself. So we get to observe something that I'm sure the vendor hasn't observed, because this is a large pool of samples, and we get to observe the entirety of the statistical distribution of uh, a large pool of resources that… that we are executing on. The fabric, the network fabric is always part of it, the individual accelerators part of it, but at the end of the day, this is the beauty of it: at… at the same time that almost everything needs to work as expected for the result to hold, and the job is to basically minimize that variance.

Second part of the question: Obviously, it's really hard—this is for all of you—really hard to do things at the edge of scale, um, so, you know, even going as we go off and do the next training run, even kind of crazier, um, but I've also noticed it gets much easier to go do things that are now no longer frontier. So it took like hundreds of people, almost all of OpenAI's effort to do GPT-4.5. If you guys could go pick whoever you wanted, what is the smallest team from OpenAI that could go retrain GPT-4 from scratch today with everything we know and have and all the systems work?

I think to get to a GPT-4 level model, it's probably on the order of maybe five to 10 people.

Yeah, we did it with that type of… that number of people. With GPT-4.5, it was different in the sense that a lot of work history was… a lot more people come together, and it was a very different effort than before, I would say. But now that we've done that work, I think like the stack we've improved a lot, and like if you were to retrain… I mean, we kind of did this a little bit in the pro… process of training GPT-4.5; we trained GPT-40, which was a GPT-4 caliber model that we retrained using a lot of the same stuff coming out of the GPT-4.5 research program, um, and I think doing that run itself actually took a… a smaller number of people, right?

What about from your perspective, Dan, or just sort of like why is… why is training big models hard?

I think doing anything new is hard. I think uh, even just finding out that someone else did something, it becomes immensely easier, because the hard part is having the conviction to do something in the first place. I feel like just the fact that something is possible is a huge cheat code that just makes it…

Yeah, yeah. I mean, we're always… we're scaling 10x beyond what we did before with these GPT pre-training runs, and it's… there's always new things that you find that are interesting that you couldn't have anticipated necessarily.

What do we need for the next 10x or 100x in pre-training? Scale, data efficiency… what does that mean? Easy answer, obviously, I know, but what does it mean?

So the… the transformer, the GPT, is spectacular at making productive use of… of data; it… it absorbs information, and it… it… it compresses and… and generalizes to some degree, but it's… its defining character, its signature, is absorbing the information very efficiently with compute, but there's a somewhat of a ceiling to how deep of an insight it can gain from the data, and so at some point, as the compute just keeps growing and growing and growing, and the data grows much less quickly, the data becomes the bottleneck of this standard paradigm, and it requires some algorithmic innovations to be able to learn… spend the compute… more compute to learn more from the same amount of data.

What do you guys think we need to keep scaling in addition to that?

I… I think this answer is system side. I think even between different GPTs that we have trained, GPT-4.5 was the sheer uh, volume of work that was required to… basically, the changes that were required for us to make uh, was a byproduct of the model specification. The same… we wouldn't have been able to train uh, GPT-4.5 on the precise same stack as we did GPT-4. So, let's say, state management; our approach with state management changed; we had to scale to more compute, and that compute was not available as part of one cluster; we had to go to multi-cluster training, uh, and imagine it's many, many different work streams like that that have to come together in a short period of time for us to be able to do this, and for making another 10x jump, of course, and other issues that pri… previously knew that they do exist, it's just that it's a choice for expediting execution that we skip for this one; for the next one, we have to do it; there's no way around it, and it's always those choices that basically make the timeline of… do… building the perfect system take much longer, uh, so we are always compromising on what is the fastest way to get to this result; the systems is not an end in its own; the product that… the thing that it produces is… so for the next 10x, for me, it would be uh, of course, fault tolerance, not uh… and a form of fault tolerance that we can co-design with the workload such that we don't have to worry… the operational burden of keeping uh, such a massive run going is not like our prior system. So I would argue that with our prior stack, 4.5… but at the edge of what we could keep up with…

Do you know what percent of steps failed in the 4.5 run due to some component somewhere?

I actually don't have a number off the top of my head, but it is usually the way things work. This is a fascinating thing; there are issues that are early on uh, in the lifetime of a new generation of hardware, is not necessarily well understood or well studied. We start the process, and we want to make forward progress in presence of such issues; of course, the failure rates are quite significant earlier in the run; they're not necessarily… it could very well be that once we find the root cause and eliminate it, the total number of failures significantly drops, and this is often the case; it's just that uh, we learn more… the infra… that is… some would call it clean up of the infra, or understanding uh, fundamental issues about the infra uh, and the state improves significantly, but that earlier phase of execution is almost always quite painful because we are learning about what are the new failure modes in the new uh… uh… and the new infrastructure while making forward progress. Of course, later on, um, the failure rates drop significantly; the uptime improves overall, and so on, but it's just a matter of… prior… it is hard to predict what this early phase of a generation of infrastructures lifetime failure risk would look like, and designing for the steady state might result in very poor um, availability earlier on in the process.

Obviously, reasoning models are a huge part of our future, and you know… but… but if we were going to put this aside for a second and think about just how far we could go on classical pre-trained models, assuming we had unlimited GPUs and, you know, unlimited networking and unlimited power, but we were stuck with all of our current problems—the stuff still broke, we didn't figure out fault-tolerant training, we only had the data we have, whatever else—how… and we kind of use the convention of each major number of GPT is 100x increment, um, how far could we go, like GPT… what could we train with what we know today?

I think on that we get to like a 5.5, ML… on the algorithm side, I don't… I don't think there's like a clear limit that we… we found. I think um… yeah, I think we're just kind of scratching the surface of more data-efficient algorithms, uh, and I think better ways to leverage the data that we have. Um, it's very interesting because I think up until this rough point in time, like if you look even through GPT-4, we were largely just in a compute-constrained environment, um, so that was kind of where all the research was going into, but now we're, you know, in a very different kind of regime, um, starting with 4.5 for some aspects of the data where we are much more data bound, um, so there's now… not a lot more excitement about this research. It is a crazy update that I don't think the world has really gro… yet. I should pick a different one… that the world has understood yet, uh, that we're no longer compute constrained on the best models we can produce; that's just… that was… so the world we lived in for so long…

What was the most interesting ML thing we learned during the 4.5 training that you want to share?

Oh, gosh… um… or either of you… I don't know what I could share in general terms. I think it is being off of the prediction and trying to figure out why we're off of the slope that we predicted to be on.

Yeah, I think that most surprising… maybe we can… yeah, I think one of the more surprising things that we found, um, was just uh, kind of how different aspects of what we were working on on the ML side and what we put into the run scaled, and… and some things that did or didn't scale well, um, that we… we kind of discovered through the process of training this model, um, it was… it… it's… yeah, we've learned a lot through this. I'd say the… the two defining characteristics of the GPT paradigm has been that the law… you can predict the test loss, and it scales predictably, and magically lower test loss means greater intelligence in all these intangible, amazing, mysterious ways.

Are you a maximalist on that? Do you like fully believe that?

Well, I was going to say that one of the interesting things we found from 4.5, we tested it again, and the model has all of these incredibly nuanced abilities that were not in… in anyone's bingo card specifically; we just… like the conviction was it's going to be more intelligent in ways that are really hard to characterize ahead of time, and then you see in the deployment, you see in the… the user satisfaction that it's smarter in all these very subtle ways; it has more common sense knowledge; it understands nuance and context, and that's… that's the magic that came out of the few extra bits of test loss, and I think that that part of the scaling held up really well.

What was the most like positive moment of the whole training run? What was like… what's like the favorite memory? There's obviously lots of pain, but hopefully that's somewhat fit…

I… I do have one, but… yeah… yeah, I can go. Uh, yeah, so one… one that comes to mind for me, um… we… uh, kind of… we’ve… we worked a lot on the ML of the run during the run itself as well, and I think some of the… the changes that we made during the run had a quite… quite good impact. I think uh… maybe perhaps better than anticipated, and this was a pretty exciting moment for us.

Yeah, I… I think for me, this was probably the biggest effort in terms of… um… I see… time during the run, while we are building… I mean… I mean, running things, we are building things in parallel, um, of course, to get there faster, we are parallelizing work aggressively, um, and there is conviction that it will pay off; we will… be… passes performance cliff that makes the model essentially untrainable in terms of the time it would take to… to train, um, and there is a plan; everybody’s executing on it, but it's taking long; it is a hard… it's hard work, definitely harder than I imagined it would be; my projections were off uh, by how long it would take to basically get uh, those issues resolved, and seeing… I think seeing the moment that the whole team… once a few of those issues got resolved, we got a big performance boost after… I remember that everybody got… I mean, you could sense that the energy changed; it’s just that everybody feels excited and now more motivated to push uh, everything through the end; it’s just… it’s fascinating part of… see the ETA on our status tracker, like… yeah, has constantly shifted from, let's say, two years to uh… something uh, tangible, uh… then the effect it has on the morale of the team and everything else is… I think that was the beauty of it. The other part I would uh, call out is the ML side of it didn’t stop; the ML code design part of this didn’t stop at the launch of… at the launch time, and people tagged along issues that were left to say, “We’ll figure it out later.” People were actively working on it and shipping things that helped um, with execution time, uh, and I think that spirit of teamwork and basically not having team boundaries of, “I did my work; I pass it on to you,” is something very powerful.

I was going to say there's been a lot of focus on how the run itself was challenging and predictions were… but this is despite a tremendous amount of sophisticated planning.

Oh, this is… of course…

Do you want to say more about it? By far the most planning…

Yeah, I think… I mean, like I said, we… we started basically working on this project like a year before we um, even started training the run itself, um, and through this we had a number of very large de-risking runs; we… we took a lot of care to kind of very carefully sequence all the changes that we wanted to put into it, starting from sort of like very high-confidence, known-good config; you can think like GPT-4, you know, we really understand this… this setup, um, from an ML perspective, and just layering things in and being very careful to… very carefully study the… the scaling uh, of all of the changes we’re making, so it’s not just good enough that we see some amount of win; we also want any… any win from any new feature to be like persistent across scale and not to be tapering off. Lots of things look good at small scale; don’t look good at large scale, uh, so we had to be very, very paranoid through this process, um, and really, I think we continue to iterate on our scaling laws methodology; we learned a lot on that front uh, through this de-risking process uh, that’s continuing to guide us for future GPTs.

I… I remember something… I miss about… another very fun moment, um… so there… this is the torch sum, uh… but imagine it is very unlikely that we launch a run and it doesn’t have bugs; I mean, all sorts of bugs; it is just a given that uh… but we need to make forward progress, and we need to be sure that, okay, are we actually sure that this is on track, and these bugs are not super negatively affecting the health of the run? While we are absolutely sure… we were initially very sure that there are bugs that are consequential; we do… we have built a lot of systems around giving us visibility and ability to distinguish between: Is it a hardware fault? What type of hardware fault it is? Is it some form of corruption, or is it some ML… potentially an ML bug or something like races in our code? Um… what happened was that we… we, of course, had a few open threads about all sorts of issues, that different symptoms uh… all correctness-related issues, um… and at the end of the day, uh… so… so of course we have found bugs, fixed them, and so on; we arrived at this point that we have multiple open threads, and there’s a lot of uh, thought around what is causing all these; are they different bugs, or it’s one bug and causing all of these? Uh… people went… I mean uh… around the table and say, “Everybody vote, which one do you think is the most probable cause of this bug?” And the one that turned out to be the bug got the least votes. It’s just that it is a torch sum bug, a simple summation and uh, upstream PyTorch… and no…

What was the bug?

So the… the bug is… for this… this is funny because uh, for this specific code path, we were triggering it, and for the… for context, remember that we are mostly using Triton kernels; it’s just that for some corner cases that let’s say the ops don’t matter much, we basically fall back to… I mean, it is okay to run Torch ops, uh, and our specific uh… the specific code path I… or data triggered for uh, in the torch sum function uh, was basically had a bug, um, and it was only occurring very infrequently; it’s just that it’s a… it’s data distribution dependent, and it was triggering in… in a good case, illegal memory accesses, because it was computing some offsets and some bit of memory. The fun revelation at the end is that once uh, somebody fixed the bug, I mean our engineers figured out, “Oh, I found the bug; it is this line; it’s torch sum; let’s ship a fix and see if it fixes everything,” uh… it fixed all the… way of… sanding bugs that… where they had seemingly distinct uh, symptoms, and it was quite fun, and everybody… I think we were renaming the Slack channels of multiple theory to single bug theory, or uh… and… um… that was quite a lot of fun to basically… okay… how… when… in the… when did that happen? I can’t remember now; it was live from very early days of the run up until… I… I think good portion of the run… I would… 40% in event…

Do you guys remember how someone found it?

Uh… I… I do remember that… I think it was in the list of… so imagine there… there’s a sequence of uh, kernel invocations, and the um… the second one was the one that was triggering illegal memory accesses, and that was some very complicated uh, kernel that we wrote, um, and of course our team would suspect that there is a bug there; obviously, there must be a bug there, and pe… I mean, several very clever people, just line by line, everybody is looking at it; a bug was found, but we rolled it out; some of those issues disappeared, but not everything, and at some point somebody was… I mean, in the list… torch sum was the one feeding inputs to this kernel, one of the many inputs of this kernel, and somebody started uh… one of our engineers started looking through the code and different code paths; I said, “Oh, this very unlikely code path that probably most people don’t hit, we do hit,” uh, and he said, “Okay, this line is buggy; let’s…” I mean, of course, the only verification and validation we have for it is ship the change, observe if all the crashes disappear, and it did disappear. I think…

That was the validation we needed, so but uh, it was t—I mean, this is the thing: it's just that it's a crash at a very, very—I mean, a slow rate. It's one every hundred steps, one every thousand steps, and it's something very easy to dismiss. But it's just that we should not have that in the run, as a discipline that we should—we do have—and it's just not giving up on it is the story.

Alex, I understand what your life is like. I think most people can imagine it like leading up to like pushing go on the run, but after that happens, like what is your day-to-day like? Are you just like sitting there watching loss curves? Like, how does it go?

Definitely a lot of watching loss curves; we all did a lot of that. Um, no, I think uh, past that, there's a variety of things: still trying to work with with systems uh, to uh, kind of, you know, get out improvements that we didn't get in um, on the code design front uh, before launch time. Um, there's a lot of things that we try to continuously monitor of the run to see if anything is kind of trending like we're not uh expecting. So this is loss curves, but there's a bunch of other statistics that we can look at. Um, and then, yeah, kind of uh, working a bit towards like any other kinds of improvements we could we could do to the run as well, from an ML perspective. Uh, so on the data front, it becomes less busy immediately once you click go, but uh, otherwise, there's still plenty to be done. I, I think what we lean on for ML is a lot of correctness. Imagine there's a lot of noisy signal, and you are at times reading tea leaves. It's just, is this healthy or not? Of course, if you wait long enough, you would know it was—it was—it healthy or not. It's just that the responsibility—and how often are there false alarms? There like, how often are you like, "Oh, this is looking really bad," and then it's fine?

Pretty frequently, I think probably about half the time, maybe, because we're a paranoid bunch. So I think, yeah, if it wasn't half the time, we wouldn't be looking closely enough.

I have a short lightning round of questions.

Sure.

If you could have any one ML question answered before the next big run, what would you most like to know?

I think uh, what we should—what—what algorithms we should employ with uh, for—for limited data in certain domains is the main thing. It's a kind of vague question, but big answer.

And if you could have any change to current hardware—you could have like a new kind of network invented or like a totally new chip architecture—what is the most—what is the limiter on systems at this point? Not—you don't get to like say like, oh, I want—yeah. So where—at this is a transport level—networking transport level change. It's just that where there are faults that uh, could be worked around uh, at a different level than the application level, I would rather uh, the transport—the network transport do its job and uh, keep running and give me the available bandwidth without me worrying about it.

Is there anything promising on that front?

Uh, yes, we can talk about—

Okay, that's good, at least.

Two-part one for you, Dan. Uh, how—so on the data efficiency question: humans, for whatever other flaws we have about learning things, we seem unbelievably data efficient. Yeah, how far away is our very best algorithm currently from human-level data?

Really hard to measure apples to apples, I think, just like vibes by—in language—astronomically far away. 100,000 x 00x something in that—in that range. Uh, it depends on whether you count every bit of pixel information on the optical nerve, but—but we don't know algorithmically how to leverage that to be human level at text. So I think algorithmically we're yeah, quite—quite—quite far away, and it—apples to apples.

And then part two is: do you think with our current—our like the direction of our current approach—we will get to human-level data efficiency, or is that just not going to happen and doesn't matter?

Well, I think for—for decades, deep learning has been about compute efficiency, and what's—what—what's magical besides the data and compute growth is that the—the algorithmic changes stack so well. You've got different people, different parts of the world finding this little trick that makes it 10% better, and then 20% better, and they just keep stacking. There just hasn't yet been that kind of mobilization around data efficiency because it hasn't been worth it, because when the data is there and your compute limited, it's just not worth it. And so now we're entering a—a new stage of AI research where we—we'll be stacking data efficiency wins—10% here, 20% there—and I think it would be a little foolish to make predictions about it hitting walls that we have no reason to predict a wall, but—but it's there. The brain certainly operates on different algorithmic principles than anything that's a small tweak around what we're doing, so we have to hedge a little bit there, but I think there's a lot of reason for optimism.

This next one is the same question for all three of you. Uh, yes or no, or you can add—you can add—you can add explanation, too. Will humanity ever do a 10 million GPU or greater synchronous pre-training run?

I don't know if it'll exactly be a pre-training run, but I think there'll probably be some kind of training run that—there will be 10 million GPU training.

Yeah.

Yeah.

I don't know if it'll look—it'll probably look totally different than what we're doing today, but there will be something that is kind of in the spirit of unsupervised learning that is like at that scale.

I think so.

I think so.

I would call it semi-synchronous.

Uh, and the scale of it—

I hope so.

I think it sounds very interesting.

You would call it semi-synchronous?

You said yes?

I think—not fully synchronous, but it's just—laws of nature can't bend it completely.

I think it's likely to be more decentralized than that. There'll definitely be 10 million GPUs working together on an AI system that is learning and doing things, but it might not—all the parts of the brain won't necessarily all be talking to each other.

That makes sense.

Um, can you say anything that we've learned or observed about how uh, smarter and bigger pre-trained models correlate with how good a model is at learning reasoning?

Um, yeah, so I think what we've kind of observed is uh, it is better pre-training, and unsupervised learning just tends to lift kind of broad-based intelligence of the model and aid a lot in generalization uh, which we found to be like really complimentary, I think, with reasoning, which can tend to be a little bit spikier, lumpier in terms of like where it's increasing intelligence. Um, so yeah, I think they're—they found them to be good compliments.

To go off on a little bit of a tangent, do any of you guys have like a intuition of—is it weird, or is there anything to take away from the fact that pre-training seems to be so general across everything, and when we teach a model reasoning, we can get it so good only at one category of things?

Yeah, I don't—I don't know if it's the most—Yeah, I think—I think it's interesting. I think it's kind of not surprising to see this out of pre-training when you're just look at what you're—what you're training with. Um, you're—when you construct like a training data set for pre-training, it's inherently very broad. We're—we're targeting breadth and diversity. Um, and I think it's—it's kind of difficult to always get the same breadth when you talk about doing RL and having like environments that you can kind of cleanly get uh, good reward signal out of and good—good environments out of.

I—I—I agree, but I think there's another factor that pre-training is essentially—it's compressing the data, and compressing the data is about seeing connections between different things. It's about analogies, it's about uh, abstractions, and reasoning is on a particular problem—it—there's like—there is a skill and a craft to thinking carefully, and thinking carefully can unlock many different kinds of problem solving in different domains, but there's something uh, more learning at a more abstract level when you're compressing across domains in the way pre-training does that—that makes—

Oh, that—I'm going to change my question for you in a second. I just thought of something else. Um, what's going to limit us in terms of systems progress—chips, processors, memory, network, or power? Like, what—what will be the bottleneck of continuing to scale this up?

Um, it is—this is the beauty of systems uh, in that if you do code design, the workload is adaptable to the infrastructure that you built. Um, there is—I think—let's say there is no statement that broadly network is a bottleneck or memory bandwidth is a bottleneck or computer is a bottleneck. We have the option to basically shift, and even for the same specification of the model, we have the option of shifting resource demands to basically create a more balanced system. However, having said that, I think—let's say the pre-training answer is different than the inference answer, and so on, but it—it never hurts to have more memory bandwidth. Uh, I would—yes, I think this is a hard question to answer, and without qualification—

Speaking of that earlier point, how much do your teams work together on like the spec of the model as we get ready for the 4.5 run? Like, how much—how much is it you're just like, this—

Yeah, very closely. I mean, down to like the shapes of the—the map moles that we want to be doing, um, making sure those are optimized well. Um, but for this project, it was a much deeper collaboration going back um, to like, I guess six or nine months before the launch of the run, um, trying to do for some of the—the functionality we wanted to put into the run and like aspects of what we needed to do to make 4.5 happen. We took on like a—a specific like very large d-risking run uh, that was specifically focused on co-design with systems uh, to make kind of the—the ML and the systems work together well at scale. Um, so there—we—I think it's the first time we put like such a large emphasis just on the codeesign aspect. Was really key.

Yeah, I think that was the first big um, scale effort that I—I remember that it is not just about fine-tuning one aspect of it is that fundamentally you want a property—whole—to hold system-side, and that property doesn't emerge out of nowhere. You really need to steer the system to give you that property. Uh, so that code design effort is something uh, that uh, formed the architecture and architectural elements that goes into the model and somehow ties the system and ML side together. It's probably a property we would not prefer to have.

Yeah, I would ideally want everything to be decoupled to basically give maximal room to each other, but at times things get tied together where you are really catering to the requirements of your infrastructure or how things should be. I mean, oftentimes it is—you really want to have a balanced system uh, and balanced communication and a very symmetrical type of system, and you have the best knob we have at our disposal is all code design.

How close are we to the idealized mean system?

Do you mean being like fully happy—we have all the hardware we want, it works for what we know about ML?

We are nowhere near that.

You think it is—

We are nowhere near, but it is fun. It is—the—the practice of building systems is always about that—that you have an idealized view of how things should work, and it's just about reconciling the differences of that with what you have. I think we are not doing theory for the sake of—let's say—just talking about what we want it to be. We just want to make it happen and approximate that ideal to the degree that we can. So to be honest, I think this u is probably as exciting as it gets for systems—you—be—get to really come up with hypothesis of what is a good system design uh, and take it from there and basically see the results in action very quickly. Things that previously one would say—this is an elegant system design—and uh, we have to just—history will tell us if this is the right thing to do or the wrong thing to do. We have a lot of compute, we have a problem, we—we—we know the target, and just—we go and see if our choices were correct or not.

How much is your team thinking about the sort of constraints of system design when they're trying to decide what—what's going to go into the run?

Yeah, I think it's—it's a huge consideration for just doing pre-training runs at large. Um, I think like since 4.5, a lot of the work on the architecture side—there—there's also kind of ongoing threads around further code design, further places that we can design for future hardware uh, build together. I think there's been a lot of promising uh, work already since then.

Okay, my changed question for you: why does unsupervised learning work—compression so—

So the ideal intelligence is called Solomon induction. Basically, it's—it's uncertain about what universe it's in, and it imagines all possible—un—it considers all possible universes, with simple ones more likely than less simple ones, and it's—it's fully basil in head and—up—updates its views as it progresses. And you can approximate this by uh, finding the—the shortest program that computes everything you've seen so far. And what we're doing with pre-training—or one way to think about what pre-training is doing—is it is compressing—it is trying to find the shortest program that explains all of the data that humans have produced so far as a way of approximating—

And why does looking at next token prediction do that? It's actually a subtle question. There—there was a paradox—or somewhat of a paradox—in statistics for a long time: why do deep networks generalize when they don't seem to compress? Normally in statistics, you have lots of data, and you got small models. The models predict the data; therefore, you can—the models must have compressed and must have learned something. In pre-training, generally the models are pretty gigantic, and they scale roughly with the—with the data. Uh, so it was always a question: are they actually compressing? Are they generalizing? And of course, there are critics who have said, well, it's just memorizing and interpolating and superficial. Um, but there's a way of viewing pre-training such that you do see that it is a compressor in a—in a different unintuitive way. And basically, the idea is called prequential compression, and the idea is that the fact that it learns quickly during training means you can turn it into a great compressor. So even though the weights are big, the binary doesn't need to store the weights; the binary can pre-train from scratch to decompress. And so the fact that it's learning really, really quickly means that most of that data you can encode with very, very few bits. So basically, for a subtle reason, it really is quite a good compressor, and I think that is a—a pretty satisfying explanation for why it really does lead to intelligence.

You guys have anything to add?

No, it's great. Thank you. Uh, one—one thing somewhat relatedly that hasn't come up yet is the discipline of metrics. It's like the—the thing—the thing you get when you do these scaling laws and you—you do all the ML science is very dependent on the metric that you choose.

What do you mean—what do you want to say?

Oh, yeah, just talking about the—the—what test side you're—you're evaluating your perplexity on. So—

What you're referring to—even—even looking primarily at perplexity—

Oh, yes. There's already some viewers might think we're looking at college tests or something.

Oh, yeah. Um, so yeah, I mean, do you want to explain perplexity then? I think it's worth it.

Yeah, so it's very—it's very tempting to try to evaluate your model for intelligence on things that are legible to humans as tests, uh, but if—if you—if you do this, probably you're going to be favoring changes that make memorization easier at the cost of making systems actually smarter, because almost every test that we have out there's something similar online. Like, if you can actually train on the entire internet, tests become somewhat uh, degenerate compared to tests for humans where they can't do that. And so the—the main approach in the field is to look at how much it compresses some held out data that's thought to be good data. And e—even then, if you're not careful about that held out data, it's too similar to your training data; training changes to your training algorithm that make it memorize better will seem to make it smarter because it'll have already—it'll already know your test set. And we don't want to just be measuring memorization—after memorization—we're after generalization. And so out-of-distribution generalization. So yeah, this is why—I guess—yeah, maybe you're alluding to—so like the—the key test sets that we're looking at, we—we care a lot about them not being present in any degree—to the slightest degree—uh, in our training set, because that kind of throws off all—all the way that we do our scaling laws. Um, so yeah, this is a very key point.

With—what's our best thing for that? Uh, so our—our internal codebase uh, which we know is not out there—yeah, it's a very good held out—has that held—as our best thing across like many—it's still the best thing.

Yeah.

It's remarkable. I mean, we joke that a model is its monorepo loss—like everything else—there's some incredibly meta recursive thing there.

Oh, yeah. Somehow you—you—you pre-train the model, it has a monorepo loss, and somehow this tells you so much about how it's going to be behaved—down—it's—it tells you a lot about how a philosophy grad student is going to find the nuances of its responses, but it's incredible.

It is incredible.

Related to that, and sort of last question in some sense: this whole effort, which was hugely expensive in terms of people and time and dollars and everything else, was an experiment to further validate that the scaling laws keep going and why—and turns out they do—and they probably keep going for a long time. Um, I accept scaling laws like I accept quantum mechanics or something, but they still don't—like—I still don't know why—like, why should that be a property of the universe? So why are scaling laws a property of the universe? You want—

I can—I can take a stab. Well, the—the fact that more compression will lead to more intelligence—that has this very strong philosophical grounding. So the question is: why does training bigger models for longer give you more compression? And there are a lot of theories here. There's the one I like is that the—the relevant concepts are sort of uh, sparse in the—in the—the data of the world, and in particularly it's—is a power law, so that the—like the hundreth uh, most important concept appears in one out of a hundred of the documents or—or whatever. So there's long tails.

Does that mean that if we make a perfect data set and figure out very data efficient algorithms—I mean, can go home?

It—it means that there's potentially exponential compute wins on the table to be very s—sophisticated about your choice of data, but—but basically, when you just scoop up data passively, you're going to require 10xing your compute and your data to—to get the next constant number of things in that tail, and there's just—that tail keeps going—it's long—you keep—you can keep uh, mining it. Although, as you alluded to, you can probably do a lot better.

I think that's a good place to leave it. Thank you guys very much. That was fun.

Yeah, thank you.