Transcription
Okay, let's get started. So, today is the second day on data. Um, last time we talked about how data doesn't really just fall from the sky. You actually have to think about where it comes from. Um, so in general, the internet consists of live services. Um, these, um, the data on those services have to be either dumped or crawled. And then there's an additional step where you have to process the data. And we also talked about various, um, social societal considerations, um, terms of service, copyright. You have to either get a license or appeal to fair use. So there's a lot of complexity that goes into data. Um, so today we're going to look more at the data pipeline, from transforming the data, filtering the data, deduplication, and mixing. Um, and then we're going to top it off by talking a bit about, um, post-training data, um, in particular, how folks are using synthetic data these days. So the first part is going to be mostly about pre-training. That's what you should have in mind.
So we talked a little bit about data transformation already, but just as a kind of reminder. So raw data doesn't come as text. Even if you've, you've scraped something, if you ever look at inside Common Crawl, it's, it's not text. It's either HTML. Sometimes it could be PDFs or directories in the case of, uh, GitHub. So most of the tension on transforming data is dealing with HTML because most of the web is in HTML. And for this processing, there's a lot of, this is fairly heuristic, um, there's removing of boilerplate, um, like navigation and ads, um, and extracting content, um, which is like the main part of the page. And there's some, you know, subtleties around what constitutes content. Usually, um, you get rid of the footers and headers, um, and maybe menus and those things, and you try to extract the content. But you could imagine in cases where some of the navigation elements might be, might be helpful to learn what web pages look like. So what is content and what is not content is not always clear. And then what do you do about images and tables on in web pages? Um, so inherently, this is a lossy process because you need to linearize HTML, which at least is either hierarchical or visual if you think about the rendered, uh, output, and do a sequence of tokens. In particular, tables are, um, a bit tricky to deal with. Simple tables you can render, um, using markdown, but if you have nested tables, then that becomes quite, uh, challenging. You have to give up at some point or approximate at some point. Um, so typically HTML to text processing is, is rule-based. And the reason for this is that rule-based, um, processors are very fast, um, and and also, um, you're not trying to do too much here. You don't need too much intelligence. So rule-based generally works. Um, now, I think there could be a case for model-based, um, interventions at this point. They have to be very fast and they have to, you know, do something, um, more, more intelligent. But if you ever look at data, you'll notice there are, um, imperfections in the data just because any rule-based, um, processing is going to be, um, is have some failure rate. As we showed last time, the accuracy, uh, does matter depending on which tool you choose. Um, if you use Brazil parse or Trafilatura, then it's, you know, I guess Brazil parse on these extended DCLM, you know, eval, um, works better, um, than the others.
Um, I'll talk also briefly about PDFs. So there's this, uh, Hugging Face work, um, which released a dataset called `findPDFs`. Um, so PDF files, if you have open them up not in, um, PDF reader, um, looks, looks kind of like this. Um, and this needs to get kind of rendered. Um, so PDFs can be found on just like web pages on the, the internet. And Common Crawl, um, does, um, have, you know, some PDFs. Um, generally Common Crawl focuses on text, but sometimes if you're just given a URL, you might not even know before you fetch it whether it's a PDF or or not, if it doesn't have the extension. Um, there's a lot, if you read this blog post, there's a lot of details which I'll spare you of. Um, for example, one thing is that, uh, many of the PDFs that are in Common Crawl are truncated because PDFs are big. So then that demands that re-crawling is necessary. And then there's a question, once you have the PDF file, how do you actually convert it to text? Um, and, you know, some PDFs are might be just scans as well, right? So they're essentially images. So there's a bunch of tools that, um, this paper, uh, tried out, um, but mostly this kind of involves, um, running OCR, um, using a VLM. And this obviously can be much more expensive than what we were doing before with text. Fortunately, or unfortunately, uh, the PDFs are a very small fraction of the whole internet. But, um, they are very valuable, um, because generally, if you bother to make a PDF, that means you probably have something, um, interesting to say, as opposed to a web page. So the, uh, quality of a PDF is, average PDF is generally higher than, um, for, um, for an HTML file. There's a lot of cleanup and filtering in PDFs, more so even than than web pages. A lot of layout information is missing, um, because, um, in HTML you have various tags like H1 and P that give you some semantic information. PDFs are by design all about layout, so you don't necessarily, um, they preserve the, the kind of the semantic structure. Okay. So there's a bunch of transformation information that happens at this point. You have, um, text, um, um, and but you're not done. You're far from done.
So the next step is filtering. So, um, I'm going to talk somewhat abstractly about what filtering is. Here's a sort of the building block. So suppose you have some target data that you, uh, want to get. This is usually a small amount of high-quality data and lots of raw data. This is the, the, the fresh shipment of the, uh, the tokens that you do transform from the, the previous step. And the goal is to find, uh, a subset of this raw data that is similar to the target. So this is the general skeleton for filtering, and almost all kind of filtering falls into this, uh, this, uh, schema. So filtering, you could, there's many reasons you might want to filter. Uh, if you're training, um, an English language model or a German language model, you might want to identify the language and filter things that don't match that, that language. Um, the main reason for filtering is, is quality filtering. You want to find things that are high quality as opposed to low quality. You don't want just, you know, spam. You want, um, encyclopedia information. Um, and and then another application is, uh, toxicity filtering. Of course, the internet has plenty of nasty content, and maybe you don't want to train your language model on that.
Okay, so for a filtering algorithm, you want that you want some generalization from the target data, right? Because you already have the target data. You don't want to just only get the target data, but, um, you also want it to be extremely fast because you have to run it on the whole internet. So this could be, uh, 100 trillion tokens worth of worth of data. So generally filtering, you end up with a very small fraction, single-digit fraction of your entire, um, data. Okay. So, so remember the general framework: given the target, and raw, you're trying to find a subset. Okay. So the general scheme is that you estimate some sort of model based on R and T and derive a scoring function, and then you keep the examples in R based on their score. So typically there are two, uh, types of classifiers. One are generative models. Uh, so this is basically, um, you have your target data, you can just, um, you know, estimate a model of that data. And remember, this has to be cheap, so probably you're not training a big language model. Generally, um, a K-LM basically says, I'm going to train a 5-gram, um, model. A more common thing, um, which I think we're mostly seeing these days, is just train a classifier. And the classifier says, I'm going to predict a positive label for examples that are in T, my target, and negative labels for things that are in R but not in T. So you basically, it looks like T are your positive examples. Um, some random subset of R, your negative examples. You maybe balance it, and then you train a classifier. And the tool that people generally use is fastText because it's fast, um, and it's generally, it's just a linear classifier, bag of words. And then once you have this model, now for every new document, you can score it and you set some appropriate threshold depending on your quality bar, and you keep the examples, sometimes stochastically, sometimes not.
Okay, so, so this is a very much a model-based filtering approach. Remember from the last lecture, there are many datasets, um, from, you know, generally a few years ago that did not use model-based filtering because they wanted to not bias things too much. These days, I think, basically everyone does some amount of model-based filtering because, um, unless you are compute plentiful, which in which case you probably don't need to do as much filtering, you can just train on everything. Most people are compute, uh, poor. Um, you have to be very smart about how you're filtering. Otherwise, you're just wasting FLOPS on kind of low-quality content.
Okay, so let me talk through some instantiations of of filtering. So remember I mentioned language identification. Um, so the goal is just to find, given a piece of text, detect whether it's of a particular language. Um, so Meta has trained this set of fastText language identification models, which often you can just use off-the-shelf. So it supports 176 different languages. It's been trained on, uh, a bunch of multilingual sites. So, Wikipedia, remember, has a lot of languages. There's also translation sites, um, and sites for different languages. So, um, a language ID is, is generally a fairly easy problem compared to many other tasks that we're dealing with. So, a simple classifier, if you look at a few words, you can tell that it's Spanish or or or Japanese. Now, this does, you know, there are subtleties because of code-switching and text, and there's some dialects. So it's, I wouldn't say it's like an absolutely solved problem, but, you know, this is not really the bottleneck for training a good language model. Um, okay, and then you can just, so once you have your classifier, you just choose some threshold. This is generally fairly heuristic.
So, another thing you can do with filtering is that suppose you're looking for data of a certain type. So let's say you want to get really good at math. So you want to go and find a bunch of math data. So, OpenMathText is this, um, paper from 2023 where the goal is to create a large corpus of math text. Um, this pipeline consists of a few steps. Um, so it's not training a single classifier. So first, you use some rules to filter: does it contain LaTeX commands? Um, you, they also use KenLM. So this is a generative approach, um, and trained it on the ProofPile, which is a known dataset of math, and keep it if the perplexity is, you know, below some threshold. Um, and then you also train a fastText classifier to predict whether it's mathematical writing, um, or not. And there's, you know, two different thresholds. If it has LaTeX in it, then it's a higher bar. If it's, sorry, if it's a lower bar, but if it's, uh, doesn't have LaTeX in it, it's a higher bar. Um, and then as a result, they got, you know, 15 billion, um, tokens which they use to train models. And as a paper, they show that this more targeted, curated data collection results in models that are better at math than models that were trained on 20 times much data which were not filtered in this way.
Okay, so the, the quality filtering, again, think about it as a tool. Um, you can define quality however you want. There's no universal notion of quality. If you want to define quality to be math, then you can go and get math, and you get better at it. Um, so GPT-3, I think I mentioned this last time, but just to put it in this framework. So positive examples are Wikipedia, um, WebText. So these are pages that, um, link out from high, you know, star Reddit posts and some books. And negatives are generally sampled from the, the web. They train a linear classifier and keep documents if the linear classifier is scored, um, highly enough. There's the first LLaMA paper, the positives were pages referenced by Wikipedia, not Wikipedia articles themselves, and the, it's just kind of the same idea. So, uh, 51 from Microsoft, um, is interestingly had, um, you know, also falls into this framework. So, um, they, um, started with the raw data, which is not already a Python subset of the Stack, remember, which is this, uh, all, you know, code and, um, they, uh, their, um, prompt, they define a prompt which is "determine educational value," and then they use GPT-4 to classify a subset of R, a 100k subset of R, with this prompt. And see if it's, and the ones that are positive are kept and called the target. Right? So the target here is actually the output of an expensive, um, you know, classifier. And then you train a cheaper classifier, in their case, a random forest, but you could have probably used fastText as well. Um, and, uh, and then you select data that is, um, classified positive by this, this classifier. And they also show that using this, uh, dataset, you do, um, much better than if you're just using, um, the raw data. So this performance goes up in fewer steps to a higher value.
Okay, so toxicity, kind of works the same way. There's this dataset, Jigsaw Toxic Comments, um, where it came from this project where the goal is to help people have better discussions online. So here the data is the Wikipedia talk pages, which can, I guess for controversial sites, get quite, uh, heated. And this has been annotated with whether there are any any toxic comments on there. And you can similarly define positive and negative examples, and you can train a classifier on that.
Okay, so, so now I think you have the tools or some examples and inspiration. You can identify, um, a particular type of data that you want. You build a classifier for it, and then you can go filter Common Crawl for that. I want to talk about one, uh, subtlety here, um, which is that, you know, the notion of what you want for data is actually depends on what model you want to train. So in particular, it depends on the number of tokens you're training on. And so there's no optimal threshold. So before, when I looked at the, the classifier gives you a score, um, there's you can't say 0.9 and say that's the best, because it depends on what you want to do. Intuitively, if you are going to train for a longer period of time, then you can tolerate lower quality data. If you're training for shorter, then, uh, you, you want higher quality data in general.
Okay. And of course, you know, if, if you could wave a magic wand, if you're training for longer, you want more high-quality data, but that's not an option that you're given. The data pool is what it is. So here's an, a kind of a, a preliminary experiment, um, uh, that Michael Ryan did. And, uh, so basically this plot shows for a 157 million parameter model, um, we're taking a very small, uh, pool, so, 100, um, you know, words, which is a tiny fraction of Common Crawl. And, um, we're look training and training, um, for more, uh, over time. So let's take a look at the blue curve. So the blue curve is DCLM. And so the, the loss starts here, and then it comes down. And each of these lines is when you, uh, epoch over the data. So this is one epoch, and then the next blue is the second epoch, and so on and so forth. So there's not that much data. So eventually you have to repeat your data. And, um, you know, the loss continues going down, um, because the second time you look at the data, you're still learning things. And at some point, you start to to overfit.
Okay. Um, whereas if you look at resilient parse, this is basically no filtering. Um, so here it's, it's much worse in the beginning. Um, but as you continue and continue, um, at some point, you also epoch, it starts to go down, you know, slower. So you kind of see this trend where high-quality data is better in this regime where you're not epoching. But once you get to lots and lots of tokens, um, high-quality data is no longer that, uh, great. And of course, you with high-quality data, you, you wouldn't even get, you wouldn't want to go into this regime because you're overfitting. You'll probably stop here. But even this, at this point, this is worse than if you had trained for longer using low-quality data.
>> Yeah.
>> Just a question about like the, um, computing the metrics. This is actually not like totally about graph, but like when you're like each of those dots correspond with one training, one training run, right?
>> Yes.
>> Um, do you ever need to do the confidence interval when you're doing these kind of pre-training experiments, or like, is it kind of enough just to do it one time?
>> So the question is, each of these points is a single training run, and, uh, should you do it multiple times and get confidence intervals? Um, ideally, that would be good practice. Um, often you'll see in these papers that, um, these are kind of scarce because each training run is fairly expensive. Um, so you, but, but in reality, when we have done these experiments, it's generally tends to be stable, I would say, at least for pre-training.
>> Yeah.
>> Yeah.
>> So you mentioned, like, trading for longer and high-quality data as that's not like a combination that we're considering, but like, like if, if we are trading with, for longer on this high-quality data, would it have diminishing returns compared to like training for longer with quality data?
>> Yeah. So the question is, if you were able to get higher-quality data and you train for longer, would it still have diminishing returns? Um, every dataset is going to have diminishing returns eventually. Uh, that's finite. Um, but it would just, it would probably be here. It would just like be down here and just just keep on going down.
Okay, great. Let's, let's move on. So, summary, filtering is, is pretty critical for building a good model, especially when you're compute-limited, like most of us. Um, technically, if you have infinite compute, you don't need to filter and you can train on everything and it'll be a giant model. But, um, realistically, everyone has to filter. Um, so, and the recipe here is: figure out what good data looks like, and then you can train a classifier, and that will extrapolate to the rest of your model. And what the good data looks like, you can either find it by saying, "Aha, there's this dataset out there that I really like, and I just want more of it." Or you can craft a prompt, um, to a language model and then use that to construct, um, to do a sort of a preliminary filter of a large pool, and then use that, uh, good-quality data to train a smaller classifier and extrapolate to everyone else.
Okay, next I'm going to talk about deduplication. So at this point, we have, uh, filtered our dataset, so we only have what we deem to be high-quality data. Um, but, you know, often data still has duplicates in it. And, um, there's, you know, two types of duplicates. There's exact duplicates. So this happens when, um, for example, if you look at these, um, mirror sites, the whole point of a mirror is that it's a duplicate. And, you know, sometimes the web crawler isn't smart enough to know that this mirror is exactly this mirror, so it just crawls, um, many, uh, sites, and you'll get this exact same content. Um, also when you fork a repo, um, that is also, um, a duplicate. Um, even if you make changes, probably you're making changes to a few files, so 99% of that repo might be, um, the same. So duplicates are, is, um, abundant, um, and, yeah, sometimes there's actually near duplicates, which are not mirrors or derivatives, but they just happen to be, uh, same text, differing by a few tokens. Usually, probably this came from a kind of a, it could be copying or came from a different, uh, common source.
So here are some examples of near, you know, duplicates. So terms of service and licenses. Um, so I guess the MIT license probably shows up, um, you know, on a lot of places. Um, and so in many cases, it is an exact, um, copy, but only of that license, unless someone made a typo when they copied it. But the rest of the page, um, that the license might be part of, is might be different. And we talked about how a lot of websites have the same, um, headers and footers. Those are also, you know, duplicates. Um, there's also cases where for whatever reason, um, you know, there's, uh, this is from LM1B, so the one billion or benchmark, there's like these articles where you just have, you know, typographic, uh, differences, like, uh, there's one version with a comma and one version without a comma. I don't really know why, but that's happens. And sometimes you see these templates where there, this is like some, you know, probably low-quality content where it's like a, it's a essentially looks like an ad of some sort, and someone just, you know, templatized, replace Canada with USA. Um, so if you train like on this data, but you are just training on different variations of this with different entities, this is going to be wasting your GPUs. There's more extreme cases. So this is why it's good to look at your data. So this audit of the C4 dataset found this product description 61,000 times in the dataset. And if you trace back, this is really bizarre. I don't know why this is like some, um, description of this gas mask. And this just showed up in this dataset, um, in Common Crawl 61,000 times. So, um, yeah, the web is weird.
Um, why deduplicate? So the first clear, uh, idea is clear because you want to train more efficiently. Duplication reduces your dataset size without really losing information because you're just removing duplicates. And it also has this benefit of avoiding, um, memorization, which this, uh, paper, you know, talks about. For example, if you have some, um, copyrighted content that's duplicated a lot, then if you train on it, then you memorize it. And also privacy concerns, uh, but mostly I think, uh, it's, uh, just to, uh, make sure that you're not, uh, wasting FLOPS related to deduplication. That's also decontamination, which is arguably even more important to, and it's the same sort of, uh, deal is that you want to make sure that your, uh, test set is not in your training set.
Okay. So, so how do we dedup? So here's the design space to think about. So first of all, what are the items that you're duping? Do you do it at the sentence level, the paragraph level, or the document level? How do you determine a match? Is it an exact match? Is it the existence of a common sub-item? Is it the fraction of common sub-items, uh, for near deduplication? And then once you've found a duplicate between two pages, what do you do? Do you remove all the instances, or do you remove all but one?
Okay, so, um, and the key algorithmic challenge is that deduplication is fundamentally about comparing items to other items. Um, and normally, if you're doing filtering, this is about an individual item. This item, is it good or not? And this can be parallelized. It can be, you know, it's sort of linear time, which is good. Um, and even in linear time, we're trying to make it fast by having rule-based or very small models. But deduplication, clearly you can't do the N-squared thing where you compare everything to everything. You need linear time, uh, algorithms to scale, especially at this web scale. So, um, typically the deduplication literature, um, uses hash functions to get around this. Um, so we'll develop some of these, uh, ideas. These are quite nice ideas from, um, thanks to, um, you know, the algorithms, uh, community. So I think everyone knows what a hash function is. It takes, uh, some sort of value, like a string, and maps it into something like a string or integer. Um, and a hash value is much smaller than the item. Hash collisions are when two distinct items map to the same hash.
Okay. So, um, when you look at hash functions, there are really fancy hash functions which are cryptographic in nature. These are collision-resistant, used in crypto, um, cryptography, and, um, and Bitcoin and things like that. And then there's these, like, uh, faster ones which are used for hash tables, where hash collisions aren't the end of the world. And so we'll be using, um, these.
Okay. So take a string and map it to some value. That's what, um, hash functions do. Okay. So exact duplication is, is conceptually very simple. Um, you, you take a string, um, you see if there's an exact match, and you remove all but one. Okay. So here you have a bunch of, um, elements, and you, you hash them, and you dedup. Okay. So this is, um, and exact duplication is, is very nice. It's very clear what's happening, but this isn't really good enough for the messy web data because often you have these near duplicates. Um, and one note is that there's many ways to have, um, you know, written this. This is written by in a sort of like, um, a bit of a MapReduce, uh, style way, which makes it more easy to easily parallelizable and scale. Um, so C4, this, uh, this paper from the, the T5 paper, um, which processed Common Crawl, did exact duplication. The items they operate on were three-sentence spans. Um, and you, uh, do, um, exact match, and they remove all but one. So, one, if you're kind of paying attention, you realize that there's something kind of a bit strange about this because you're looking at three-sentence spans. And if you find two documents with three-sentence spans, you remove all but one. That means you're just going to rip out three sentences from that document, which is a little bit strange because it breaks the coherence. But, you know, this is what they did. Um, okay, so that's exact dup, deduplication.
So how do we do near, uh, deduplication? So first of all, we have to define what approximate, uh, match means here. Okay. So to do that, um, we're going to define this, uh, called Jaccard similarity, which is a fairly standard notion. And Jaccard of two sets is basically the size of the intersection over the size of a union. Okay. So given these two sets, 1, 2, 3, 4 and 1, 2, 3, 5. Uh, when you compute Jaccard, you take the intersection, that's 1, 2, 3. You take the union, that's 1, 2, 3, 4, 5. And you divide, and the Jaccard is 0.6. Okay. So the Jaccard is a number between 0 and 1. Zero means that they're disjoint. One means that they're identical. Okay. So a fairly natural notion. And, um, we're going to say that two documents are near duplicates if their Jaccard similarity is above some threshold, let's say 0.99. So now the question is, how do you find near duplicates in linear time? Okay. So fortunately, this is a solved, um, you know, algorithms question, and the answer is to use, um, MinHash. So, well, okay, so the first step is to use MinHash, that's not the final answer. So MinHash is a random hash function so that the probability of a hash collision is exactly, uh, the Jaccard of A and B. So this is a very, kind of nice property because, um, hash functions, hashing is good for making things linear time. Jaccard is the metric that we want, and we're sort of connecting these two, uh, right now, or in expectation. So what's interesting is that normally you want hash functions to define hash functions so that everything is, um, hashing to different, distinct elements are hashing to different things. You don't want collisions. But here you actually want collisions, uh, not arbitrary collisions, but you want to control the collisions in a certain way to align with a similarity. So similar things you want to collide more than the disparate things.
Okay. So, um, so the MinHash is essentially you, um, actually, let's see, maybe I will, okay, so here's the, the MinHash. It's fairly simple. You take the set and you hash every element in that set and you take the minimum element. Okay. So, um, this min, might, if you're seeing this for the first time, it may seem a little bit strange. Why are you taking the min? You can take a max too, doesn't really matter, it's just a way to break, um, ties. So here's the, the picture that, uh, you should have in your mind. So, okay, so if you have, um, A, remember A contains 1, 2, 3, 4, and B contains 1, 2, 3, 5. So this is a characteristic matrix representation of these two sets, right? And so what the random hash function is doing is that it induces a permutation over the items. Okay, so it might be 4, 3, 1, 5, 2, or something else. And then you look at which item is first according to this permutation in A and which item is first in B. So each item has the same probability of being first. Okay. So if you have, if it, the random hash function puts one first, then, uh, first in A will be equal to first in B. And if it's two first, then it's the same. If it's three first, it's also the same. But if it, the random permutation puts four first, then the first element in A is going to be different from the first element in B. And same with five. Okay. So look at this representation. Basically, the random hash function says, elect one of these rows to be, you know, first. And then when I, the min is just telling me the min, uh, the, the min equal of A equal the min of B is basically telling me whether that row has, um, is the same. Okay. So that is the, I guess the proof of, um, why MinHash, why this property holds, which is that you have a hash function in expectation over your random choice of hash functions that, um, the probability of collision is the similarity metric you want. Okay. So let's just check this in code. So, um, I'm going to generate 100 different hash, you know, functions, S, each hash function is given by a seed. And I just check whether the MinHash is equal to MinHash and the estimated Jaccard. So, um, I'm going to just look at the number of the fraction of matches, and you get 0.6. Okay. And the key thing with a MinHash is that, um, I don't have to do the, the N-squared thing. I can compute the MinHash of A and the MinHash of B and MinHash of a different set, and I just look for collisions.
Okay. Any questions so far about MinHash? So now we can hash our items, but we're not done yet because a collision doesn't tell us that Jaccard is above some threshold, which is what we want. We want to find A and B such that Jaccard is greater than 0.99. All we've done is said, okay, well, if we got a collision, the probability of that collision of two things colliding is the Jaccard. But that's not really that useful by itself. It's stochastic, and I can't really get anything reliable here. So, um, the next thing idea is this thing called Locality Sensitive Hashing, which essentially solves this, uh, problem. So this is a very classic idea in theoretical computer science, and it's quite nice. Um, so the idea here is that we have A and B colliding with probability equal to Jaccard. Okay. So it is true that more similar items will collide more often, but it's very stochastic, right? The variance is, you know, quite large here. So our goal is to have A and B collide if the Jaccard is greater than some threshold. So we, in some sense, have to sharpen these, these probabilities somehow. Like the probability can't just be literally equal to the Jaccard. So the solution is to use more hash functions, uh, and these hash functions are going to be independent. So, um, this part is, you know, gets a bit technical, but, um, we'll walk through it. So you break the N hash functions into U bands of R hash functions. So if you have 12 hash functions, you have three bands. Each band has R, uh, four hash functions. Okay. So there's a band here, H1 through H4. Here's a second band, H5 through H8. Third band, H9 through H12. Okay. And then, so what we're going to try to do, so, so each hash function gives us either, um, collision or not collision. Okay. Um, and so every hash function, there's a probability of, uh, colliding. So the key here is that we want to say, compute the probability that A and B collide. We say that A and B collide if for some band, all of its hash functions return the same value. Okay. So, um, if H5 and H6 and H7 and H8 all return, um, the kind of the same value. So, sorry, if HA of A equals H5 of, uh, sorry, H5 of A equals H5 of B, and H6 of A equals H, uh, six of B, and so on, that means this band is triggered, and then I would say they collide. Or if this band, if all of these say, um, coll, uh, agree, then I say it's a collision, or these two. But you don't have to have all the hash functions, um, you know, uh, return the same value. Okay. So there's sort of this AND or structure here that is, um, is is doing the lifting here, and we'll see why this, uh, this works.
So, um, so now let's say you have a particular Jaccard of A and B, what is the probability that A and B collide according to this, uh, definition? Okay. So we can calculate this. Um, so let's say that, uh, this Jaccard similarity is 0.8. Okay. And we have five, um, you know, bands, and each band has 10 hash functions. So then we can say, what is the probability of a fixed band matching? So that's 0.8 to the R, because each band has R hash functions. So the probability that they all have to match is, uh, 0.8 to the R. Okay. So the probability of a band of a fixed band matching is, is generally quite low. It's exponential in R. Okay. And then now the probability of collision is the probability that some band matches, and that's going to be the probability of one minus the probability of, so this is a probability of, um, a band not matching, and this, if you raise it to the B, there's B bands, this is a probability that all of them don't match, and then one minus that is the probability that some match. Okay. So the probability of collision is, you know, four. And you expect it to be higher because you're given, you know, each tries to get a match. Okay. So if you plot this, um, this is what it looks like. Okay. So you have, um, on the x-axis, the similarity here, um, and we were looking at 0.8, um, and and you look at, uh, the probability of a, a collision here. So we would expect that if it's similarly zero, then it should be zero. If it's one, it's one. But notice that this is kind of an interesting S-shape, which is, is nice because this is what we wanted to sharpen. We wanted to basically say, if the similarity is below some threshold, then we want the probability to match to be as low as possible. And if the similarity is above some threshold, we want it to be as high as possible. So we're trying to get this to be like a phase transition.
Okay. And if you, you know, plot this, this function as a function of similarity, that's, um, what you get. Okay. So, um, Oops. Okay. So, so let's just look at some examples. So, let's, uh, concretely, we have, um, similarity 0.7 to 0.98. Okay. Um, and we're going to look at how, uh, B equals 10, R equals 10. So we, um, look at the collision probability according to our definition, and we see that, um, this gives us a range of, um, you know, 0.25 to to one. Okay. So if you were to set a threshold, you know, this is, this is not, you know, bad. Uh, we can like set it here, and you know, these with, um, you know, some probability will, uh, you know, there's still false positives because even if these are below a threshold, let's say 0.9, these might still collide, but we can filter them out, if we, if we want. But, um, the ones above are mostly kept. So what happens when you move, uh, so increasing R, so remember R is the number of hash functions within a bucket. So when you increase R, um, the, the threshold sharpens and moves the curve to the right. Everything becomes harder to match, right? Because you have more hash functions inside a bucket, you have that it sharpens that exponent. So the probabilities you see here are kind of sharper here. And now everything moves to the to the right. Um, and now it's fairly unlikely that if something has low Jaccard, you're going to to match it. So before it was like 0.25, and now it's, you know, 0.08. Okay. So we're sharpening the probabilities. And then increasing B has the effect of moving the curve to the left. It makes it easier to match. B is the number of bands, and some band has to match. If you have more bands, then there's more chances of matching. So if you look at this, then, um, before and after. So now we're making, before this 0.9 was only 72%, and now it's 0.92. And of course, this, this also increases, but, you know, not by that much.
Okay. So, and you can drive these, uh, phase transitions to be as, you know, as sharp as you want by increasing B and R. But of course, if you can increase B and R too much, then it can be more expensive. But, okay, so let's look at a more real-world setting. So in this paper on deduplication, they had, um, you know, B equals 20 bands and R is 450. So to give you an idea of the magnitude of these, um, these numbers, um, and in general, this phase transition happens at the at a threshold which is one over B raised to the power of one over R. So if you want to, uh, find a, you know, if you want to filter based on Jaccard greater than 0.9, then you basically have to set B and R such that this is 0.9. And then by changing B and R, you can make that phase transition, uh, sharper. Okay. So remember, probability that a fixed band matches is one over B. If you have at this threshold, then the probability that A and B collide is, um, you know, approximately, it's a, it's a constant. So the probability of collision is 0.64. Okay. And this, this makes sense that basically, if you have a phase transition, the, sort of center of that phase transition is like point, you know, 0.64. And everything below it should go to zero as B and R increases, and, um, everything above it should go to one as B and R increases.
Okay. Any, uh, any questions about LSH? So this, this method is called MinHash LSH, because LSH really works for any hash functions. For, um, the duplication of language model processing, we're using the MinHash, which approximates your Jaccard. And so, um, that, uh, that basically is the one you should use here. Okay. Questions about, um, dedup? So one note about dedup is that, um, there's often, as we'll see, datasets that are coming in, and sometimes the, um, duplication will happen within a dataset, but you actually have to do duplication across your entire dataset because often datasets can be redundant. So, you know, sometimes that's not, uh, done, but it should be.
Okay, let's go on to the next topic, which is data mixing. Okay, so, so far we've transformed our data from really, you know, raw HTML or PDFs into text. We filtered for high quality. We deduped. So we have a smaller set of high-quality, um, documents. And, um, and this generally happens, um, for a given, you know, data source. But language models are trained on multiple data sources. So, in Marin, currently, this, this website tracks the different data sources that, um, the next model will be trained on. And you can see that there's a bunch of things. There's some, uh, neatron, there's fine PDFs, which we talked about, um, there's some, uh, you know, institutional books, um, and, um, some code, and all these things. Okay. So you have all these different, you know, sources, um, and the question is, how do you combine these? So if you look at an older paper, The Pile, this is the set of sources that they had at that time, and they essentially assign a particular weight to each component. So basically, you have a distribution of resources. Okay. And so where does that weight come from? So more, you know, just cut and dry, let's say you have three sources, you want to find a data mixture, and a data mixture is just a distribution over your sources. Okay. So if you're just thinking about this problem, um, from first principles, well, not from first principles, but like just from what would you, what you might do, vibes, which is I guess opposite of first principles, is you just manually set it based on some intuition, which is more often than you might think what people do, right? So I think this is, this is definitely fairly vibes-based. And even more recent papers, um, you know, you just look at it, maybe you use some method, and then you just like tweak things. Um, you can also, um, do uniform sampling, which is you just put a uniform distribution over your sources, um, and you sample every, um, you know, chunk, you, you get, you just sample a source. Um, you can also do, uh, proportional mixing, which is that you sample proportional to the number of tokens in a source. So if you have a dataset with more, you know, tokens, then you put higher weight. So this is also, you know, generally a rational thing to do, but you might worry that if you have a, a huge low-quality dataset, that's going to eat up a lot of your tokens. So this doesn't seem quite optimal either. So intuitively, you should upweight your higher-quality sources.
Okay. But there's two things that, um, are important to keep in mind. One is that you do want to ensure some diversity, right? So often sources are incomparable. For literature, code, and papers, um, you can't really say that, uh, you know, this paper is higher quality than this code because they're just incomparable, you know, objects, maybe. And if you want your language model to do well, you don't want to just put all your, um, uh, your mass on just papers. The second thing, which we'll talk a bit more about, is, uh, each source is actually finite. So if you put too much weight on a small source, then you need to, you essentially run out of that source, and you're, you need to just epoch over it. Meaning that you keep on training on the same, literally the same tokens, and this is going to be bad for reasons we'll, uh, see.
Okay, so let's, let's, uh, try to unpack this a little bit more concretely, this last point. So imagine you had just have two sources: a low-quality source, you have 10 trillion tokens, and you have a high-quality source that has 10 billion tokens. Generally, high-quality sources are smaller. Okay. So if you just do a naive data mixture, let's just do uniform. So half on high, half on low. And I'm going to train for one trillion tokens. Um, so when I say train for one trillion tokens, that doesn't mean unique tokens. I just means, uh, train for that many steps divided by batch size, essentially, or times batch size. Um, okay. So then you look at, let's look at this, which is the number of times a particular element in the low-quality dataset was trained on. So that's the number of epochs a particular data, uh, point was trained on. So P low times train tokens is the number.
Of, uh, times I'm want to request, uh, tokens from, um, this low-quality source and divided by the total number of low-quality sources. So, um, this is less than one. So, basically, that means, um, 5% of the tokens in this low-quality data set I'm even going to touch and train on once. Um, and then if you look at high quality, you'll see that, uh, the number of epochs is 50, right? Because I only have 10 billion tokens here. And if I'm half-half, that means 500 billion tokens. Um, I need to train up 500 billion tokens of high-quality data. But I don't have 500 billion tokens of data. So I have to repeat each data point 50 times.
Okay, so this is actually really important and, you know, some big model training runs have kind of messed this up. Is that you can't just like naively look at the data sets and define a distribution and then you sample because, um, if you just look at the quality, if you look at the data, you don't, you don't, you're sort of missing how many data points are there. Okay, does this make sense? Maybe pause for questions. This is a kind of important point. Okay.
So, yeah. >> Um, I guess like, does it like, why do we need in the first place? Like, doesn't it achieve comparable performance level, way, I don't know, fewer, you know? >> So the question is, why do you need 50 epochs? And I guess the point is that you don't need 50 epochs. Um, worst case, it's wasting compute. Sorry, best case, it's wasting compute, and worst case, you're overfitting. Um, but 50, the reason it's 50 is that if you just go in and you define this data mixture, then you're going to end up doing 50 epochs without realizing unless you're paying close attention. So this is basically the lesson is, like, look at how many epochs you're actually doing on your data.
>> Yeah. >> How does this, how do the mixtures get expressed during training? Like, so when you switch every step, do you train like steps of 10 saying, "Oh, I want to do 10 code, 10 to see because I feel like it was like, is there also any intuition between >> what decisions are made there? >> Yeah, good question. So the question is, like, how, when you train, how do the mixtures get, uh, kind of realized? And so, uh, in general, um, you, you sample, um, so you have, you want to fill a batch, right? So you can sample, uh, essentially for each element, um, which, uh, mixture component it comes from and you fill up a batch with, um, you know, sequences from that mixture component. Usually, you, every sequence comes from one mixture component. You're not sampling like per token.
>> I guess so, each batch should be mixed. So I guess the mixture assumptions are at the batch level and then you just train for that. It's not like, oh, there's one step is one step is >> Yeah, in general, you want to, to reduce variance, you want to train from multiple, um, you know, a batch should have some mixture in it. Yeah.
Okay. So, so this idea, I mean, this problem has been noticed, uh, you know, quite a long time ago. Um, and so in this paper, they introduced a unimax. And I, in there, that that case, it was, uh, they were training in multilingual models. It was like very clear that some languages, uh, were very low resource. So, um, you had to do something about this. So in this paper, they noticed that before, um, some works would, uh, essentially, um, take the proportional mixing and raise it to some power, um, to kind of flatten out the distribution. Um, but the idea in this paper is that, um, let's be a bit more kind of explicit about that. Let's sample the sources uniformly, but with a hard cap on the number of epochs.
Okay, so this is the key idea here is that you say, "I'm only going to take 20 epochs over a particular, you know, data source." So if you've done that, then, you know, too bad. You go, you, you don't get any more, um, tokens. You move on to the next thing. So this is sort of like a, a safety net in, in some sense. So in particular, if you look at the number, the probability assigned to a source times the number of training tokens, that should be less than or equal to the cap. Okay. And there's a simple procedure for actually just determining the mixture, uh, subject to this, uh, constraint.
Um, okay, so let's switch gears a little bit and talk about how we define the mixture in the first place. Right? Because if you have 50 sources, there's 50 numbers you have to fill. And we noticed that proportional mixing isn't really enough. Maybe you can try to get estimate some quality metric and then you can sample proportional to that. Um, but that is also heuristic. So the, the most principled, uh, methods and most kind of easiest to understand, like, what's happening is these regression-based mixing methods. So RagMix, um, Mix, and other papers like this. And the idea here is, is kind of relatively simple. So first, let's say you're trying to train a large-scale model. So you go to a small-scale, um, let's say one, you know, I guess let's say 300 million parameters, um, and you, you try different data mixtures. So let's say you have three, uh, components. You try different mixtures and then you train these small models. So you train a swarm of small models. Each model gives you a loss on some target metric. It could be some downstream evals, it could be perplexity, whatever you want. Okay. So you use these data points to basically fit a regression where the regression model maps these, um, inputs, which is the, the data mixture weights, to the loss. Okay. So now you have basically a very cheap model that tells you, "If I were to train on this mixture, what loss would I get?" Then you can essentially optimize that, um, you know, um, data mixture, uh, sorry, optimize that function for the optimal data mixture, and then that is the data mixture you use to train the large-scale model. Okay, so that's the kind of the simple procedure. And you'll notice some kind of similarities with with scaling laws where you're trying to do some cheap computations to figure out some answers and then, you know, scale up.
Okay, so there's a few design decisions here. One is, you know, what are the mixtures that you're trying out here? You need a distribution over distributions here. Uh, so often people use some Dirichlet distribution. Um, you also have to define the regression method. Um, people have tried linear models or boosted decision trees. Um, you have to figure out the target here. Um, often this is based on downstream eval. So, we have to be very careful not to overfit, right? Because we're doing pre-training and then, you know, supposedly pre-training is supposed to be training this general-purpose model, and we're not trying to fit some downstream evals. And if you're not careful, for example, if you have a bunch of code evals, then, well, guess what? You're going to upweight all the code data. That's, you know, not rocket, uh, science. And if you then go and you say, "I want to generate some poetry," you might, uh, you know, realize that you've overfit. So you have to be very careful about this. So proportional mixing and uniform mixing don't have this problem because you're not looking, there's no downstream evals. And the, the final thing is that, you know, how much of a difference between small and large scale, um, is there? And this is a cost and accuracy trade-off. Of course, if you train really, really small models, um, this might not be representative. But if you train large models to do this, you, well, there's no point in doing any of this. You just train, you're basically doing hyperparameter tuning at the largest scale, which is too expensive.
Um, so this Omix paper actually does a nice, um, has this nice table which shows you for a number of different, um, methods that all fall into this kind of, um, framework, um, this, the size of the proxy models, they're generally fairly, you know, tens or, uh, uh, tens of millions of parameters. How many, you know, points do you sample, mixtures do you sample, um, M here being the number of parameters in your mixture, sorry, the number of domains rather, and you can use Dirichlet, you can use exponential, um, and then the regression model, uh, you can fit a log-linear model, which, um, you know, tends to work, uh, pretty well, um, and then, uh, you can use different ways to solve this optimization, um, you know, problem.
Okay, so there's sort of two leaps of faith here that you just have to be very careful of is that, um, the regression model was trained on a bunch of these small proxy runs and you're optimizing this. So the hope is that your, the minimizer, um, your optimal data mixture, that regression model is still accurate. Um, and this is a little bit kind of, you know, you have to be very careful because, um, if you, let's say, sample a random mixture, you'll probably be fine because, um, because of, you know, classic generalization, you sample a mixture, you, and you fit a function, it should be able to predict in distribution. But when you're optimizing, you're essentially trying to go to potentially the extremes where you might not have as much coverage. So that's one thing to be careful of. And the other thing is that, um, optimal, uh, data mixtures, you just hope that they transfer for small scale to large scale. Um, and this, in general, it seems like, at least at the scales that the open community works with, um, tends to be true or not blatantly false. Um, but clearly, there are, uh, scale-dependent effects. If you just think about the, the previous slide when we were talking about filtering, if you're training for many more tokens, then probably low-quality data is, is okay. So clearly the optimum is not the same, but you just kind of hope for the best here.
Okay, so hold on. There's one scale-dependent effect that we, we already talked about but, and we need to address here, which is that, um, you imagine, so we have a lot of low-quality data, 10 trillion tokens, 10 billion tokens of high-quality data, and if we train a small model on low token counts, you know, remember, uh, what what happens is that you might, this is a made-up, um, you know, example, but, um, you might put a lot of mass on your high-quality data because if a low token count, you're not epoching. So it's like, "Oh, wow, Wikipedia is so great. Let's just train on Wikipedia." Um, but if you then go and train your large mixture on this large model on this mixture, um, then you're going to end up epoching a lot on this high-quality data and you'll overfit, and that's definitely, um, bad. So one way to fix this, which is done in the Omix, this Omix, uh, paper, um, is you can just, uh, cap the number of epochs. Solution. There's another solution here which, um, goes by probably multiple names, but one name for it is simulate epoching. And the, the, the principle here is that make your small scale look like your large scale. This is kind of a general theme for the course. We saw it in, kind of, when Tatsu talked about MUP. Um, you parameterize your model so that your, your hyperparameters transfer. And so the kind of same principle applies here. And the problem with, you know, doing this kind of naive scaling is that, um, at low, you can't be not epoching at small scale and epoch then epoching at large scales because then those are qualitatively different, um, operations on your data set. So the way you would handle this is you, um, you downsample your sources proportionally. So, um, so if you have your small runs were 10 billion tokens, um, and your big run were one trillion tokens, then you have, um, a, you know, one in 100 kind of downsampling, um, and, and then, uh, you basically use the downsampled mixture. And so the idea is that if you have the downsampled, you know, data, um, then this solution is not going to look good because you're not training on all, you're not, you don't get to just train on all of Wikipedia, you're going to train on some minuscule fraction of Wikipedia, and you'll realize, "Oh, actually, I'm going to be epoching a lot," and that's going to have really bad, uh, loss, which means that in the optimization, you're going to look for more balanced, uh, uh, approaches. Is so you're kind of simulating the data scarcity that you would get at in your high regime at a low scale.
Okay. So to summarize this section, um, the problem is, how do you weigh different sources? You have Wikipedia, you have your Common Crawl, um, you have your, your code, you have some math data that you, you scrape from somewhere. Um, how do you balance them? Um, regression-based mixing is, is a nice framework, I think, to think about things. You, you estimate your, a function form, uh, that maps your mixture weights to a loss at small scale, optimize, and then you generalize to large scale. And you just have to be very careful with, uh, this, uh, because of epoching and overfitting kind of issues. And you have to either use capped epoching or simulate epoching. So one lesson here is that, um, if you're trying to optimize anything, you have to be very careful because you are in danger of optimizing the wrong, uh, thing.
Okay. Any questions about data mixing before I move on? um, when you're a dev sample, assume something is like too small that like you can't really generalize as well. >> So, uh, the question is, if you're downsampling, is there's a risk maybe you get too small of a data set? Um, yeah, absolutely. So, um, I think, right, you might get into a case where you just have so few tokens. Um, I think then what will happen is that your optimum will just put very small mass on that, and when you scale up, maybe you get the right set of tokens. So in principle, like, it should work out, but there might be like rounding errors where like you end up instead of training once on this data set, you train like zero times by accident. But you can, you can sort of work around these by just like, um, think you always train once on this data. Yeah.
Okay. So, yeah. >> Within data mixing, um, do you ever, because like, I guess these sources just happen to align with different topics, but do you ever take like a diverse data set and then try to do like data mixing within that? >> Yeah. >> So, the question is, uh, do you do what do you apply data mixing to the like different, you know, domains or even within a data set? That's actually, I forgot to mention that. So I'm glad you brought it up. So in the Neotron paper, they, uh, actually, in, in some of the OMO stuff as well, you basically think about, let's say I give you a just Common Crawl, right? You can actually break this up, so you can group by domain. So they, the AI2 folks have this like web organizer, so you can group into topics, but you can also quality filter. So then you, you have this like two-dimensional grid where you have domains and you have quality, and each of these like cells is something that you would data mix. So that's one automatic way to determine the, the domains, and then on top of that, you add extra sources that people hand you.
Okay. So I'm going to quickly go through post-training data here. Um, so up until now, this is basically pre-training or maybe mid-training data. It's generally fairly task-agnostic, except for the RegMix stuff where you're, you're trying to minimize the loss, but still the data itself is still task, uh, relatively task-agnostic, and you're trying to develop, uh, basic skills. So when you look at post-training, a lot of the data becomes very kind of task, uh, you know, dependent, um, and I'm not going to do a comprehensive review. I just want to point out some of interesting, um, post-training, you know, data sets that have been recently released, in particular for coding, since that's, uh, of I think great interest these days. Um, so the general recipe is, um, you define a set of, you know, environments, um, in the case, think about code, which might be GitHub, um, you know, repos. You define a set of, uh, tasks or or prompts, and then you collect responses from a strong model or teacher. So implicitly, you know, at least in the open community, almost all the post-training data that's, most of it is, you know, synthetically, you know, generated. Um, you can also replace a strong model with, uh, a human, but, you know, that is slow and costs a lot of money, but, you know, if you're at the frontier, and of, um, if you, I guess a few years ago, you kind of had to go and pay a lot of people a lot of money to give you responses. Um, and nowadays, even at the frontier, you can do sort of hybrid human-AI things. But anyway, the, the point is that there's some teacher out there that's giving you responses.
Okay. So, um, a few kind of, I'm just going to talk to a few works here. So, one is this work, uh, called Open Thoughts. Um, and this idea here is, um, this came around, it was motivated by, you know, when GPT-4 came out and there was a lot of intention on reasoning, mostly for like math and science, how do we get really good post-training data sets for this? And, um, you know, they eventually came out with 1.2 million examples, using this teacher model. Um, so the steps are, first of all, um, there are a bunch of, so what are the environments and the tasks and questions? There are a lot of different sources, um, that they, they drew from. There are human sources like Stack Exchange, um, or Num-Math, and then there's like some synthetic sources as well. These are just a subset of the, the coding ones, but there's also more for math and chemistry. This is a big project that involved a lot of different, uh, contributions from many, uh, people. Um, so you can see that there's, um, you know, some of these are like coding exercises, some of these are, um, you know, more realistic things that show up on, um, I guess Code Golf is not realistic, but there are some more realistic, um, examples here. Um, so code review, this is probably more realistic. Um, and they noticed that they did a fairly comprehensive, uh, analysis of, you know, given this data set, what, how do you generate responses? They show that, um, having a few sources was actually good, but rather than trying to do all sources, sampling multiple generations is helpful, like 16. One thing that was interesting is that having better models, aren't necessarily, um, you know, better teachers. So, for example, QWEN-32B, which is, you know, now a very old and small model, is was a better teacher than DeepSeek-R1, which was at the time probably one of the strongest open models. They also found that basic answer filtering wasn't helpful. Um, and so the entire pipeline looks something, um, you know, like this, where you have a bunch of these sources, um, you know, going in, um, and then you, you duplicate, you randomly, um, you know, sample, I guess, this downsample the questions, and you generate multiple answers, and that gets, uh, into the final data set. So the 1.2 million isn't, um, is examples, but divided by 16 gives you the number of actual questions.
Okay. So there's an another, so that was for, um, you know, math and science and, and some code. Um, I think more recently, there's been a lot of interest in developing, in particular, a generative coding, um, model. So not just a model that can just generate some code, but actually a model that can do software development. Um, so this Swiss-Smith paper, um, um, had the idea of, you're given a repository, you're using a language model to automatically generate tasks. So they have an agent that, um, takes a repository and actually makes it usable for, um, uh, like, uh, installing dependencies and so on. Um, and then you generate tasks, which are generally, um, you know, you modify the code in some way, maybe introduce some bugs, um, and those get verified, and you get some task instances. So these are synthetic tasks, but you get, you know, 50k of them, which is actually, um, at the time, which was last year, quite large. Um, so since then, there's been, um, a bunch of other, uh, work which I'll, uh, you know, mention. Um, so there's this Sweet-Zero paper from, you know, Nvidia, um, which had this kind of, um, actually, okay, I'll also talk about that. So this is a kind of an interesting idea where the observation was that unlike math, code tasks have just a lot of heavy dependencies. Most GitHub repos, like, don't even run, and you have to install all these dependencies and out of date, especially if you roll back to like when a PR was. It's just kind of a mess. Um, and this is just a nightmare. Um, and so they were thinking about how can we actually get a data set that just, um, on all the, the, the repos, and rather than having a repo-specific Docker image, um, they noticed that the models are actually good enough that they can solve a lot of these tasks without execution feedback. So, you know, if you look at some of these, uh, models, if you were allowing execution, you get like 80. If you don't allow execution, you get almost 70. So which is, you know, not bad for not being able to execute code. So, um, somehow these models have some internal semantics of, of, of code. Um, so they were able to generate 300,000 agent trajectories. All of these are real GitHub, um, you know, PRs, so they're realistic, unlike the, the Swiss-Smith examples. They use the OpenHand scaffold and did some, there's a lot of details like how to prevent, um, agent hacking. So, um, normally you, uh, hand an agent this, um, instruction, you explore, you test, you implement, and their, uh, 3.0 version was basically saying, you cannot run Python code, you can only do like `set` and `grep` and all these like basic, um, um, operations.
Okay. So then they distilled from a big coding model, um, and filtered, um, because sometimes a coding model will ignore these instructions and still try to execute anyway. Um, and then there, they showed that with this, um, oh, they also had 13k agent trajectories that do require execution feedback. So they train models where they first fine-tune on these Sweet-Zero examples, and then fine-tune again on these Sweet-Hero examples, and they were able to, you know, I guess the frontier is still pretty, uh, high up, but they were able to make some, you know, progress here. Um, I'll quickly go through ReBench was essentially another, uh, attempt to grab tons and tons of, you know, uh, PRs. Um, so, and all of these kind of look like, get a bunch of GitHub repos, try to install the repo, like most of them probably, you know, fail, and try harder, and then you use a, um, a language model to give you the, um, the responses. So in this, this actually just came out today. Um, so you can take the Sweet-Zero idea and you can now scale up to 12 million agent trajectories. The nice benefit of Sweet-Zero is that it's just, you know, almost, it's very lightweight. Um, here they use the SweetBench ReBench tasks. So in this one, remember they were trying to get things to execute, they only got 32,000 of them to execute, and 120 of them didn't execute. So you just, but Zero doesn't care, you can use all of them. Um, this is a very small model because now this data set is quite large. Um, and so, um, so, you know, I think as you can see, the data sets are, you know, getting more and more sophisticated. You go from, you know, environment-free things like math to now coding, and now the coding data sets are growing, quite a bit. Um, but the general idea is that, you know, you have these, these prompts, which, you know, there's kind of trade-offs. You can have fully synthetic, you can have semi-synthetic where you have a real environment and synthetic tasks, or you can have real. And the responses generally are coming from just capable models, but they need also need to be good teachers. Code environments are a pain, and you, there's a lot of filtering and other details which we don't have time for.
Okay, to summarize this lecture, we talked about filtering. So define what good looks like and then train a lightweight classifier. Go over your web crawl and you can get a a small subset that matches what you're looking for. Deduplication is important to avoid overfitting and saving FLOPs. Uh, mixing, try mixture at small scales, extrapolate to large scale, and then we looked at some, you know, post-training data. Um, I will say that though that a lot of the data work is can be very grungy, it's very domain-specific, and requires looking at concrete examples to make these high-quality data sets. So this lecture is not really representative what data work is like, but hopefully I've given you an idea of the data landscape out there. Okay, that will be it for today.