Transcription
GUY VAN DEN BROECK: It's mostly about convincing machine learners that all the stuff that this community and my community has been doing for a long time is still very valuable in the modern age of large language models. And so the first thing I have to remind people of, not this audience, but I guess the general audience is that reasoning used to mean something different. So reasoning used to mean deductive reasoning algorithms in logical probabilistic reasoning. And I'll be talking about both. And they are nice because they're giving you the right answer on all problems, if you're able to ask your question. Because obviously the scope of reasoning algorithm is limited to whatever the query language is. And then another downside of this old-fashioned way of thinking about reasoning is that it's very often intractable, undecidable, or high complexity.
And so whenever you talk to the kids, if you were just at ICLR, then 30 years later when people say reasoning, they mean something very different. When they say reasoning, they mean inductive reasoning from data, meaning I don't know how to do reasoning, but I see lots of examples and I try to inductively generalize from them and build things like chains of thought that do the reasoning. And they're correct on many inputs, not on all of them. You never really know. And what's appealing, though, is that they have unlimited scope. You can ask anything and hope to get an answer. And you will actually get an answer. It just might be a wrong one. And so the goal of this talk is to see how can these worlds meet. Can we still use some of these beautiful things we like in this age of transformers? OK, so do these algorithms have a purpose today? What went wrong with the algorithms we used to have to do reasoning? How do we make them more modern and appealing to the deep learning crowd and the large language model crowd? And then for the specific topic of this workshop, how can these algorithms help with alignment, safety of foundation models?
OK, first question-- how are these tools useful? And I would like to motivate this using an example. I asked ChatGPT, generate a sentence using the words "frisbee," "caught," and "dog" in this given order. And it does an OK job, but gets it wrong. It gets the words in, but not in the right order. And so the way that we typically interact with these models is we say, oh, you made a mistake here, please correct it. You kind of beg until they do the right thing. So you're saying let's do the right thing. It's again wrong. And so what I'll present this one system called Ctrl-G that is guaranteed to always follow these instructions and generate things that satisfy the constraints that you give it. And so it generates the obvious answer of "A frisbee is caught by a dog." But then somewhere in this generation, I also found "A pair of frisbee players are caught in a dog fight," which I thought was a very nice answer that I would not have come up with myself.
All right, so how does this work? So this Ctrl-G architecture, it's really a large language model with reasoning built on top. And so just a dummy version of the task I just showed you is, we have a prompt like "the weather is," and you would like to continue generating words from this prompt. However, now we have a lexical constraint we'll call alpha. And it might be something like generate the word "winter." Now our large language model only has a next-token distribution. It only knows what the next word is, given the prefix. And so that could be cold or warm. It doesn't really know that if you're going to say winter, you're probably going to talk about cold rather than warm.
Now, in principle, if we didn't care about the complexity of the computation here, what I would like to do is use Bayes' rule. I would like to say, given alpha, my constraint, there is a posterior distribution on my next tokens that is conditioned on not just the prefix, but also what comes later. And by Bayes' rule, this is the product of two distributions and a normalizing constant that doesn't matter. It is the next-token distribution given by our large language model, this one here. Great, we have that one available. We have a great next-token distribution. And then we need to compute the probability for any choice of next token of satisfying the constraints. What's the probability, if I were to sample from my large language model, that I will say "winter," that I will say "Simons Institute?" And sadly, this is completely intractable. It's easy to prove to be sharp-P hard. No one knows how to take a modern transformer and compute the probability that it will eventually say something even within some window size.
But suppose that we had a different type of language model that had different properties, somehow, that was tractable for reasoning. And so then we could ask this other type of language model, what is the probability of saying "winter," given that I say the weather is cold? Or what's the probability of saying "winter," given that the weather is warm? And obviously when it's cold, it's much more likely. And so if I have such a tractable probabilistic model available, then I can simply do a little bait and switch and approximate the language model's probability of success by this other model that is able to do reasoning. I won't tell you yet how to get this tractable probabilistic model, but I'll tell you soon. For now, let's just assume that we have it, OK? We have this ability to do symbolic deductive reasoning, probabilistic reasoning about all these future generations. Now you just multiply these probabilities and you figure out that "cold" is probably a better answer. And the nice thing here is that the pre-trained language model is really good at grammaticality. It knows all of the knowledge on the internet. It knows how to form English sentences. And this model, its only purpose is to determine whether you will achieve your goal or not. So there's a nice separation of concerns here.
OK, we won't really deal with these very simple keyword constraints alone. We will actually have a slightly more general language for the constraints alpha, which will be any deterministic finite automata. I have this slide to explain to machine learners what is a DFA, but I think in this audience I don't need to do that. And of course, it can represent many things. And the only thing I want to emphasize for this audience is that when we generate from a large language model, typically we have a fixed sequence length. And so everything you know about, oh, DFAs cannot represent x, y, z-- it doesn't really hold because in a finite sequence length they can represent anything. In fact, they just become binary decision diagrams. So this is quite flexible, in theory. In practice, you still wonder, can I get a small DFA? Can I get a small BDD for the problem I care about?
OK, so this will be our constraint. And we wanted to see if we can get this pipeline to do something useful and impress the machine learners. And we found this application in interactive text editing. And so the story is that someone's writing a story. And they're not very happy with what the large language model generated here in the middle. So they delete it and they say, redo this part here, but redo it in such a way that, of course, it has to fit with what comes earlier and what comes later, the prefix and the suffix. The suffix is already something we don't often use. And then I would like this blank here to actually mention "alien mothership," "far from over," just some phrases that I would like to use in my writing. And I also want the length of this text to be 25 to 30 words-- not tokens, but words. I want to control how much is inserted here. OK, so we built a small domain-specific language for this, just five lines of code. We can say, these are the things I want for my generation, this DFA. And now I'm going to use this Ctrl-G sampler, this generation algorithm to sample things from the model. And this is what it sampled. You can read this. It perfectly fits. It uses the keywords. It has the right length. It makes sense. OK?
AUDIENCE: Does the DFA give you a constraint-- Just a clarification, the DFA gives you a constraint on what you going to-- the string that you put there has to satisfy this?
GUY VAN DEN BROECK: Yes, we need whatever we generate to be accepted by the DFA.
AUDIENCE: OK.
GUY VAN DEN BROECK: So it's quite flexible. And generating something that is accepted by the DFA, that's the probability of alpha, the probability of our constraints that we're trying to achieve.
AUDIENCE: OK, thank you.
GUY VAN DEN BROECK: OK, so this is the setting. And we then did some human evaluation where we asked people for various language models. So these are the usual language models. This one is a specific creative writing one.
PRESENTER: There's another question.
GUY VAN DEN BROECK: Oh, sorry. Back to the DFA? We have DFA fans here.
AUDIENCE: So it looks very nicely sequenced, the words in the brown colored part of the text. Yeah. But I guess, if you just say that, I don't know what, DFA-all-of, "alien mothership," "far from over," it could mean some arbitrary combinations of words also, right?
GUY VAN DEN BROECK: Yeah, so I'm not randomly sampling accepting strings from the DFA. Because then I would get garbage. I would get "alien mothership," "far from over," [? "fubar," ?] "Berkeley," whatever. What I want is-- and this is crucial-- I want to combine the probability of the large language model's prediction of the next token, which will make sure that whatever is generated is realistic and the probability of achieving the goal. I want both. I want the [? product. ?] So if the LLM says this next token makes no sense, then you will never generate it. If the tractable model says this next token will not allow you to succeed and get an accepting string, it will not do so.
AUDIENCE: So this is an iterative process?
GUY VAN DEN BROECK: This is just for generating one next token. And you just keep going just as you would for a large language model. Yeah. I'm glad I could clarify that. That's an important point. So it's not a coincidence that this looks like natural language. It is actually coming from the LLM. All right, so we asked people, how good is this text comparing Ctrl-G, which I just showed you, with GPT-4, where we just ask GPT-4, we give it the prefix, the suffix. We give it the instructions, and say find something that fits. And it turns out that if we focus on keywords and length constraints, people actually like the Ctrl-G system a little bit better. If there's no constraints, then GPT-4 is obviously better. This is an interesting one. This is asking how often do we succeed at generating something that satisfies the constraint. And it turns out even with GPT-4, only 27% of the time is it generating something that has the right keywords and the right length. Yeah.
AUDIENCE: Thanks. Sorry, just a clarification. If there's no constraints, why is this worse? What do you do if there's no constraints? Don't you just get rid of the tractable language model?
GUY VAN DEN BROECK: Great question. GPT-4 is a model we don't have access to the next-token distribution, so we cannot actually use it for Ctrl-G. For Ctrl-G, the only thing we can use at that time was Llama 2 7B. So this is a worse language model because we need to know what the probabilities are from the next tokens. And OpenAI won't tell us. OK. Good point. The most important thing here is 100%. Why is it 100%? Because going back to our earlier slide, suppose that I'm about to generate the next token that makes it impossible to get to an accepting state. Then this will be 0 probability, which means that 0 times whatever the LLM thinks will be 0, we will never generate the next token that makes it impossible to get an accepting state. OK, so we have a hard guarantee that whatever the system generates will always be satisfying the constraint. Yeah? Good, I like the questions.
PRESENTER: [INAUDIBLE]
AUDIENCE: Have you ever run into deadlocks? You could run into this problem that you--
GUY VAN DEN BROECK: You might because of some numerical issues perhaps. But as long as there is a token that leads to an accepting state, you will choose that one. And the LLM doesn't actually have the ability to give zero probability to something. So therefore, you never really get stuck, yeah. This is really like verification people type questions. I like this. Yeah, verifying whether my claims are true. Yes?
AUDIENCE: So I think you didn't use the word constraint decoding. So I think this is awfully like SGLang and so on.
GUY VAN DEN BROECK: Yes, exactly. So what's the difference between SGLang and this, and constraint decoding? SGLang just says here, I'm not going to compute the probability of reaching my goal. I'm just going to say whether it's still possible or not. And this will be a 0-1 number.
AUDIENCE: It does do the max masking thing, right? So what it basically--
GUY VAN DEN BROECK: Yes, exactly.
AUDIENCE: --says that, so that is kind of approximating this a little bit.
GUY VAN DEN BROECK: Yeah. So SGLang and guidelines and all of these tools to constraint generation, they are purely approximating this number by either a 0 or a 1.
AUDIENCE: Did you actually compare it with--
GUY VAN DEN BROECK: Yes. Yeah, they're terrible. Because if I have a constraint that says, I want to generate "Simons Institute at Berkeley," then these language models will generate, "Oh, the weather in Australia is great. The kangaroos are jumping about Simons Institute in Berkeley." So they have no way of actually knowing how to lead up to the constraint satisfaction. Because they only know when they're about to fail that they must generate something. And so they are not Bayesian models. They're not actually goal-seeking. They're just stopping you from driving off the cliff at the very last moment. Yeah. OK. Great question. Yeah, more questions.
AUDIENCE: Yeah, while we're on this semantic point-- this pedantic point, so I had experience with these constrained decodings that they have this issue where language and tokenizations don't always match up. And I'm curious how this is resolved in this setting.
GUY VAN DEN BROECK: Yeah, behind the scenes here, where's my DFA? Whenever you get a DFA over strings, you have to turn it into a DFA over token sequences, which is a little bit of annoying work, but can be done. So you basically get the canonical tokenizations for things. And yeah, so there's annoying work to be done there, but it's interesting scientifically. Is there another question? So back to the results table because my punchline is still not here yet. So my punchline is if you look at how often does the system generate text that is good enough to give three stars or better and actually satisfies the constraint, 82% of the time, Ctrl-G succeeds; GPT-4, 14% of the time. And so the highlight is that we really wipe the floor with GPT-4, even though that's 100,000. I don't know how much bigger it is. Probably, it's just so many orders of magnitude more expensive and bigger that people just think, oh, you could never academically build something that beats GPT-4 at some task. But it's actually quite easy if you build in the reasoning algorithm in the right way. And so that's my belief. OK, good.
If you are one of the young kids who likes to do reasoning with chain of thought, there's actually a nice way to also combine Ctrl-G with that type of reasoning. So here's a typical benchmark for reasoning, grade school math. And there's a question. You don't have to read it. The vanilla LLM gives some answer, and it's the wrong answer. This is not correct. Ctrl-G gives you the right answer, 64. But I didn't tell you what is the DFA that I use to generate this answer. And I don't know if you see what could possibly be a useful DFA to control the generation of this chain of thought, to give you more accuracy. If you do this on the whole dataset, you get higher accuracy. Can you see what is the trick? I'll tell you. Just use all the numbers in the problem statement. You have to at least mention every number in your chain of thought. You can say number 5 is given, but it's irrelevant. You have to at least mention every number in your chain of thought. OK, it's a very simple DFA. It's kind of a custom DFA per question. But if you put this on the generation, you get the right answer. So this is another fun use case here. Yeah?
AUDIENCE: How do you get the DFA if it's a custom DFA per question?
GUY VAN DEN BROECK: Oh, it's pretty quick to just look at this, get all the numbers out.
AUDIENCE: So it's just a human?
GUY VAN DEN BROECK: And then say, use these keywords, which is either-- and you have all the inflections of these things, like 60% could be 0.6. It just gives you a list of, you have to say 0.6 or 60%, and you have to say 5, and you have to say 16 in some random order.
AUDIENCE: But it's always, you should use all the numbers in the question? You never have a DFA that is anything other than, you should use all the numbers in the question?
GUY VAN DEN BROECK: Yeah, that's all it is. But of course, there are different numbers. You don't want to have a DFA that somehow captures that concept. It's specific to the question. Yeah.
AUDIENCE: Have you experimented with a language model generated DFA based on a few examples? Basically, can you generalize it to any kind of domain?
GUY VAN DEN BROECK: I would love to get the LLM to spit out the DFA that we then enforce on itself or something like that. That would be fun to do. We've never done that. I feel like we need more GPUs than I have to do that. Yeah, that would be fun to do. Good idea. OK. I love the questions. Keep them coming. I'll just skip the next half--
AUDIENCE: So how correct is this? You've given us a correct answer, but percentage wise?
GUY VAN DEN BROECK: Oh, I don't have the numbers. I think it's just a couple percent on these benchmarks. But because this is based on Llama 2, I think the accuracy is something like 30% or something. So it becomes like 32%. So it's a small boost. It's kind of a proof of concept more. Yeah. OK. I'm making up these numbers by the way. So internets, do not hold me accountable for this. So this wraps up the constraint generation part. We have the constraint guaranteed to be satisfied, which is nice. And that's incumbent with these structure generation tools. Except we're Bayesian, so we can actually achieve the goal. And what's also really nice from an inductive versus deductive reasoning perspective is we never train on the reasoning task. The reasoning task is just given at test time. And so that means that we don't have to deal with the fact that we have to train our model on a huge dataset of reasoning problems and reasoning chains. And then the distribution over problems changes and everything breaks. It's quite robust. And training a model to do reasoning is much slower and much more brittle. And takeaways-- you can use a tractable model, which I will explain next, to do the control generation of these intractable models.
OK, so next part of the talk is how did we achieve this probability of alpha. How could we compute the probability of the DFA and actually control the generation? This was previously shoved under the rug. And what is different from that algorithm than what we did maybe 30 years ago? OK, so obviously the probabilistic reasoning task that I care about is marginal inference, which means I need to sum entries in my joint probability table to get the marginal probability, ignoring some variables I don't care about. This is exactly what we need to do in Ctrl-G. We need to look at all future texts, sum the probability of all future text that satisfies the constraint. OK, so that's a high-level task. It is probabilistic reasoning, but of course, if there's a constraint involved that is a DFA, it's also logical reasoning in a way. And here's a little bit of a landscape of machine learning models or generative models where this is what we do today-- diffusion models, transformers, GPTs, and so on. And they're extremely expressive. And if you open your machine learning textbook from 20 years ago, you will find things like Naive Bayes, tree-shaped graphical models, Gaussian mixture models, hidden Markov models, Bayesian networks. And we're not using these for language modeling because they're much, much less expressive than these models, and they're just harder to train. But there's something that we forgot about these old models, which is that they actually had the ability to do symbolic probabilistic reasoning. You could actually compute a marginal probability on any of these models. And you would get your answer in polynomial time. And so the game that we're trying to play here is take these really old fashioned ideas and upgrade them to the modern age. And then, spoiler, they will look like circuits that will somehow be simultaneously very expressive and also very tractable.
So how do we get circuits to represent distributions? So the key idea is that a distribution can be represented in many ways with a transformer and so on. But a joint distribution can also be represented simply as a polynomial. So this polynomial, if I feed in values for x1, x2, x3 coming from this table, I get exactly the probability of that world. OK? So a simple polynomial. Now this is not a practical machine learning model, obviously. Because I need as many monomials as I have possible worlds, exponential in the number of random variables. This will never work. The way to fix that is to make it a deep architecture, in machine learning lingo, or to make it a circuit. And so we use circuits to compactly represent exponentially larger polynomials that represent the distribution. Simple enough for, I guess, a theoretical audience. So this is really just your usual arithmetic circuit from complexity. And there's one more thing we need. We need these to be tractable. We need to be able to compute a marginal probability. And it turns out that as long as this circuit represents a multi-linear function-- meaning every variable here appears with a degree of 0 or 1, never 2, 3, 4-- then it is actually possible in linear time to compute a marginal probability. And you can do efficient probabilistic reasoning this way. So that's the high level story here. We are going to build circuits that represent the distribution. And they are tractable because they represent multi-linear polynomials. And now I'm going to hand wave a little bit more because this is maybe more in the machine learning side. But once you have such circuits, you can use them as neural networks to evaluate the likelihood, meaning the probability of a complete sentence. Or later I'll show you a marginal likelihood, which is the probability of a partial sentence, or even the probability of a constraint.
So here's how it works. I want to compute for a complete sentence. Now I'm using these pixel colors as data, the probability. And I can take all of the inputs to my circuits. And they represent a very simple distribution where it's either 0 or 1. And I can compute the likelihood of the input of the model given those inputs. I can then take sums of products. And in the end, at my output layer, I get the probability of this feature vector or this sentence or whatever your data is. So this is like a dummy machine learning view. It's just a neural network, a feedforward neural network. What I want to do is something more difficult. I want compute a marginal probability. And again I'm going to hand-wave a lot here. But now I only know that x is a gray pixel. And I don't know the value of z or y. So that means I have to integrate these other random variables out of my distribution to compute a marginal probability. I have to do this for the distribution represented by the whole circuit, which means the output of my network. And the output is a sum. And just by the linearity here, we can write this as the sum of integrals. So now what I got is I took my integral at the output layer and pushed it one layer backwards to the second to last layer. And now I have a product in my polynomial circuit. And so here, I'm going to assume just for convenience that this can be decomposed. And I won't get into those details, but in essence, it doesn't matter. With the right assumptions-- or even without, with a slightly different algorithm-- you can simply write this product, the integral over the product as the product of integrals. And then at the inputs, I have an input node, which if I integrate over it, it gives me a 1. Or if it has a value associated with it, like data, then it could be a 0 as well. So this is the backward integration. But it really means that now when I need to compute a marginal probability, I just plug in 0 and 1 at the input. And I evaluate feed-forward sums and products of sums and products. And at my output, I get a marginal probability. I apologize for the hand-waving, but I hope you trust me here.
Now for Ctrl-G, we had to solve a harder task. We didn't just have to compute a marginal probability over some tokens in the sequence that were unknown. We actually had to compute the probability of a DFA. And so you can take this intuition about how to do marginals on circuits and generalize it to a simple dynamic programming algorithm where, given a DFA with m edges and a circuit that represents a hidden Markov model with h hidden states, you can compute the probability of the DFA under the hidden Markov model distribution in time nmh squared, where n is the sequence length, so linear in the sequence length. And so this is exactly the algorithm we use in Ctrl-G to come up with this probability of alpha.
AUDIENCE: Is it essential that this is [INAUDIBLE].
GUY VAN DEN BROECK: It's essential for writing the paper in an intelligible way. Technically, also essentially you can use any circuit where the order of the products is always cutting a linear order in half. So it's slightly more general. It's almost like a context-free grammar. Yeah, actually with Marcelo at the last Institute, we wrote this paper showing that, yeah, you can use a richer language here, but then it gets more complicated. DFA, what's nice about the DFA is that in the paper, this dynamic programming algorithm is just a couple of equations. And it's kind of nice to see how it works. Yeah. It's a little bit more flexible than that. OK.
So now maybe you felt tricked a little bit because I told you we're going to use deductive reasoning algorithms. And in the end, what I gave you was another feedforward neural network. So this seems like I tricked you. But actually in a paper that we just put on Arxiv last week-- or it's on my website and pending on Arxiv-- we actually proved that if there is a polynomial time real RAM algorithm that computes marginal probabilities exactly and efficiently-- because it's polynomial time-- for a certain family of distributions, then there also exist polynomial-size circuits that represent these multi-linear forms of the distribution. So if there is an algorithm that computes marginals, there also exists a circuit of this form. And so you don't lose generality by assuming you have the circuit. So check out this paper, The Limits of Tractable Marginalization. OK, so this was the reasoning side, lots of hand-waving. There's also the machine learning side. We actually need to train this circuit from data to actually be a pretty decent language model so we can compute the probability of alpha. Well, the way we do this is by using these toolkits where we have to essentially implement super-scalable CUDA kernels, custom data structures on the GPU, to be able to train these models to have essentially billions of parameters. And so the hidden Markov model turned into a probabilistic circuit that we used for Ctrl-G actually had two billion parameters and was trained on the GPU. So that's the machine learning side, which I realize many of you don't care about. But you cannot achieve this without solving that problem as well. We also compared to some code from Google for state space models and turned out to be orders of magnitude faster. So that's the practical side of things.
So this wraps up the what's happening behind the scenes part. And so now I would like to talk a little bit about alignment and safety, which I guess is the main topic of this workshop. So often when we generate text from an LLM, we don't really care about the specific DFA. We have more vague concepts. We want it to be poetic or concise or factual or non-toxic. And here what I would like to do is say that I'm going to generate "It's a pain in" or "It's a pain to." The LLM has two probabilities. And I kind of want to know if I generate further from here onwards, am I going to generate something that is non-toxic or something that is toxic? And this is not a DFA, obviously. I need some other way of defining what the goal is here. I guess you can think of this probability of this attribute, toxic or non-toxic, as being the exponentiated reward function, if you're into reinforcement learning for alignment. People train these models that say, here's a positive reward for non-toxic and negative reward for toxic. So in theory, what I could do is look at all the future text according to my large language model. All that text has some distribution. And then I can also see, what is the amount of toxicity that this text has? "It's a pain in the ass," is a little toxic. Not so much, but a little. "A pain in the neck" is very much non-toxic. "A pain to deal with or handle" is non-toxic. So you have to see what is the expected value of my toxicity if I were to go this way versus that way? So it's again this problem of looking into the future. Again, it's intractable to know if you're going to generate something with high expected toxicity or not. So we'll use the same trick as we used in Ctrl-G. We will say, well, instead of using a large language model's continuation distribution, let's take a tractable probabilistic model and use its distribution as a proxy. OK? Further, instead of a DFA, let's just train a log-linear classifier to predict this attribute for a complete sentence, whether it is toxic or not. And this is our goal, is to have this classifier say whatever you generated is non-toxic with high probability. It turns out that a very similar dynamic programming algorithm that solves the Ctrl-G problem is able to compute this expected attribute probability. And it knows that if you say "It's a pain in," very likely you will say something toxic. If you say "It's a pain to," very likely it will be something non-toxic. And so now we invoke Reverend Bayes and use the product of these two to get our Bayesian posterior. And we will now sample, "It's a pain to" rather than "It's a pain in." Similar setup as Ctrl-G, but for these non-lexical, non-logical constraints. The beauty of this is that it's extremely fast. So the way that people do detoxification of language models today is they often train models on the side that predict whether something will be toxic or not in the future. Or they do reinforcement learning, things like DPO, where they essentially fine tune the model. And this takes on the order of hours. TRACE takes 10 seconds to train a log-linear model to classify toxicity from complete sentences. So it's from 10 seconds to hours is the difference at training time. Once you've trained your DPO model or your alignment control generation model, you still have to run it at test time. And even there, there's actually a huge benefit to using this TRACE algorithm. TRACE is 10% slower than the baseline. And all of these models that use these auxiliary neural networks or sampling trained for alignment, they're orders of magnitude slower. If you do this, you actually get state of the art LLM detoxification. So here you're measuring the amount of toxicity you get on average or in the maximum case. And these are all baselines that are in the literature for GPT-2 Large, where most of the literature is experimenting. And so we have less toxicity in our generation. Same holds for Gemma-2B, which is the more modern LLM. Also crucial is that when you do this, when you detoxify your model, you actually keep almost all of the diversity in your generation. So one way to detoxify a language model is to just get rid of all of its behavior, except for one specific thing it always says over and over. So that's kind of a failure mode of detoxification. And it turns out that if you do things like DPO, which is what Silicon Valley is investing billions in, for example, then you're actually losing a lot of the diversity in your generation. You get rid of a lot of the useful behavior, and you just pick a few obvious things to say that are safe. We are somehow being more safe while not losing the richness of the behavior of the model. You can also look at the fluency. So here you also see that, yeah, these models that have mode collapse, they're extremely fluent because they say the same very fluent thing over and over again. And we actually keep the same fluency as the original model. This is not a machine learning audience. So I'll skip over this. I'll show you-- Yeah, question?
AUDIENCE: I didn't understand the crossed out entry for DPO. What does that mean?
GUY VAN DEN BROECK: So what we did here is say these fluency numbers are, in a way, too good to be true. You cannot really be more fluent than the original model. Or you can be a little bit maybe, but if you're this fluent, it means that you have basically collapsed your diversity. I can be very fluent by keep outputting the Declaration of Independence over and over again. But my fluency comes at a cost of mode collapse, essentially. Yes?
PRESENTER: OK, question?
AUDIENCE: I have two questions. Do you use a list of words to control?
GUY VAN DEN BROECK: No. So if I wanted to use Ctrl-G for this task, I would have to use a list of bad words and say, never say these words, right? Here, what I'm using is a log-linear attribute classifier that takes in the token sequence and classifies it. It's a statistical classifier. It's a very simple one. And it's also looking at words, of course, which words occur. But it doesn't mean that a word is necessarily good or bad. It just changes the distribution of toxicity.
AUDIENCE: And the follow-up is, have you thought about using embeddings?
GUY VAN DEN BROECK: We have. And hopefully that's the next paper. So don't scoop us. OK, thanks. I guess I'll wrap up here. So this work is state of the art for detoxification. However, because it's so fast, now you can do anything you want. Specifically, suppose you're a big fan of Twilight Sparkle, which is this character here. I may want to have a chat with Twilight Sparkle. And so I might want to ask, "How is the weather?" I can train in three seconds a Twilight Sparkle classifier that says, this is sounding like this character. And then this character likes to say, "Gosh, it's sunny and very beautiful." And so I can essentially customize my language model to just be whatever I want. I can really efficiently control the generation. Previously, what you would have to do is-- I'm going to go back to my animation-- previously, you would have to ask through prompting and say, you're Twilight Sparkle, Twilight sparkle sounds like this. Here's some examples of what she says. And then you get this answer. So this is actually three times slower than this. Because here you have to put everything in your prompt. You have to process all of these tokens. And the answer is actually worse. So this is a very lightweight, yet very effective way of doing the control over your generation. You can do this for 76 models in three seconds each. So you want to talk to Doctor Who or Gregory House or whatever, you can do so using this model. And at inference time, it's only 10% slower than the baseline. Even prompting is three times slower than the baseline because of all the in-context examples. And if you want to do more fancy things, it gets even more slow. OK, I guess people in this audience care about RL. And I guess I have a few minutes left. All right, so let's switch gears. None of you care about large language models. You're all into actions and RL. So we try to also think, what does this framework mean in the context of model based or offline reinforcement learning? For background in offline reinforcement learning, we have a joint distribution over states, actions, and rewards. And we observe a bunch of people driving around, and we see their traces. And now we can train a transformer or whatever machine learning model we like to predict what would be a natural sequence of states, actions, and rewards. Now, at inference time, we are given a bunch of states, actions, and rewards, and we'd like to predict the next action that is likely to be done by a human in our dataset. Now, sadly, humans in our dataset are not great drivers. And so you don't really want to sample from the average driver. You want to say, I would like a driver that actually gave a high reward in the future, someone who found a good route and didn't have any accidents. I want to say, avoid certain states that are unsafe, avoid certain actions that are unsafe. And again, these are constraints you have about the future of the generation, where you want to condition your action on these constraints in the future. So same picture as before, we invoke Reverend Bayes. We say we will use our offline RL model, whatever you like, trajectory transformer or something like that, to predict the next actions. And we will use our probabilistic circuits to say, given a choice of action, what is the probability that we will satisfy our constraints? Meaning, we'll get high reward, we'll avoid bad states, and so on. Now, if we sample from this Bayesian distribution, things work really well. So if you just condition on high reward, you already get state-of-the-art performance on these standards, [? MuJoCo-like ?] offline RL benchmarks, so outperforming a decision transformer, trajectory transformer, and so on. Check the paper for details. What's nice is that because this is a proper Bayesian way of doing this, it actually works well in stochastic environments where you don't know perfectly what will happen. And these existing methods are quite brittle in terms of their performance on stochastic environments. And then you can also condition on safe actions, say, do not use this actuator more than this amount. And you then also get state-of-the-art results. So this stuff just works. Bayesian deductive reasoning works really well in combination with modern foundation models. OK, wrapping up. Do these algorithms still have a purpose? Yes, there's actually many more other cool applications, image inpainting and so on. There's a whole bunch of other stuff you can do that's really fun that's kind of state-of-the-art in terms of machine learning people, but is using algorithms behind the scenes. What did we do to change the 30-year-old algorithms that we all love to make them useful in this context? Well, we had to figure out how to learn them at scale to get a $2 billion-- $2 billion? I don't have that much money-- 2 billion parameter HLM on the GPU. And we also need it to be tractable. We cannot afford to call a SAT solver to please ask if your DFA will be satisfied. We have to do this on the GPU, on the fly, extremely fast, in linear time, or even in constant time for every single next token. The way that we got this is by thinking of everything as being circuits, circuits, circuits. The DFA is a circuit. The classifier is a circuit for the attribute. The distribution is a circuit and doing circuit algorithms on the GPU. All right, thanks for your attention. [APPLAUSE]
PRESENTER: There were plenty of questions, but it looks like they're still coming.
AUDIENCE: It's a wonderful talk. Here's a verification question for you. Is the circuit tractable enough that you can also convince us that the circuit that you learned is actually safe or trustworthy, that it has properties that we care about that you can use to convince us that it's doing the right thing?
GUY VAN DEN BROECK: Are you thinking about the DFA-based algorithm or the [INAUDIBLE] one?
AUDIENCE: Let's take the DFA one.
GUY VAN DEN BROECK: So the DFA one, the beautiful thing is that even if my hidden Markov model is a uniform distribution over all text, which is the worst language model, then the failure mode is the structured generation that people do, which means you just stop right before you drive off a cliff. Is that nice? No, you will get really crappy generations. But the safety is always there if you trust your DFA. The quality of the text is very much a function of how good the hidden Markov model is. And we also didn't know beforehand if it would be good enough. Turns out it was good enough to get people to rate it highly and beat GPT-4. But this is not for granted. And depending on your application, this may be easier or harder. Yeah. Same with the offline RL-- we didn't know that we could have a hidden Markov model that was a good substitute for a trajectory transformer. But it was good enough for the task, yeah. And of course, if you would only use the HLM and not use a transformer, these results will all be terrible. Yeah.
PRESENTER: Another question.
AUDIENCE: I have a question. So it seems there's a huge application for this. So what's your path? Are you collaborating with industries? Safety is a huge concern for a lot of companies.
GUY VAN DEN BROECK: I would love to collaborate more with industry on this. Yeah, we have some partners, some gift funding here and there. But I think if you really want to scale this up, you need some serious investment. And the funny thing about Silicon Valley, of course, is that everyone's looking at the same algorithm over and over again. And it's hard to convince people to take a risk on something out of the box, but hopefully that will happen. Yeah.
AUDIENCE: Thanks for a really good talk, and very impressive work. I was wondering if you've applied the first part, the DFA constraint, on large language model output to vision language action models in the planning setting, where these sort of DFA constraints might come in more naturally as specifications?
GUY VAN DEN BROECK: Yeah, that's a great question. In fact, we just got a new DARPA grant accepted on studying exactly this problem in robotics. How can we get constraints into these robotics pipelines using this framework to ensure safety. Yeah, so hopefully by fall, you will see some papers come out on this. Yeah, thanks.
PRESENTER: Are there more questions? OK, if not, let's thank speaker once again. And now we will have a half an hour break. So let's meet precisely at 11:00 so that we are on time. I'm stealing two minutes from your break.
AUDIENCE: So a quick question. So we are thinking of doing a group photo. And it's likely to be right before the afternoon session begins. So if people can come back from lunch by around 1:45, then we'll do the group photo outside before we start.