Transcription
[Music] Hello everyone. Uh, thanks Rick and the whole AI engineer conference team for putting this together and having me. I am Will Brown, a machine learning researcher at Morgan Stanley. And today I want to talk to you all a bit about what I think reinforcement learning, or RL, means for agents.
I was in grad school at Columbia for a while, and I mostly worked on theory for multi-agent reinforcement learning. Over the past couple of years, I have been working at Morgan Stanley on a wide range of LLM-related projects, some of which look kind of like agents, but I will not really be talking too much about that today. Uh, I'm also relatively active on X, the everything app, and that will become relevant later in the talk.
This talk, I think, will be probably a little different from most of the talks at the conference. Um, it's not about things we ship to prod. It's not about things that definitely work and you should go do tomorrow that are like proven science or best practices. It's about where we might be headed. And I want to really just tell a story that will synthesize some things that have been happening in the broader research community, um, and where these trends might be pointing, do some speculation, and also talk about some recent open-source work of my own. The goal is to help you plan and understand what reinforcement learning means, what it means for agents, and how to best be ready for a potential future which may involve reinforcement learning as part of the agent engineering loop.
So, um, where are we today? Most LLMs that we work with are essentially chatbots. I think it's helpful to think about OpenAI's five levels framework here. So we did pretty well with chatbots. Seems like we're doing pretty well with reasoners. These are great models for question answering. They're very helpful for interactive problem-solving. We have the 01, 03, R1, Grok, Gemini, etc. models that are really good at kind of thinking longer. Um, and we're trying to figure out how we take all of this and make agents level three. These are systems that are taking actions. These are systems that are doing things that are longer and harder and more complex. And currently, the way we tend to do this is chaining together multiple calls to these underlying chatbot or reasoner LLMs. We do lots of things like prompt engineering, tool calling, eval, giving the models tools of their own to use, having humans in the loop. And the results are like pretty good. Um, there's a lot of things that we can do, and then there's a lot of stuff that it feels like is around the corner that we're all imagining about AGI, but we're not really to the point yet where these things are going off and doing the things that we would imagine an AGI is really doing to the degree of autonomy that that would, I presume, entail.
So I think it's useful a bit to distinguish between agents and pipelines. I think Barry's talk earlier was a good way to kind of frame this. I'm going to use pipelines to encapsulate what Barry called workflows. Um, and I think these are really systems with fairly low degrees of autonomy. And there's a very non-trivial, non-trivial amount of engineering required to determine these decision trees to say how does one action or call flow into another? How, uh, to another? How do we refine the prompts? Um, and it seems like a lot of the winning apps in the agent space have very tight feedback loops. And so whether or not you want to call these agents or pipelines, these are things where a user is interacting with some sort of interface. They're telling it what to do. The thing will do some stuff and come back relatively quickly. Things like the IDEs, like Cursor, Wind Surf, and Replit, um, and search tools that are really good at harder question answering, maybe with some web search or research integrated. But there's not that many agents nowadays that will go off and like do stuff for more than 10 minutes at a time. I think Devon Operator and OpenAI's Deep Research are the three that really come to mind as like feeling a little more in the like autonomous agent direction. And I think a lot of us might be wondering how do we make more of these? And the kind of traditional wisdom is like, okay, we'll just wait for better models. Once better models are around, we can just like use those; we'll be good. But I think it's also to kind of take note of like the traditional definition of reinforcement learning and what an agent means there, which is this idea of a thing that is interacting with an environment with a goal. And the goal of that and the system is designed to learn how to get better at that goal over time via repeated interaction with the system. And I think this is something that a lot of us are either doing manually or don't really have the tools to do, which is once we have our thing that it's set up to make the calls we want and the performance is like 70%, and we've done a lot of prompt tuning and we want to get it up to 90%, but we just like don't have the models to do it or the models struggle to get the success. What's our path forward?
Um, and so in terms of model trends, I think I won't spend too much time talking about this, but uh, pre-training seems to be having diminishing returns to capital, at least. We're still seeing kind of like loss go down, but uh, it does kind of feel like we need new tricks. Um, reinforcement learning from human feedback is great for making kind of friendly chatbots, um, but it doesn't really seem to uh be continually pushing us at the frontier of smarter and smarter and smarter models. Uh, we talk a lot about synthetic data, and I think synthetic data is great for distilling larger models down into smaller models to have kind of really tiny models that are really performant, but on its own it doesn't seem to be an unlock for like massive capabilities, getting better and better unless we throw in verification in the loop or rejection sampling or any of these things, and that really takes us to the world of reinforcement learning where this seems to be the trick that unlocked test-time scaling for 01 models in R1. Um, it's not bottlenecked by needing manually curated human data, and it does seem to actually work. I think we all kind of took note about a month ago when DeepSeek released the R1 model and paper to the world, and I think this was really exciting because it was the first paper that really explained how you build a thing like 01. We'd had kind of speculation and some rumors, but they really laid out the algorithm and the mechanisms for what it takes to get a model to learn to do this kind of reasoning. And it turns out it was essentially just reinforcement learning where you give the model some questions, you measure if it's getting the answer right, and you just kind of turn this crank of giving it feedback to do more like the things that worked well and less like the things that didn't work. Um, and what you end up seeing is that the long chain of thought from models like 01 and R1 actually emerges as a byproduct of this. It wasn't kind of manually programmed in where the models were like given data of like 10,000 token reasoning steps; this was a thing the model learned to do because it was a good strategy, and reinforcement learning at the core is really about identifying good strategies for solving problems. Um, it also seems like open-source models are, are back in a big way. There's a lot of excitement around the open-source community. Um, people have been working on replication efforts for the 01 project, um, and have also been trying to distill data from 01 down into smaller models.
And so what next? How does this relate to agents? Um, I think it'll be helpful to know a little bit about how reinforcement learning works. The key idea is to explore and exploit. So you want to try stuff, see what works, do more of the things that worked, less of the things that didn't. And so in this feedback loop, um, demonstrated here in the image, we can see a ch- a challenge where a model is uh supposed to be writing code to pass test cases, and we give it rewards that correspond to things like formatting, using the right language, and then ultimately whether or not the test cases are passing. And so this is kind of a numerical signal that rather than like training on data, uh, where we are kind of curating this in advance, we are letting the model do synthetic data rollouts and seeing scores from these rollouts, which then are fed back into the model. And so the GRPO algorithm, which maybe some of you have heard about, is the algorithm DeepC used. I think it's less of like a technical breakthrough in terms of it being a really important new algorithm to study, but I think it's very conceptually simple, and I think it's a nice way to think about what reinforcement learning means. And the idea really is just that you for a given prompt sample n completions, you score them all, and you tell the model be more like the ones with higher scores. Um, this is still in kind of the single-turn reasoner model non-agentic world. Uh, and so the challenges that lie ahead, um, are going to be about how do we take these ideas, uh, and extend them into more powerful, more agent, more autonomous systems. But we do know that it can be done. So OpenAI's Deep Research still has a lot of questions that we do not know the answers to about how it works, but they have told us that it was end-to-end reinforcement learning. And so this is a case where the model is taking up to potentially a hundred different tool calls of browsing or querying different parts of the internet to synthesize a large angler. And it does seem, I think, to many people's vibe-check opinions, very impressive. Um, but it also is like not AGI in the sense of you can't get it to go like, uh, work in a repo or like solve hard software engineering tasks. And people have kind of anecdotally found that it does struggle a bit for like out-of-distribution tasks or like if you wanted to fill out a table with like a 100 very manual calculations, it can struggle there. And so it seems like reinforcement learning on one hand is a big unlock for new skills and more autonomy, but it's not a thing that so far has granted us agents that can just do everything and know how to solve all kinds of problems. But it is a path forward for teaching a model skills and having the model learn how to get better at certain skills, particularly in conjunction with environments and tools and verification.
Um, and so there is infrastructure out there for doing this on our own kind of. Um, a lot of it is still RLHF style, by which I mean it's about kind of single-turn interactions where the goal is we have reward signals that come from kind of human data that has been combined into a reward model. Um, and if we want to have RL agents becoming part of our systems, maybe we will get really good API services from the large labs that let us build these things and hook into GPT whatever, um, or cloud whatever and train these sorts of models on our own with fine-tuning, but we also don't really have these options yet. Um, OpenAI has kind of teased their reinforcement fine-tuning feedback, but it's not uh multi-step tool calling yet. And so I think if we want to plan ahead, it's worth kind of noting and asking what would this ecosystem look like? And there's a lot of unknown questions like how much this will cost, how small can the models be, will it generalize across tasks, uh, and how do we design good rewards and good environments? And there's a lot of opportunity here. Um, open-source infrastructure, there's a lot of room to build and grow and determine what the best practices are going to be, what the right tools will be, as well as companies that can build tools for to support this ecosystem, uh, whether or not they're already in the fine-tuning world or not. Um, and services for supporting this kind of agentic RL. And I think also it is worth thinking about things that are like not literal RL in the sense of training the model, but at the prompt level there's all sorts of automation we can do. So if you've used DSPI, I think that is kind of adjacent to RL in the flavor of having a signal that we can then uh bootstrap from to improve our uh underlying system based on improving some downstream scores.
Now I want to share a story with you about a single Python file I wrote a couple of weeks ago. Um, so this was the weekend after R1 came out, and I'd been reading the paper and thought it was really cool. We had not had the Nvidia stock crash quite yet. Um, and uh, I was just playing around with some experiments. I was taking the a hug a trainer from HuggingFace that had the GRPO algorithm, and I was getting a really small language model, Llama 1B, to do some reasoning and then give an answer for math questions. And I started with like a pretty simple system prompt, and I was just training the model to let it see what it did. And I had kind of manually curated some rewards in terms of what the scoring function should look like. And I just kind of like tweeted it out, um, where I had an example of the model kind of looking like it's doing some self-correction, and so showing that the accuracy gets better as well as the uh length of response will initially drop once it learns to kind of follow the format. Then it goes back up as it learns to kind of take advantage of longer chains of thought to do its reasoning. And this was not the first thing to replicate in any sense; I wouldn't really call it a true replication. Um, it was far from the most complicated, and I think that actually caught a lot of people's imaginations, and it became kind of a thing. Um, so over the next two weeks after that, it just took on a life of its own where a lot of people were kind of tweeting about it and forking it and making modifications to it and making it something you could run in a Jupyter notebook, making it more accessible, writing blog posts about it. And it was interesting, um, because it to me didn't feel like a thing that kind of merited this level of excitement. But what I think was catching people's imagination was that it was one file of code. It was really simple, and it invited uh modification in a very user-friendly engaging way, which I like to call rubric engineering. And so the idea of rubric engineering here is that similar to prompt engineering, um, to have a model do reinforcement learning, it's going to get some reward, but what should this reward be? In the most simple version, it's just like did it get the question right or wrong, like does a equal b? But there's a lot more you can do beyond this. And so I think the single file of code exposed uh examples of this where you can give the model points for things like following this XML structure. Like if it gets a certain tag right, you give it plus one point. Um, if it has an integer answer that's still the wrong answer, but it's learned that the format should be an integer answer, get some points for that. Um, and there's a lot of room here for getting creative and for designing rules that are not just downstream eval to for our own sake know whether a thing is working but to allow the model itself to know whether it's working and use that as feedback for going further and training more. Um, and this is very early stages. There's a lot of things we don't know, and I think there's a lot of opportunity to get creative and explore and try things out, such as using LLMs to design these rubrics, uh, autotuning these rubrics or autotuning your prompts with frameworks like DSPI, um, incorporating LM judges as part of the scoring system. And then also I think reward hacking is an issue to be very cautious of, where the idea is you want to ensure that the reward model you're using is actually capturing the goal and it doesn't have kind of these back doors where a model can kind of cheat and do something else that ultimately results in it kind of getting a super high reward without learning to do the actual task.
And following this, I have been trying to learn from those lessons of what I saw people using out in the wild and make something that is a little more uh robust and uh usable for actual projects beyond just one file of code. Um, and this has been a kind of very recent effort. It's not a thing that I'm telling you to go use for all your problems tomorrow, but I think it's my attempt doing some open-source uh research code, um, that will help people potentially try these things out easier and answer some questions uh about this. And so what this really is, it's a a framework for doing RL inside of multi-step environments. So the idea here is that lots of us have built these great agent frameworks for using API models. And the hope would be that we can leverage those existing environments and uh frameworks to uh actually do RL. So here the idea is you can just create this environment thing that the model plugs into, and you don't have to worry about the weights or the tokens; you can just write an interaction protocol, and then this gets fed into a trainer, and so once you build this environment, you can just kind of let it run and uh have a model that once you give it some rewards learns to get better and better over time.
To conclude, I want to talk about what I think AI engineering might look like in the RL era. So this is all still something that is very new. Uh, we don't know whether the off-the-shelf API models are going to just work for the tasks we throw at them. It might be the case that they do. It might be the case that they don't. Um, one reason I think that they might not be the entire solution is that it is really hard to include a skill in a prompt. You can include knowledge in a prompt, um, but a lot of us when we try something we don't nail it the first time, and it takes a little bit of trial and error. Um, and it seems to be the case that models are like this as well, where a model does get better at a thing and really gets a skill nailed down by trial and error. And this has been the most promising unlock we've seen so far for these higher autonomy agents like Deep Research. Um, fine-tuning might still be important. And I think a lot of people wrote off fine-tuning for a while because open models were far enough behind the frontier that like a prompted o- frontier model API was just going to beat a smaller fine-tuned model. I think one we're now seeing the open and closed-source gap be close enough that this is less of a concern. A lot of people are using open-source hosted models in their platforms. Um, and also uh RL, the most kind of true version of RL that DeepS did for their R1 model that OpenAI has talked about doing for uh Deep Research, requires doing some reinforcement learning. Um, there's a lot of challenges here. There's a lot of research questions we don't know the answers to. Um, but there's a lot of things that I think these skills we've learned from doing AI engineering over the past couple of years translate very directly to, which is that the challenge of building environments and rubrics is not that different from the challenge of building evals and prompts. We still need good monitoring tools. We still need a large ecosystem of companies and platforms and products that support the kinds of agents we want to build. Um, so I think all the stuff we've been doing is going to be essential, and it's worth looking ahead a little bit to see if we end up in a world where we have to do a little bit more reinforcement learning to unlock things like true autonomous agents or innovators or organizations that are powered by language models. Um, what does that look like? Uh, we will find out. [Music] [Applause] [Music]