How GPT-5 Thinks — OpenAI VP of Research Jerry Tworek

01 to be perfectly honest, it was really mostly good at solving puzzles. It was almost more like a technology demonstration.

03 has really been something like a tectonic shift the trajectory of AI. GPD5 in some way, like I know, can be considered as like 03.1. What I am after is something that would be that would be the next pretty significant jump. We understand that there's only one time in history where AI is being built and deployed and developed. We are together in the skull that's larger than everyone of us.

>> Hi, I'm Matt Turk from Firstark. Welcome to the Mad Podcast. Today, my guest is Jared Foric, VP of research at OpenAI and a member of the Metis list of the world's top AI researchers. In this episode, we go deep on how models actually reason. We also go behind the scenes at OpenAI, how a few big bets get staffed, why everyone knows everything, and how that culture ships fast. Please enjoy this great conversation with Jerry.

>> Hey, Jerry. Welcome.

>> Hello. Very happy to be here.

>> We are going to talk about reasoning a lot in this uh conversation. At a high level, what does actually mean? When uh we talked to chat GBT and Chad GPT says it's thinking, what actually is happening behind the scenes?

I I think that that the thinking process is at least a good analogy. As we were in the early days of AI, always had this goal dream of trying to teach models to reason. We were thinking about it, spending more time to get better results. If like a human is posed with a very hard problem in front of them, uh, very rarely they have they have answers straight away. Sometimes they need to find that answer. Sometimes they need to perform certain computation. Sometimes they need to like look up some information. Sometimes they need to teach themselves something. And the process is reasoning is like getting to an answer that you don't yet know. Like in some way, can be called search, but it's not really like a very naive search. Search is a loaded word. But reasoning is the process of getting to an answer and a work that you need to do that is like longer than what usually is considered answering a question. I think the the difference is here, like answering a question usually means you already know the answer and you just just just elicit the answer you know. And the process of reasoning is getting to the answer that you don't know. And usually the longer you spend on getting to this answer for whatever you need to do to get there, the better the better it gets.

>> And uh, we've all become familiar since you guys uh released 01, I guess a little over a year ago now, in September 2024, with the concept of ch of thought, which is in layman's term, the little messages that you see when you query chat GPT and it tells you it shows its work, it tells you what what it does. What does that actually do? Is that a logical tree and it eliminates option after option? What actually happens?

Language models do on on their own, like fundamental level, is they they are often called as next token prediction machines. And that that's not completely accurate in the age of reinforcement learning, but they still operate on Mosium tokens that are most text. The language models again, these days, also multimodal and they operate on on on multi text. But to simplify a little bit for a second, uh, language models generate text. And what chain of thought is, is their thinking process verbalized using human words and human concepts. So the the magic that we are seeing, why this is all possible, is that while you are training on all of internet, on a lot of human knowledge and human thinking process, the model starts learning in some ways to think how humans do and in some ways get to the answers how humans do from seeing humans do it a lot in the text that was that was pre-generated and that that that was based in a training data on humans. And then and then the train of thought, it's basically eliciting that capability in language models of like thinking and getting to an answer like like like like like humans do.

H A lot of what's what what's like what early chain of thought work was doing was kind of solving math puzzles. And the first like most famous prompts to elicit chain of thought in language models was so called like, let's solve it step by step. There is this this this like very classical result in language models that if you ask them like what is some like either either mathematical expression or some puzzle, then and they will try to give you an answer. They will try to predict the next token, but they they fail. It's a hard thing. They cannot compute it in one one token jump. But if you ask them like, please do it step by step, and they will start thinking, okay, I don't know the answer, but the first step of getting to the answer is this. And then and then they write like chain of thought, which is a series of text, series of tokens doing the first part of the computation, the second part of the computation, the last part of the computation. Then they connect those things and then they can get to the answer. So the chain of thought is basically a process of thinking encoded in in words, how how humans would solve a problem on a piece of paper, going going step by step from the from start to the end.

And and since uh time, and by that I mean the time spent thinking, is so uh important to that concept of reasoning, how does a model decide uh how long to think when we're in JPD5 and we're in auto mode and it says uh that is going to decide automatically how long to think? What what happens there?

It's basically part of our optimization process, partially for for for the the happiness of the users and what they want to expect. Because when when you have a thinking process, you need to balance two things, which is like the quality of the result. That's as we said, and there have been like those pretty great scaling laws that we demonstrate with the release of 01. The longer the model thinks, the better result you get. But also people don't like waiting. Waiting. Waiting is time lost that you could that you could do something. Everyone wants to get results as quickly as possible. And like, you know, there is this there is this this saying, you can get like, you know, cheap, fast, or good, and you can take two. And that's that that applies to language models as well. There is a there is a trade-off, and it's it's delicate. That's why like we also expose some of that trade-off to the users, where you can have like a high resing model and a low resing models. And this is this is like in the end, the same model. You just we just tweak the parameter which says, we want you to think longer or shorter. We try to encode some heristics of what we think the users will want when when get thinking on an answer a little bit longer will and getting to a better answer is worth it waiting versus not. But it's a it's a bit of a trying to guess the anticipation of the users, what's the what's the right amount of thinking for them in this particular situation.

>> Oh, fascinating. So it's more uh user-driven. So it's more like a user experience uh kind of thing.

>> In the end, it is because the question is like, how long do you want to wait for answer? You can always wait longer and get get even better answer.

>> It's been uh a little over a year since the release of uh the world's first reasoning model 01, which is an effort uh that uh you led. What has been the journey since? So there was 01, then there was 03, uh then most recently JGB5. How would you characterize the evolution of reasoning specifically across the three models in the last year?

>> In in some way, how I characterize our reasoning or scaling up reinforcement learning research program is we do a series of scale up runs that are progressively more and more ambitious. Everyone we try to do something more, something larger scale, something that that should result in a better trained model than the last one. And and obviously, we don't release all the models that we train. Some some we release, some we think like, I need to need to wait a little bit longer for the moment where where they will have their time to shine in the hands of the users. But like 01 was the first model we decided to release as kind of uh like to demonstrate to the world, there are those models. And 01, like to be perfectly honest, it was really mostly good at solving puzzles and like maybe a few kind of thinking problems here and there, but it wasn't like it wasn't yet a very useful model. It was almost more like a technology demonstration than that that an actually really polished product. But we were thinking, we have something cool and we wanted to share it with the world as as OpenAI. 03, I think like changed that pretty significantly. In some way, it is a model that is meaningfully useful. And like, you know, a little bit self-serving, but was a moment when I when I started using CH GPT quite a bit, and I I'm basically a user completely hooked on on reasoning model. And in chpd right now, I use basically exclusively reasoning models because those are the only models that I trust the the output and layer and the result. And I think 03, like, it's its ability of of using tools and getting to like answer leveraging a lot of contextual information from various sources and persevering to to towards getting to that has has really been something like, you know, I think I think that there was a lot of a tectonic shift in the in the trajectory of AI. And like, you know, I think I think like we did we did something really really great there. Like GPD5, in some way, like, you know, can be considered as like 03.1. It's it's a little bit of like like, you know, iteration of of of like the same thing. And the same concept. And what what I am after and my team right now is like, you know, something next that would be that would be the next pretty pretty significant jump of how we interact with with like models that that that are even more capable, thinking even longer, and interact with with even more uh systems and sources on information on their own on their own journey. But but like separately, in the meantime, we continue to build a lot of things on top of all free technology, like Codex, which I think like coding agents are at the moment, the first like really successful agentic products built on top of AI. There are things like computer using agent, it's called GPT agent right now, I think, and like the prearch and a few a few other things that that we will will keep on building on like three generation technology.

>> Great. All right. So, we're going to go into uh all of this uh in much greater detail in a minute. Um, but before we do that, let let's talk about your journey. I think it's it's super fascinating topic like for for all of us, like you guys are changing the world. So, uh I think I'm I'm curious and we're all curious, I think about um the the people, the human aspect of like who those people are that are, you know, just having such an impact. So you you starting from the the beginning, you you grew up in Poland, I believe, right?

>> Yes. I grew up in Poland.

>> Walk us through your formative years and uh how you got to get started in this field.

>> Yeah. Yeah. Yeah. Happy happy to do that. An interesting fact. It's almost like a like a crystal starts from something and you put a little bit something in in in the beginning. There's I think one part that was like important and part like like the the starting point of my my journey where I where I don't know where it came from because it was with there with me from the very beginning of my life, in a in a moment that I don't really know when it started. It just was always there with me. I always thought that being a scientist and doing science is the highest calling a human can have. And I don't don't really know where it came from. My parents maybe like were were singing the light bright lullabies to me like when I was when I was one or something like that. But basically since I remember, I wanted to be a scientist. In the early years, I also discovered like, you know, have I have talent for for those things. I like I was going to school and I saw I get things slightly faster than than people around me, at least like, you know, in a regular school and in in the middle of Poland. And which which like, you know, made me made me kind of like those doing those things, like studying maths and science a little bit more because because like, you know, I felt it it felt good in a way. It felt like this is this is this is something that naturally naturally fit me. Like, you know, I grew up as a like, you know, very regular kid, just just like, you know, being slightly nerdy guy and trying to balance my my side of of being interested in science, programming, maths, and like having some social life. And I I definitely have had some kind of like party arc in my in my life. But I think I I think that the most important part and moment was when I actually went to university, college, University of Warsaw is where I went, and I decided again to study mathematics at the at around that time of being 18. My idea of life was to be a mathematician with a pencil sitting in a room with a piece of paper and solving equations. This is kind of my my my 18-year-old dream of how how how life should be lived and what I what what I want to do in in my life. And like, you know, I I do have like, you know, my my personality is built again in a way of really appreciating like, you know, solid science, pursuit of truth, great great engineering, and all the all those aspects. But I definitely have like also a little bit of like a misfit kind of rebellious uh tinge to it. And that that resulted after a few years of studying mathematics. I what I realized about myself and about about the world is that I really like maths and I am I'm I'm quite good at it, but I didn't like academia that much. And I realized I don't want to like stay in academia. I don't want to stay in university. And that this this would not be an environment where I thought I would be long-term very happy and and fit in. It felt a little bit too like rigid, a little bit too structured in a way that I that I kind of like didn't didn't know if I if I will will feel good.

>> And in some way for young me, like I was I was around 21 years old at that moment, that was pretty big crisis of faith for me. I had had a moment like, you know, lost purpose of life. So I just did like, you know, a very simple like first principles thinking. H you know, I am graduated with a degree in mathematics. I need to get a job to get food and like, you know, what job can I do to use mathematics and in in that job? And like, you know, looking at the job market that moment was like, you know, 2011, I think, or or around or or 2010, somewhere somewhere around that. Like I I I decided to become a trader and trade for a living as as like, you know, the the one way where I can like do what I like, which is mathematics and and gets uh gets get a career. I got a I got a quick internship at JP Morgan uh tra investment bank trading floor and equity derivatives group. Spent six months there learning a little bit how how does trading work and what does what does what does it look like a little bit. Finished my degree. H I got I got a message from like uh boss of my boss at JP Morgan, something saying, hey Jerry, you were like, you know, one of our best interns ever that we had. We really really liked you working with us. And we are we are leaving bank and starting a new hedge fund. Would you want to come with us? And like for for 20 21 or 22 2-year-old Jerry, that sounded like kind of like a cool adventure type of story that that that I I was interested in in going there and do it. It had enough like uh enough interesting problems to be solved and at the same time, it had like, yeah, this this kind of trying trying something new, trying something ambitious kind of kind of bed that I generally like. I was in London. That company didn't really work out unfortunately, but it was was hard and ambitious and not everything works out. I did try that again, starting another hedge fund from scratch with a few other people in Amsterdam. I worked there for a few more years and eventually eventually I got bored. Like I I generally working in trading is an interesting and exciting problem. Market is very hard and the depth of what you can go of trying to understand it model is very deep. And I worked with pretty smart people overall, but I I stopped feeling I am growing after a few years of doing that. And at the same time, together with a friend I was working with, we just started chatting about AI and about this this like artificial intelligence. And what really drew me to artificial intelligence was reinforcement learning. And and specifically the DQN agents trained by by people in deep mind in in like 2013. But I think it was much a few years later that I actually learned about those results from from my perspective. And again, this is this is just just how my brain works. The the 2012 image net results like weren't that significant like during during my my university years. I learned bunch about like how classical AI, the the neural networks weren't very fashionable back then, but I still learned about what they are. I learned about SVMs and all kind of methods, how you train classifiers. And for me, it was kind of like obvious and natural. If you have enough parameters and tweet it hard enough, you will fit a classifier to whatever you want. It was was kind of obvious. What like what was to me not obvious is I never considered classifiers a smart thing. Classifier is a you learn a function on some set of inputs to some some set of outputs. H we can can keep training it to approximate better and better. What what kind of what what was some thing that I missed back then is that when you can fit any function better and better, you can start shaping behaviors and strategies. And what I when I really saw that was in the um in the like DQN results, where they applied the same things that worked in image net, like neural networks, and they weren't particularly big or impressive neural networks, with with a classical field of reinforcement learning to to solve simple computer games. And turns out those simple neural networks with a simple learning algorithm started learning pretty complex computer games and exhibiting very interesting behaviors. I saw those behaviors. I saw those results and I was like, this is what I want to do for for the rest of my life. Which is like, you know, not not very long horizon, what the 20some things about, but I was like, this is this this is what I want to do. Where where where do I do that? Like, you know, Google search, where are places where you can do reinforcement learning in this world? H like, you know, Google DeepMind and OpenAI uh came up with this kind of at that moment, pretty small and like somewhat known, but they were

>> Yeah, you joined you joined OpenAI in 2019, 19, right? So, like very much u very much in the early days still, very much in the kind of like nonprofit era.

>> Yes.

>> Of open air. How so? How did you connect with them?

>> I just just applied for the website. I the most the most like boring and uninteresting thing in the world, which is like, you know, openai.com jobs, like, you know, apply, send resume, and hope hope they respond. And then, you know, luckily enough, I was they did. I don't know, I don't know how many résumés open AI was getting in that time. I think it's definitely much less than today. But I was I was like, you know, I I came there and I was like, no, that that doesn't matter what do I do as long as it's reinforcement learning.

>> So you joined in 2019 um and with a passion for reinforcement learning. So was that around the Dota 2 moment? Because open interestingly in in those early days of 2019, did a lot of reinforcement learning focused work, right? And then then there was a whole like unsupervised learning uh GBT moment that happened afterwards, but like it started from roots in reinforcement learning, right? So did did you work on that project specifically, or was it uh too advanced by the time you showed up?

So the the project that I worked was robotics project at OpenAI, which share the same code and same methods as the Dota project. Like in in one hand, the Dota project was OpenAI's way to demonstrate to the world what scaling up reinforcement learning can do. And in some way, it was like taking the the 2013 DQN agents and just just doing all the hard work of making it bigger and bigger and solving harder and harder problems. And OpenAI generally from the very beginning was aware and really like, you know, it was a it was simple but but genius inside that you need to have large scale system to to learn really, really interesting complex behaviors. And that was that was like one one way of what Dota was trying to show, that by scaling up reinforcement learning, we can solve pretty pretty complex environments. And then like we have there was another project or there I think three reinforcement learning projects at OpenAI at that time. And the second one was robotics, which is applying the same methods that we now knew or or were proving that can solve pretty complex computer games. Can they solve all the practical problems? OpenAI was always optimistic and ambitious and trying to see if we can scale around to solve time. Can it load my dishwasher? Can it fold my clothes? Can it build a house? And and and that this is this is what we are doing. The project I was working on was focused on dexterous manipulation, which was back then and still continues to be an elusive challenge for for for trained policies. And like we got to a showcase of demonstrating that a hand controlled by neural network was able to solve Rubik's cube, which which is a pretty delicate and complex task to do.

So fast forward to um today, still in the same vein of like the behind the scenes of um, you know, you all at OpenAI and sort of life there. What's a day in the life of Jerry like? What what what does somebody like you do? Like you you you read papers, you train models, you manage teams, like what's a day like?

Yeah, my my my days are surprisingly uniform, which is I come to the office early in the day after driving my kids to school. Then what do I do all day is basically talk to other researchers. I talk to other researchers all day, every day. And this is basically exclusively what I do. I take ideas from people, bounce with them, brainstorm with one partner, then move to another one and do the same thing over and over and iterate. And in that case, in in in in in in that way, like keep refining our research program. Sometimes those are group meetings and group meetings are there as well and have their own team team dynamics, but that is that is basically exclusively what I do. The only thing that that changes is the the topics of of research from from from from meeting to meeting and from from person to person.

How are uh priorities in research determined? Uh this range of possible projects. Is that top down? Is that bottoms up? Do people suggest ideas and others vet them? How does that work?

>> Yeah. Yeah. Yeah. It's the the art of like structuring, organizing, and leading a research project is like something that I generally like learned to appreciate very quickly in in OpenAI's journey and in my career. There is something we are good at is is is structuring research projects. And it's I think it's a unique mix. You cannot say it's top down, we cannot say it's bottom up. It's a mix of those two, which is which is balancing all all the important aspects. Which one one one thing OpenAI embodies and determines is like, we all work on a very few projects. Total there are not that many projects. OpenAI is not trying to do everything. We are not trying to to like have a portfolio. We are trying to have like multiple different bets. Always the the idea is we do few core things really really well and put a lot of effort there. Which means there are there needs to be a lot of people working together on the same large scale, large ambition project. And we have we have a few of those. Small number, probably three or four, depending depending on how do you how do you call it. And that's and that's it. And from that perspective, like people don't have ultimate freedom. It's not that that people come to open and say, hey, I want to do this, and they just they just do this because you need to do something towards the goal of one of those four projects. And then within those projects, like we try to be like relatively bottoms up in in a way, as long as it feeds again into into those goals. And the most important part of the research lead is keep to making sure all the researchers are working towards this one shared goals and that don't fracture in their own ways of of of thinking and and and doing things. And it's an incredibly hard thing. It's a very, very hard job and not always uh not not always easy easy to visible how how how how delicate it is. But that's that's a lot of of what it is. Like I don't think like, you know, top down uh structuring of research doesn't work in research organizations. I really don't believe in it because like you are not kind of hiring some of the smartest people in the world, and OpenAI has incredibly incredibly smart people to to kind of tell them what to do. They need to like they need to figure out what to do, but they cannot figure out in the whole space of things what are what are cool things to do. They need to figure out from within the space of what the project needs and what is what what could advance the research goals of of OpenAI the most.

>> And to which you're saying is there collaboration between the teams working on the three or four projects at the same time? Because I would imagine putting myself in your in your shoes, OpenAI's shoes, there is probably a tension between wanting to be collaborative in general, but equally uh I mean that this is probably the most important IP in the world. U so you probably want to uh make sure that not everybody knows everything about everything. Well, perhaps not. I'm I'm speculating here. How do you think about that collaboration versus um versus some protection of uh of IP?

You'd be surprised, but the truth is in research at OpenAI, which is like around like slightly less than 600 people at the moment, everyone knows everything, really, really it does. And we always have been fully transparent. And it is like, in some way, you are a little bit shooting yourself in the foot if there is a researcher that doesn't at least like have the chance to learn about everything, because they don't have the best information to do their job in a in a best way. And like it is like, yeah, it is some like risk of of losing IP, but I think I think the risk of not doing the right thing and of people not being informed about about research and not being able to do the best research is much higher. In my in my personal opinion and how how I I approach those things. So we are we are extremely internally transparent with within within research and and that is that is one of our operating principle. As as the goal is to do the best research we can and and train the best models we can. Uh consequently, and the culture generally is very collaborative. Like, you know, it is always the case when you have 600 people, when you have groups of people, they're always like one person doesn't like the other person because they looked at them weirdly, or one person thinks the other smells bad, or or just doesn't like their ideas. Like it that does happen. Those are humans and those are those are human things. But generally, in large scale, I think we really are have this belief, like, you know, we are together in the skull as larger than every every one of us. It is a very positive sum game because the AI seems to be getting like, you know, only more and more significant. And the success of OpenAI is far from guaranteed. It depends of on us doing great work every day. So there there there's a lot of like feeling of shared faith and and the fact that we all need to rely on each other to do our our job to to achieve this this this shared mission. So I generally I generally like, I think with with all the all the caveats of human nature getting in the way sometimes, I think I think on a on a large scale, OpenAI is very, very collaborative.

How do you um all manage to keep that um pace of releases? Like it seems to me from the outside that uh there's a tension again like between between another kind of tension like between research uh which in in in some ways is uh kind of feels like it could be a long-term kind of thing. And on the other hand, like you guys seem to be just shipping and shipping and shipping and shipping uh across the organization, but including in terms of like core models, like again to the point that you went from 01 to 03 to GPD5 in like a year. Um how do how do you balance all of that? Like why are you guys able to ship so uh quickly?

I I think that the fundamental reasons for it is in general OpenAI kind of in my my at least worldview is a generational company in a way that we have incredible momentum behind us. We know that we were doing pretty great in the past and we need to continue that. We have incredibly smart people, like literally the most talented people in the world are all coming and want to work at OpenAI right now, which means like people every output per single person is incredibly high and everyone really every every single person does a whole lot. So so we have we have like a momentum that carries us forward. We have really great people that work together. We have like good operating like way of structuring research and and and can borrow a lot from Silicon Valley how how to get things done quickly. And people are generally very excited about work. Everyone feels the weight and potential of what we are doing, what we are trying to do. And because of that, people at OpenAI have a tendency to work pretty hard. And and then like, you know, having great people excited about what they are doing, all working together reasonably well, results in in doing a lot of things. Like we like understand that there there's only one one time in history where AI is being built and deployed and developed. And people want to do it in the best way that that is possible.

>> Do you all use a lot of your own tools? I think Fiji Simma was tweeting the other day that um I think in the latest what you all announced um at dev day today, a lot of it was uh written by Codeex. Um is that is that part of the daily experience? Do you use a model to come up with new ideas for model? Do you use a Codec to write the code? How does that work?

Yeah, like we definitely use Codec a lot for coding, and this is only getting getting better. Like as as I said, I use chat GPT a lot, although not surprisingly, not that much for for actually coming with with ideas, but like for for a lot of questions that I have. I I I think I am pretty heavy user of of chat GP right now, happily paying like $200 a month for it. And I think I'm getting

>> They make you pay

>> for for what it's worth, they are making you pay. And I'm I'm kind of like pretty pretty okay with is because you then you get pretty generous like usage limits and and not really not really bottlenecked on it.

>> Thank you for all of that. Let's switch tags and um go back to uh how all of this uh works. So is the right way to think about modern AI systems at at OpenAI by by modern I mean as of October 2025 versus the old days of, you know, nine months ago. Uh so exactly. So is a is the right way to think about it as a a a a combination of uh pre-training and RL? First of all, is that the right way to think about it? And and and and second, if so, just at a high level, how does the articulation between both of those work? And then after that, I'd love to do a little bit of a uh, you know, deep dive on RL to make this very educational for for folks.

Today's language models are basically can be thought as like, first, they are pre-trained. Then you do reinforcement learning on it. Like the reinforcement learning would not work without pre-training. And I think in a similar way, like pre-trained models have a lot of limitations that are very hard to like resolve without doing something that looks like reinforcement learning. So I think both of those bits are here to be and to stay. Like I think the way how they are like combined and and do may and probably will evolve in the future. Nothing should be treated as as dogmatic and fixed, and we need to we need to keep generally figuring out the way how to train better models. And this is what we are trying to do. The interesting thing, and you know, I can credit that to to Ilya, how much foresight that he had. But whenever I was like h started at OpenAI early 2019, right? And and and like I remember there was like research all hands or something like that, where I came on stage and talked about like, what is what is OpenAI's research program? What are we trying to pursue? And what he said at the beginning of 2019 was to train large generative model on all data we can and then do reinforcement learning on it. That was that was the OpenAI research plan at the beginning of 2019. And this is exactly what we are doing today. Uh, the algorithms change, architectures change. Like I don't think he was even thinking about transformer at that moment. Like, you know, the GPT was like, there was some GPT, but it was like a toy example that someone was playing with. But the goal, training large generative model, all the data in the world, and then doing reinforcement learning with it, was was already there at the core DNA of OpenAI. And said that's that's that's what is happening right now.

So let's um do uh, if you will, a little bit of uh reinforcement learning 101 to make this uh really interesting to like a broad uh group of people listening to to this. So in very uh simple terms, like explain it to me like I'm 10. Uh, what is reinforcement learning?

>> Yeah. Yeah. I I usually the the metaphor and the analogy I have to reinforcement learning is like training a dog. It's it's very, very close. And I used to have a dog when I was a teenager. And even even what I remember my parents did, I didn't know anything about raising a dog. What they what they they kind of invited through some friend of a friend, a fireman who I think was working with like service dogs. And he came to me and he basically told me a little bit about how do you train your dog. And what most dog owners that are like ambitious about training your dogs know, it is always extremely important to have a bag of treats in your in your pocket. That's that's what you always do. And whenever you see your dog behave well, what you should be doing, you should you should smile and you should give your dog a treat. Whenever you see your dog do something bad, you basically like give your attention away, turn away and become sad. And before the years of breeding, the dogs discover it's a it's it's a like, you know, bad reward and bad bad behavior. And this is exactly doing that, but with models. We elicit a lot of different behaviors in a mall, put them in challenging situations, and then we give them cookie if they do something we want. If they do a good thing, and give them some kind of punishment and negative reward if they do something like that we that we don't want and that we don't like, in a good way. The the good way to do RL is if you balance those things. So if you kind of give cookies half of the time and punish the other half the time. But this is almost like a mathematical kind of uh kind of aspect of it. But that's that's that's the most important part, which is like elicit behaviors, reward the good ones, and then going forward, the model will be most more likely to do what you want and less likely to do what you not want. And through that, it it improves. It is the the way uh how to train models to like elicit actual behavior that is not first. It's next to prediction. If you if you pre-train the model, you literally train the model to to predict the next token. RL is like a completely different gradient, a completely different set of what we what we want to get out of them all.

>> And uh getting the model to do what you want, just for some vocabulary and semantics that's uh you hear sometimes the term policy. So in RL, you hear terms like agent, environment, action, reward, and policy. So I think a lot of those are sort of um self-explaining, but policy is what that's a strategy, that's a behavior of the model.

>> Yeah. Policy is the behavior of the model. As the model weights represent what it does when when put in a different thing. Like model in the end is a mathematical object, and you can define it. And policy is a mathematical function that maps observations to actions, you what what you see, and then what you what you do with with what you see.

>> Yeah. Uh, so agent is a model, action is what the model does. Reward is how you um say whether that's good or bad. Environment, you you hear a lot of things um these days about designing the right environment for RL. What what does that mean?

>> Environment is like in some way, it is everything that the model sees. But but the interesting thing about difference about RL environments and most other types of like what you can call supervised learning or unsupervised learning is that reinforcement learning environments, you want them to be interactive. You want like them to evolve as the model does things. In general, like like similarly how like if you want to like learn how to play guitar, you kind of take a guitar and you strum it, and what happens is you hear sound of that, and then you hear it, and then you can do like learn to play like like with with actual like feedback of what is happening with the the guitar. And it's like way, you know, they Environment is like how does the world reacts to your actions? And a lot of like what drives your actions is the is what is happening in your in your environment and it's in in your in your world. And that's kind of like the only way how to like really teach agents to like learn to react to changes in the environment is through is through reinforcement learning.

Can >> can you give us a little bit of a bird's eye view of the evolution of RL over the years? What uh, you know, mostly how does modern RL differ from sort of historical RL?

>> Yeah. Yeah. Yeah. I like again, there was super historical RL, like so not not even that old, but the the main like tectonic shift was when combining neural networks with reinforcement learning. The reinforcement learning, you know, predates neural network as a general mathematical method of optimizing behaviors in like mathematically defined environment and as a method of study.

>> That's what is known as deep reinforcement learning. Is that right?

>> Yes. Then then the deep reinforcement learning, that that basically like DeepMind uh like invention of combining neural networks with uh with reinforcement learning to the the DQN moment I I talked to you about. And then from there, like there was a moment where like there was a pretty active like area of research of reinforcement learning on games. Like when even when I started, like 20 2019, there reinforcement learning was was kind of fashionable at that moment, although not like very successful. But we were the reinforcement learning was able to solve a lot of games. But the bottleneck was there that the models were not pre-trained in any way. We were we're training a lot of behaviors playing games. Like we even got AlphaGo moment out of out of that, which a lot of people got very excited about it. But was still like learning behaviors without the models that were that were meaningfully smart about those behaviors. There were there's still a lot of kind of like, you know, you don't want to call it caveman intelligence, but but something some some something in that regard about malls not being really smart, even though being being pretty heavily reinforced. And there there was like a long research in that and a lot of cool like results and theoretical understanding of RL comes from those days because because people were researching RL actively. But it was like in some way, it was a dead end of doing RL without pre-training. And then in in my like moment, when I when I finished like working on robotics, I started working on teaching language models to code. But having having pre-trained models was uh was was a really big deal. And and and and the the GPT era of scaling and of like large scale ingesting lots of data to to to really train great models like enabled us already at that moment to to start RL. And that was that was one of the first things I did. Almost immediately whenever GPT3 was trained, I tried to do RL on it. And there there were always what were were bottlenecks. The systems were kind of clunky. It was hard to figure out what are the right RL algorithms, what what are what are the right like like uh what what are the right problems to work on it, and what is the right algorithm to train it on. And like the what what what OpenAI did at that moment, and what what's kind of like how how research goes, we kind of cargo culted a lot of things that was used for games and almost the same things are for robotics. And at the first RL I was doing on large mall was was kind of the same ppl we used we for everything. And like it gave some results, but those like early results weren't completely mind-blowing in RL. And there there was a long time where we're keeping on investing in it. And you know, personally, I always believed there will be a really, really big moment for RL and language models. But the defin the finally, the early early trials and errors weren't weren't super successful at the moment when we uh when we train GPT4 and there were there was that there was an interesting moment where we trained GPT4 and everyone today thinks, oh, GPT4 is such a great model, but

When we trained GPT-4, we were pretty underwhelmed internally. And then there were a lot of moments like, "Oh, we trained this model, we spent a lot of money on it, and it's kind of like, you know, pretty dumb." At least, at least like, you know, we have GPT-3. GPT-3 already does all that stuff, and GPT-4 doesn't really seem to be that much better. And like, we had this, this like, kind of, kind of question like, you know, how, how like, it kind of seemed smart on evils that were, that were like, one token long. It seemed to be able to give like, pretty detailed answers to complex questions where it was one token. But if you actually let it speak for longer, it wasn't very coherent or, or, or really like, give, gave very long answers.

And we, we, we need to ask this, answer this question: How do we like, actually make the language model that seems to have some smartness in it, ways, actually sound smart and actually be good in, in talking to it? And that was, that was the moment where a technique that was developed already a few years earlier, like, really shone, which was called RLHF, which is basically doing RL on large language models with the reward given from human preferences of seeing two parts of the text. >> Thumbs up and thumbs down. >> Yeah, thumbs up, thumbs down, like, whatever, whatever human preferences is. That, that's a very good reward because there are a lot of things how the model can generate bad text, and how early GPT-4 was generating bad text in a lot of ways. And RLHF was able to catch those things and correct it, and, you know, reinforce good behaviors, reinforce generating good text, and then, and then punishing bad text. And in the end, GPT-4 plus RLHF together as a package, like, you know, delivered the GPT moment to the world that everyone sees. And as much as it is a big success of pre-training, it actually also was a pretty big success of, of RL in the form of, in the form of RLHF.

>> Amazing. And, and just to, uh, double-click on that, the RLHF. So, we, we all are familiar as users with, I mentioned thumbs up and thumbs down, that's on the interface. But, um, the actual RLHF happened post-training. Is that, is that, is that, is that right?

Yes. And, and what did that look like as an effort? Did you have like, a bunch of, uh, just humans sitting down, uh, you know, in front of the model, industry specialists maybe, and give it feedback? How, how did that actually work?

Uh, like, so, like, RLHF was a research program that was, was already like, you know, was already happening in the background for a while. I think we did RLHF at least, I remember GPT-2 being RLHF for quite a bit, and like, that was, that was already there and already, already happening. It's like, gathering data for even for RLHF is its own like, research domain, basically. And always thinking, what is the right data to train the model? What is the right data to train, like, your rewards? And how, how to shape your rewards? It's, it's a research that we've been doing, and it's, it's a very open-ended and very deep in many, in many different ways. And like, what, like, I think there are papers written on, on what, what RLHF is, but there, there's a lot of depth to it. But like, you know, the long story short is, you have what we call AI trainers these days, and they look at outputs of the models, and they give them scores. And then you learn basically a model of those scores and use that for training.

>> And that's part of, for people who may be curious, like, the, the entire, uh, data labeling industry, so, uh, Scale AI and, and a bunch of others, that's, uh, what they do, right?

Yes, yes. Like, I think, like, in, in a way, I think it's getting more and more to be a thing of the past as the models are getting smarter and smarter. This is becoming less of a thing. But I think a few years back, and especially in GPT-4 days, this was the thing. The, the, the interesting bit about the data labeling industry, I'm not sure how much you want to go on that tangent, is that it has to constantly reinvent itself because the AIs are getting smarter. At some moment, like, certain things you don't want to label with humans, AI already can do it. So, so like, you need to, you move the frontier and, and you change the data type of data you are labeling as the, as, as you already are the previous part.

>> We've been talking about RL, but the, the first phase of all of this is the, the creation, the pre-training of the models. That is unsupervised learning, right? Do you, do you want, maybe for, again, to make this broadly, um, interesting to, to people, define unsupervised versus supervised, and, and in what way was, uh, the pre-training unsupervised versus self-supervised, or whatever nuance?

Yeah. Yeah. I think those are nuances, and I don't think they are as, as stark and as sharp as some people like to, to determine them. But the, the, why, like, pre-training is called unsupervised, because like, in some definition of it, you don't need any, like, extra labels to the data that, that you feed into the model. You just, just feed the text as is. In some way, you may, you may argue that, that the data is already labeled because it is self-labeled. If, like, you know, you give the model from the text, predict the next part of text, like, you know, in some way, it is a label. But it is self-supervised because, like, we don't clearly tell the model, like, what is right or what is wrong, or what is, what do we want from it, or what do we not want. We want it to just predict the other part of data. You can do the same thing with images. You can mask a part of the image and tell the model, like, predict the next, the next bit of image.

But whenever, whenever, like, there is this, like, classic machine learning notion of, like, targets and labels, like, I guess we're talking about classifiers. So, supervised learning was like, you have some notion of targets, what your targets are, and some notion of labels. Supervised learning was like, predict those labels from targets, and this is like, a, like, some type of mapping. But actually, what's, what's interesting is that there are many more bits usually in the targets than in the labels. And studying the structure of targets itself, it yields much more learning and much more intelligence than learning the mapping itself. So, uh, like, spending a whole compute on just, just learning the data itself without the labels is, is the right thing to do. And like, what, what is often called, like, representation learning, and studying, studying the data and its properties.

>> Okay. Great. All right. So, going back to, uh, RL. You tweeted the other day, GRP-O, the GRP-O release has been, uh, in a large way, has accelerated the research learning, uh, program of most US research labs. So, what, what is GRP-O?

Uh, yeah, it was a little bit of a tongue-in-cheek moment. And I am extrapolating here a little bit what exactly happens because I, I haven't been in most US research labs, but I have some mental model of, of what happened. And long story short, GRP-O was the open-source release from DeepSeek. And there was a, like, everyone who, like, is terminally online and follows AI discourse knows that DeepSeek moment of, whatever it was, when the, when the Chinese company that seems to be doing really, really great work released a new model. And it was a, also a pre-trained model, a reasoning model. They open-sourced the algorithm. They open-sourced a lot of things they did. Overall, like, really, really great and technically excellent release. And like, there was a lot of discourse about, about, uh, like, you know, that they, they pre-trained their model particularly cheaply. And that was, that was part of the discussion about the DeepSeek moment. But the other part of the discussion was that they, that they kind of like, released their reasoning process. It was like, not very far after our 01 release. And as far as I know, like, our 01 release mostly caught a lot of US labs by surprise. They didn't have like, similarly advanced RL research programs, to my knowledge. Basically, no one. And like, and I think, like, the only company in the world that they get, I am, as I am aware, there's probably a lot of things I, I didn't know. But you talk to people sometimes, you hear rumors. So this is my, my version of the world is that if you look at the other papers of DeepSeek, like, that company was doing pretty similar in some ways, RL research to what we are doing. And I, you know, I think I have to clarify, like, what we, what, what OpenAI is doing is not exactly GRP-O. It is slightly different in many different ways. But like, some parts are, are definitely, are definitely similar. And what's, what's most important, those are both like, large-scale policy gradient algorithms. And like, DeepSeek was doing, company was doing research in a slightly adjacent area. They were, they were, they were not very far. And when we, whenever we released 01, and we told the world that you can get pretty, like, those great results with scaling up reinforcement learning on language models, I think it was like, not very big hope for, for, for the DeepSeek company to kind of realize, okay, we are not very far from getting similarly good results. And they, they did it. They trained their reasoning model, and they released it, and they told the world how pretty, like, not very, not much later than we released 01. And I think, like, for a lot of US research labs that didn't, like, yet know, didn't have a research program, how to train, like, reasoning models, they looked, oh, there's this, this, like, Chinese company, they released how to do it. It helped us kickstart and train reasoning models much faster than we would have to otherwise, if we would have to find all those bits ourselves.

>> What does it take to scale, uh, RL? So, if there was a phase where OpenAI was very focused on pre-training, and then, if I understand correctly, the last, uh, 12, 18 months, or whatever time period, where the emphasis has been on the sort of second part of the plan, which is scaling RL. Is, is that a question of just giving RL more compute, more data, more labeling, as we were saying? What, what does it take?

The, the first thing that is important to know and understand, RL is hard, like, conceptually, if you think about it, and there's still a lot of depth to it. But, but conceptually, mathematically speaking, pre-training is dead simple. It is the kind of the simplest thing you can do. And there has been, there has been a lot of, a lot of thought and a lot of optimization already put through that, through a few years of, even of, of optimizing and doing very well at very large scale, very simple mathematical operations. In terms of, in terms of RL, it is much, much more complex. There, there, there's many more things going on in a, in a reinforcement learning run. There's many more things that can go wrong in, in doing it, especially as you, as you scale up, and many more types of bottlenecks, failures, h, like, it's, it's, it's a much more, much more delicate thing. And there's much more, uh, like, much, much more room for error. And in some way, like, you know, I don't want to go too deeply in the parallel because it's a little bit overblown. But in some way, like, you know, just, just to give some coincidence, you can have a steel factory which makes steel, steel, and like, the process is relatively standardized, and you make blocks of steel, and they are, they are uniform and nice and well-defined what it is, versus like, building semiconductors, which there are, there are very, very few companies in the world that can do it because there are so many things that can go wrong, and you have to put a lot of attention to details to make great semiconductors, and it's very, very, like, complex internally. And like, you know, in many, in many ways, this is, this is kind of like, you know, I don't want, don't want to diminish because there's a lot of very hard technical difficulty to do pre-training well at the, at a large scale. But there's just, just many more moving pieces and many more, uh, many more elements of the, of the reinforcement learning stack that's, that that need to, need to get right, be get right to, to get a large-scale run successful.

>> You, you mentioned, um, working on, so agent-like, the agentic AI, like, what, where does that all fit? The, you know, the, the tool use, like, the whole, agentic autonomy versus reasoning, RL, help us to, of reconcile, what does what, and what impacts what?

What's, I think, what is the important thing is, I, I believe, and I think that there can be a lot of positive impact of AI on our world and our on our lives through automation, through problem-solving, and through AI doing good things for us, the things, the things that we want. And for a long time, and for a long time, again, it's not, not that long, but the last two years or so, or maybe approaching three, we've been like living in this world where we kind of ask questions to AI, and it gave us an answer. For, at the beginning, instantly. Now it can think for like, a minute or two, which, which feels long, but in many ways, like, what's, what can you do for two minutes? If you think of like, how many problems you must solve, and AI is probably a little bit faster in the things it can, it can solve, but it's still a limit of what it's, what it can do. There are still a lot of tasks that, you know, that would take AI to do much, much longer. When, when I prompt Codex, it works for a while, again, few minutes. Like, there are a lot of, like, things we have internally, and we are doing that, like, allow the model to work for much longer. We still didn't figure out the right product to deploy them. But the models, like, can think for like, 30 minutes, an hour, two hours, these days, on certain, certain types of tasks and problems, like, even, even longer than that. And they, they, they generally are, are capable of doing so. And we need to figure out how to make that process more useful and more being able to actually come to various problems in real life, whatever it is, coding, or booking travel, or making plans, or, or, or, or even designing houses, or new electronic devices, or whatever, whatever else we would like models to do, we would like them eventually to, to be able to do for us. Um, like, a lot of this comes through the models thinking independently for longer periods of time, and considering more of our alternatives, bits, and just, just sometimes going through a slog of very long lists of tasks.

>> So the agentic part is, is, is powered by fundamental reasoning. Is there, is there a concept of, um, uh, I guess, online RL that, that happens where, as the agent does something and learns from the real world, the RL happens in, in real time?

So, in general, like, all, all of RL is happening, like, most of RL that, that you hear talk to, uh, language models, is online, but it's done online in a way that is still a training run. It's still being trained, like, kind of separately from the user. There have been a few models in the world, and I've, I've learned recently that I think, I think Cursor is trying to train some models online with their, with their users in the loop. And it's theoretically possible to train models in GPT or every other product, just, just responding to the users and reinforce through whatever, whatever rewards you get in there. But this is not what I am aware, at least, like, not, not what OpenAI is doing at the moment. And it is, it's can be great, but it can be also dangerous because you are not really very much controlling what you are reinforcing in that loop, and what's, what could, what could happen. So, yeah, at least until, until we have a really good safeguards, I don't think, I don't think we should try to do that in anything like, as, as, as complex and large-scale as just GPT.

>> Yeah, interesting. And very much on, on that note, um, talking about alignment for, for a minute, is alignment an RL, uh, thing? I mean, is it, do, do you create alignment in the model by teaching it what, what is right and wrong?

Kind of, kind of, yeah. It's a little bit of yes and a little bit of no. Like, in a way, alignment is about steering the model to certain behaviors, and that is, that is definitely an RL thing and an RL problem. But also, you want the model to know what is right and what is wrong, and understand the world. And not, not all of them are, are like reasoning and, and RL problems. Those are very often, like, just AI problems. And we, like, in, in the A to B, aligned, the model needs to know right or wrong to choose right. I don't think, like, you can just tell the model, like, few, show it few good things to do, and it will do them all. It needs to deeply understand its actions and consequences to really truly be able to choose the right thing. And it's a, I think it's a never-ending pursuit because, like, even, even for humans, it's not super easy to, uh, to define what's, what do we consider aligned. I think as our, as our civilization will evolve, it will, the notion of, of alignment and, and the goal of, of humanity will keep evolving, and we'll need to keep nudging the model towards those things and keep, keep explaining to it the things that we, that we want from it. Uh, but it's a, it's a very definitely very important and, and, and central part of, part of any, should be of any AI research program.

>> Yeah, which brings a whole, um, you know, next series of, of questions of where RL is efficient versus less. So, it seems that it's been particularly good for, like, math and coding. And then the next obvious question is, what about the rest of the world? But, like, taking a, a quick, um, sort of going down the rabbit hole a little bit about math. So, just in September, like, just a few weeks ago, uh, you guys did something, uh, unbelievable with the ICPC World Finals. Do you, do you want to talk about, uh, what that was and, what went on from a model technical perspective behind the scenes?

There happens surprisingly little from our perspective, from the model perspective. We just, we just have a pretty smart model. And then when we ask them to solve programming problems, we, they, they, they are, they are correct. Uh, like, what's, what's a little bit of a backstory in it is that I think we used like, specifically programming puzzles for a while as a very nice research testbed of our, of our ideas. Those are, those are nice problems to experiment on them. And they, they weren't ever like, considered part of the product. But it's, it's those are pretty like, complex problems, and they require a whole bunch of thinking, are very nice to give rewards to it. So a lot of researchers just, just liked working on those problems as a way of trying out their their ideas. Like, you always need a dataset. I, I'll take a dataset of programming puzzles and try it. And I, I think like, because of that, a little bit, our models just were always very, very good at competitive programming as a kind of byproduct. We never, we never tried to be good at it, but like, researchers were trying their ideas on it. And because of it, like, every, every training run, whatever, whatever we are doing, just, just end up being very, very good at, at, at those, those type of puzzles. And then, like, you know, it was, was a little bit of a, of a formality for us to kind of like, you know, to, to go and submit that to, to a competition and, and then, like, you know, do, do it's like, largely about demonstrating to the world how, like, what, what is the level of capability in, in those models. But I think it is important and true to acknowledge that in not all domains, at least, like, comparing to human baseline at this moment, we can be, we can be as nice and as good as, in, as, as, in programming competition problems in many, in many ways, because, uh, those, those were like, you know, tried and tried for, for, for a long time by, by, by, like, you know, many, many researchers. And researchers don't always spend as much time as, as they could, and I would, I would like them to, on, on very practical problems that people go to, go to GPT or our models with.

>> Great. So it sort of came out of the box. There was no specific training for, for it. Um, and just to remind people, so what I'm referring to, what we were discussing, is the ICPC World Finals that was just in September 2025, which is International Collegiate Programming Contest, ICPC, that happened in Baku, Azerbaijan, where, where OpenAI solved 12 complex algorithmic problems within a 5-hour time limit, basically making it take effect the equivalent of first place, um, in front of human teams.

Just four, four competitions. We, we did a little bit of a, like, a round tour of various competitions. We did ICPC, we did also like II International Olympiad in Informatics earlier this year, and, uh, Codeforces competition as well, where we went second, be behind a single human that is also a Polish person that used to be employed by OpenAI some time ago. A funny, funny coincidence. But like, I think like, we, we were looking for a moment in time where we are, where our models are kind of like smart enough to be able to compete in those competitions with, with, with some of incredibly smart and talented humans. But it wasn't never our particular goal and focus. It's kind of like, we think if we are doing good research for training smart models, they should be smart enough to do those things. And we kind of like take that milestone and now we keep on moving forward. And I hope we will see, and I think we are already seeing more and more practical and tangible things coming out, like every week or every other week on Twitter. I do see what I think are credible reports of actual scientists, uh, using some of our reasoning models to help them perform calculations, solve hard technical problems with, with our models. And I think this is like, where we want to be. Like, solving competitions is cool, but people solve competitions to prove that they can work, go to the actual, like, frontier-level job and solve new technical problems. And this is kind of what we want from our models as well.

>> We, we alluded to this a second ago. So, um, mentally, at least for somebody like me, I, I understand, uh, how, uh, RL could be used very effectively to train against math problems or coding problems. I think one of the big questions right now is, how do you do that for the rest of the world in, context and disciplines where the answer is not right or wrong, maybe a little more, more murky, and the rest of the, the rest of the economy? Like, you, you guys as an organization came up with GDP Val the other day, which is a way of, um, evaluating performance against different industries. What, what is your thinking in terms of generalization of RL as, as a, as a path to success for the rest of the world?

If I think, like, the, the short and quick answer is, like, somehow humans can learn all those things. And as long as there is any way to evaluate performance and figure out if something is going right or wrong, and you can compute that feedback, like, you need to be able to, to somehow calculate how well something, then you can, then you can optimize it, and then you can do, you can do reinforcement learning with it. I think, like, you know, there, there, there can be an argument that if there is no notion of what is right or what is wrong, then also, like, humans are not able to, to, to improve and learn, because there, there, there needs to be a learning signal that, that coming somewhere. There, there's mostly a question of, like, how convenient and how easy is it to get that feedback. And everyone doing reinforcement learning should strive and, and try to be able to train on, on, on more and more complex and interesting training signals to do. Um, like, very often, there comes a notion of what, what is often called reward hacking. And like, what's, what happens when doing reinforcement learning? It happens a lot, and it's a, it's an important problem. You shape your reward in some way to reward certain behaviors. But sometimes it is the case that's what you reward is not what actually you want. There, there is, like, one thing you, you need to train the model to do the behaviors, reward. But there, there's also a natural, like, mismatch about, about the reward that you give the model and what's, what, what you actually want. And there are sometimes moments where, where, where the model does what you, what you reward, but, but it's, it's not, not in the spirit of what, of what you had wanted. And we need to, you need to fix it. It's almost like a, like a parenting challenge. And like, you can, in some way, you can say it's a, it's a limitation of reinforcement learning. But when I, when I was thinking about it, I realized a lot of that happens in human systems as well. There are a lot of, like, incentive systems and reward systems, and even, even happens in workplaces, in all kind of human groups, that humans have rewards that are not always optimized for the, for the ultimate goals of the system, and they, they hack rewards constantly in many different ways. And there is a constant whack-a-mole game between between setting the right rewards and seeing if the system does it. And it's a huge, like, issue in any policy making, almost, and any incentives programs. And this is the same, same kind of, like, whack-a-mole game in, in reinforcement learning research, trying to make sure your rewards are better and better representing what you actually care about the model to be doing.

>> All right. So maybe to, uh, zoom out, to, to, to close this conversation. Uh, you said the other day, um, you tweeted, "We all collectively believe AGI should have been built yesterday." And the fact that it hasn't yet is mostly because of a simple mistake that needs to be fixed, which is super awesome as a, as a tweet. Uh, do you think that the combination of, uh, pre-training and scaled RL takes us to AGI?

Like, there, there's always an, an interesting question of, like, what, what do we consider something that is not pre-training, and like, where, where is the limit? I generally think something that we are doing, like, pre-training today, is necessary. I think something that, like, we are doing RL today is necessary. And there will surely be a few things more. And like, we have a lot of very ambitious research programs on some of those things. And like, I, I, I didn't think, like, the question of distance in research space is hard to say. Like, what, some for some people, like, what we, what we want to do and what we are planning to build is not very far from those things. For someone will say, oh, it's completely different, and it's, like, a very much not that. So I don't want to go into, into, like, debates whether it's the same or not. But like, we are, and want to be, constantly changing the way how we train the models tomorrow, represent what we think the, the right form of, of intelligence is, and the most useful, useful form of, form of learning is, and constantly are, are researching various things. And know, the distance from, like, what is, what is the distance from AGI is also, like, a very complex question. Like, I, I really, like, someone said it to me, but I think it is right, that you, if you talk to someone from 10 years ago and show them ChatGPT from today, they would probably call it AGI. But we are not today because it still has a lot of limitations, and we are, we are all very aware of those limitations, and we are pretty sure we can resolve those limitations. There will be probably some further limitations of the future models that, that will need to be fixed. There is an ultimate question which is very hard to answer, when, when is the moment that the model can, like, improve itself without that much, like, external output, and without humans working on it and fixing it. And I think, I think it's a, it is a very hard question. It is a serious question that, like, you know, we need to try to answer. Humanity needs to, need to try tries to answer, because, because, like, you know, most that will, will still, like, largely depend on, on, like, our infrastructure, in our systems, but we will be able to start, start fixing itself without, without us having to fix it. And that's, that's like, the, the predictions of what really AI will be able to do, and will be able to solve at that moment, start becoming, like, a little bit murkier than what we can do right now, which I think we still, we still can do that pretty, pretty well.

>> But philosophically, you, um, may have heard Richard Sutton, you know, the other day on the Dwarkish podcast, which is a, which is a wonderful episode that people should really listen to, effectively saying that, um, the only path to AGI was going to be pure RL, and that fundamentally LLMs, and maybe, I hope I'm characterizing what he said appropriately, but that LLMs were a flawed premise because effectively that was imitation of reality, whereas RL was enforcement of reality. Do, do you have any thoughts, sort of, like, philosophically on, on that question?

Yeah. Yeah. Yeah. I, I haven't had a chance to, to, to fully listen to that episode yet. So I also don't get all the details of that, uh, of, of that thought. But what I can say is that we are doing quite serious RL on language models these days. And like, I don't, like, in terms of a pure RL, I don't think, like, really pure RL makes sense. RL needs pre-training to be successful. And I think pre-training, as, as I said before, needs RL to be, to be successful as well. I, I don't think without RL, it would make sense, the research program we are, we are doing. But, but we are, like, OpenAI is, and I'm pretty sure all other, all other AI labs as well, very serious about about doing a lot of reinforcement learning on our models. And I think, like, I, what, what kind of, like, a lot of people are saying that, that whether LLMs are an on-ramp or off-ramp of AGI, very often they, they do mean like pre-training. But it's also clear that, that, like, you know, the current way how we are doing things, like, also, it's not yet enough, and it's not yet everything, and there will need to be a further, further changes to, to the setup. But sometimes, like, people say, oh, if you are, if you are doing RL, it's not LLM, it's something else. Sometimes say, oh, if you can write a program in your, in your rollout, and it's a chain of thought, it's not in this neural network only, it's a neural-symbolic system. So it's like, you know, it's easy to get, like, some people consider something LLM and the other thing not. Uh, but, but, like, personally, my view is, what we have is a pretty good foundation for the next step of, like, we did F transformers first, trained for trans, for translation, then we were, then we were pre-training them on large-scale data, then we were doing RLHF on them, now we are doing large-scale reinforcement learning. We'll do a few more, uh, more and more complex things. There is a chance somewhere along the line, the architecture will start changing more or less significantly. And, and, you know, I think, I personally think we are on the, on the right path. And it will feel less like, uh, complete, completely turning around, and more like, more like keep on adding more things, and maybe, like, you know, dissolving some, some old elements that didn't, that are not, that, that carried us to that, the particular level of intelligence, and we're not needed anymore.

>> Well, that, that feels like a wonderful place to leave it. You've been very, uh, generous with your time and thoughts and giving us a glimpse, uh, into, uh, OpenAI, what you work on, what it looks like behind the scenes, and the key aspects of pre-training and scaling reinforcement learning. So, it's been a wonderful conversation. Jerry, thank you so much. Really appreciate it.

Thank you very much. I enjoyed being here a lot too.

>> Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks, and see you at the next episode.