Transcription
Back in 2021, I had coffee with Il. He was asking me about my AGI timelines, and I told him, "To be honest, I think it's going to take a very long time." I explained, "Look, we're not going to get to superintelligence until we can figure out how to scale inference for this computer in a very general way, and I think that that's an extremely hard research problem." So, I thought it would take at least a decade. It took like 2 or 3 years. I have no doubt that there are others; in fact, I know that there are other research questions that aren't solved, but I don't think that any of them are going to be harder than the problems that we've already solved.
Noam Brown is a research scientist at OpenAI, where he was a key part of their work on AGI. He is at the forefront of reasoning in LLMs and has a really interesting past track record at FAIR, where he worked on problems in diplomacy and poker. We hit on the biggest questions in LLMs today, on unsupervised learning. We talked about whether these models are hitting a wall, how far test time compute can scale, how Noam defines AGI, and what he's changed his mind on in the last few years in AI research. This was a really fun one to do right after the general release of AGI, and I think folks are really going to enjoy it.
Without further ado, here's Noam.
"Noam, thanks so much for coming on the podcast."
"Of course! Great to be here. I've been looking forward to this for a while, and certainly well-timed with some exciting launches going on with Ship Miss."
"Yeah, I'm looking forward to it. We're going to be releasing AGI tomorrow, which I guess by the time this podcast is out, it's already going to be out there. I'm pretty excited for it; I think the community is going to love it, but I guess we'll see."
"I'd be remiss not to start around what I feel like has been the question of the past month: have we hit a wall with model capabilities? There are obviously different parts of that question, so maybe to start, would just be the extent to which you feel like there's still more juice to squeeze on scaling pre-training."
"My view on this, and I've been pretty public about this, is that I think there's more room to push across the board, and that includes pre-training. I think the right way to think about it is that every time you want to scale these models further, there's a cost to that. You look at GPT-2; it costs like between $5,000 to $50,000, depending on how you measure it. You look at GPT-4; obviously, there's a lot of improvement, but the fundamental thing, the most important thing that's changed, is the amount of resources that have gone into it. You go from spending for frontier models like thousands to tens of thousands of dollars to hundreds of thousands to millions to tens of millions, and for some labs, possibly hundreds of millions of dollars today. The models keep getting better, and I think that will continue to be true. If you throw more money, more resources, more data, all this stuff into that, you're going to get a better model. The problem is that, okay, if you want to 10x it each time, then at some point that becomes an intractable cost. If you want to make it better, and you want to do another 10x, now you're talking about billions of dollars, and if you want to do another 10x, you're talking tens of billions of dollars. At some point, it's no longer economically worth it to push that further. So, you're not going to spend, presumably, trillions of dollars on a model. There's no hard wall; it's more of a soft wall that eventually the economics just don't work out for it."
"And it seems like, obviously, there's, you know, in many ways, you're able to push this forward with test time compute, and there's lower-hanging fruit there from a cost perspective to push that forward."
"Exactly. This is why I'm really excited about test time compute. A lot of people are excited about it because we're still in a state where it's kind of like we're back in the GPT-2 days. When GPT-2 was figured out and the scaling laws were figured out, it was pretty obvious that, oh, you just scale this up by 1,000x, and you're going to get a better model. You could do that. It's a little harder now to scale things up by 1,000x with pre-training, but with test time compute, we're still pretty early, and we have a lot of room, a lot of runway to scale that further. There's a lot more low-hanging fruit for algorithmic improvements, so I think there's just a lot of exciting stuff to be done in that direction. That's not to say that pre-training is done, but it's just that there's so much more headway to push the test time compute paradigm further."
"I should also say, it's not like even going back to pre-training for a second, it's not like there are two orders of magnitude or something that you can push in and then you're done. You're still going to be Moore's law; I think costs are going to continue to come down. It's just a question of how quickly can you scale it. There was this huge overhang where it was very easy to scale it very quickly, and that's becoming a little bit less true."
"I realize this is probably an overly broad question, but how high is the ceiling on test time compute? How do you think about where that could go?"
"Again, I think about it in terms of dollar value. So, you know, how much does a GPT query cost today? Ballpark, a penny. What cost could you spend on a query that you care a lot about, and what cost would you be willing to pay? I think there are some problems out there that people would be willing to pay a lot of money for, and I'm not talking about like a dollar or $5; I'm talking like a million dollars for some of the most important problems that society cares about. So, how many orders of magnitude is that? That's about eight orders of magnitude. I think there's a lot of room to push it further, and I also think there's a lot of room for algorithmic improvements. So, it's not just like, oh, we're just going to dump more money into the query and then you get a better output. It's like, no, we can actually improve this paradigm further and make this scaling a lot better."
"One thing I thought was interesting is, I guess maybe a month ago, Sam Altman had tweeted, 'We basically know what we've got to do to build AGI,' and I think you tweeted that his view matches the median view of OpenAI researchers today. Can you say more about that? Because obviously, there are so many people now talking like, oh, we've hit a wall. What do you think they're missing?"
"I feel like we've been pretty open about this, that we see things continue to progress pretty rapidly. That's my opinion. I think that Sam expressed his opinion. I've heard some people say that, oh, Sam is just trying to create hype or something, and I'm kind of surprised by that because we're saying the same thing. Yeah, I think it's a common opinion in the company that things are going to progress quickly."
"Do you think pre-training and test time compute alone kind of get you most of the way there, or is there also, it seems like, this algorithmic bucket as well?"
"It's not by any means that we've cracked the code to superintelligence. That would be pretty cool if you came on the pod and announced that you had, though. But I think, okay, so the way that I think about it, back in late 2021, I had coffee with Ilia Sutskever, and he was asking me about my AGI timelines. I told him, 'To be honest, I think it's going to take a very long time. I'm pretty skeptical that we'll get there within the next 10 years.' The main reason I gave him was that we don't have a general way of scaling inference compute, scaling test time compute. I saw how much of a difference that made when it came to games, and the fact that it wasn't there in language models in a very general way just seemed kind of silly to me that we're going to get to superintelligence just by scaling pre-training. You look at these models, and yeah, they're doing pretty smart things, but also, back then, they couldn't even draw a Tic Tac Toe board. Yes, you get to GPT-4, and suddenly they can draw the board and make mostly legal moves, but sometimes they still make illegal moves; they make suboptimal decisions in Tic Tac Toe. I have no doubt that if we scale pre-training another order of magnitude or two, it's going to start playing Tic Tac Toe really well. But if that's the state of things, that we're spending tens of billions of dollars to train a model, and it can barely play Tic Tac Toe, that's pretty far from superintelligence. So, I told him, 'Look, we're not going to get to superintelligence until we can figure out how to scale inference compute in a very general way, and I think that that's an extremely hard research problem.' I thought it would take at least a decade. To my surprise, by the way, he agreed with me. He agreed that scaling pre-training alone would not get to superintelligence. I didn't realize it at the time, but he was also thinking very carefully about this scaling test time compute direction. So, I thought it would take at least a decade; it took like two or three years. I thought that was the hardest unsolved research question at the time. I have no doubt that there are others, and in fact, I know that there are other problems that aren't solved, research questions that aren't solved, but I don't think that any of them are going to be harder than the problems that we've already solved."
"Yeah, and for that reason, I think that things will continue to progress. Obviously, you've had just a massive impact in this test time compute work, and your research career had obviously been in search and planning, you know, games like poker and diplomacy. From others' accounts, it seems like when you joined OpenAI, you were pretty clear that this was the direction to push in. It seems to have really paid off. I'm curious how consensus was that approach when you joined. Maybe talk about getting the kind of research organization oriented behind that."
"It's interesting. When I went on the job market and I was interviewing in a bunch of places, people in general were quite receptive to the idea that the research labs, I think everybody, actually for the most part, among the frontier research labs, believed that pre-training alone would not get us to superintelligence and that there was something else that was needed. So, there was a lot of reception to this idea of, okay, yeah, maybe we need to figure out how to scale test time compute. Some labs were more bought into it than others, and I was actually kind of surprised that OpenAI was really on board with this because they're the ones that pioneered large-scale pre-training and had scaled it farther than anybody else. But they were very on board with it, and I didn't know it at the time when I was talking with them that they had also been thinking about this for a while before I joined. When I did join, it's interesting because I think the motivation was different. The motivation that they had in mind was more about overcoming the data wall, not so much about we need to figure out how to scale test time compute. It was more about how do we get over the data wall. But the techniques ended up being, you know, the agendas ended up being pretty compatible. So, it actually wasn't too hard to get, you know, look, when we started, it was still this exploratory research direction, and there were some people working on it, but it wasn't like half the company was dedicated towards this large-scale effort by any means. But a few months after I joined, we were trying things, many of which didn't work, but there was one thing that one person tried that ended up getting some signs of life, and people were like, 'Oh, that seems interesting; maybe we should try some more things and get more and more signs of life.' Eventually, I think the leadership recognized that there was actually something here that seemed different and valuable, and we should really scale this up. I was supportive of that, but I think others were, and I think it was a testament to OpenAI and its organizational excellence that it was able to recognize that there was a lot of potential here and was willing to invest a lot to scale it up."
"I mean, I think people really underappreciate the point that, in many ways, it's really surprising that something like AGI came out of OpenAI. It's disruptive to the paradigm that OpenAI pioneered, and I think it's a really good sign that OpenAI is not getting trapped in the innovator's dilemma and is willing to invest in a risky direction. I think, in this case, it's going to pay off."
"Yeah, no, it's really interesting because, obviously, if the script had continued to play out of just scaling pre-training continuously and raising more money to do that, OpenAI is in a great position to do that. Any sort of orthogonal approach is different, and so it's cool that it came out of the same place. Obviously, your original timeline was, 'Hey, it's going to take 10 years to do this,' and you did it in two. What was the first thing you saw that kind of made you think, 'Yeah, okay, actually, this might be way faster than I thought?'"
"First of all, it's not just me; it was me and a lot of other people that managed to do it in a shorter period of time than I predicted. What's the first thing that I saw? I think when I joined, we had a lot of discussions about the kinds of behavior that we would like the model to do, and that included things like we want to be able to see it try different strategies to solve a problem. If a strategy isn't working out, we want to see it take a hard problem that involves many steps and break that problem down into smaller pieces that it can tackle one by one. We want to see it recognize when it's making a mistake and correct those mistakes or avoid making them in the first place. There were a lot of discussions around how do we get those individual things, and that kind of bothered me, the fact that we would even try to tackle them individually, because it just seems like, okay, ideally, we just get something that figures out all this stuff on its own. We got the initial signs of life, and then one of the things that we tried that I was a big fan of, that I advocated for, was like, why don't we just have it think for longer? When we had it think for longer, it would just do these kinds of things emergently. It wasn't like, oh, suddenly we have a one, but it was like, oh, there are indications here that it is doing things that we wanted, that we were strategizing about how to enable it to do these things, and it's just figuring out on its own that it should be doing these things. It was also clear that we could scale it a lot further. I think for me that was the big moment where we just had it think for longer, and suddenly you see a qualitative difference; you see this qualitative behavior that we thought we would have to somehow add to the model, and it figured that out on its own. Of course, the performance was better, but the performance wasn't that much better. It was really seeing the qualitative change, seeing those behaviors, that really gave me the conviction that, okay, this is going to be a big deal."
"Yeah, I think that was probably October 2023."
"Wow, and it got out pretty fast after."
"Could have been faster, I guess. How would you contextualize for our listeners today where planning in an AGI-type model is helpful and where it's like, you know, you should stick with GPT-4 or it's not as helpful? How do you expect that, I guess, to change going forward?"
"I think eventually there's a single model. Right now, we're in a state where GPT-4 is better for many things, and AGI is better for many things. Certainly, AGI is more intelligent, so if you have a very hard problem, AGI is extremely good for that. I've talked with researchers at universities; I have a friend who's a professor who loves AGI. He's a real power user because it can tackle these hard research questions that normally you would need somebody with a PhD to handle. I think for some tasks, like creative writing, might be one of them. Though actually, I'm not sure if I know that for something like creative writing, GPT-4 is better than AGI preview. I'm not sure what the comparison is for AGI. But certainly, the big benefit of GPT-4 is that you just get a faster response. So, if you just want a response immediately and it's not a very hard reasoning task, I think GPT-4 is a reasonable thing to try. But I should say that eventually, where we want to end up is like there's a single model, and you can just ask it everything. If it requires a lot of deep thinking, then it can do that. If it doesn't and it can respond immediately with a quite good response, it does that as well."
"What does the intersection of multimodal models and these models look like going forward?"
"AGI takes images as input. I think that's going to be pretty exciting. It's going to be exciting to see what people do with that. I don't see any blockers to having them be as multimodal as GPT-4 and these other models."
"One of the fascinating parts of AGI is I feel like a lot of the previous work that you had done in reasoning was built on reasoning that was kind of specific to that problem. As I understand it, Go was Monte Carlo search that maybe wasn't as relevant for poker. One of the things that is so impressive about what you've built is you scaled inference compute generally. Could you talk a little bit about what's required to do that versus maybe some of the more specific work that has been done in the past toward specific types of problems?"
"I think it requires, I mean, I can't go into details about the actual technique, but I think the important thing is that it requires maybe a change in mindset. I think when I was a PhD student and then afterwards, once I saw how much of a difference scaling test time compute made in poker, I was like, okay, this is great, but unfortunately, it only works in poker. So, how do we extend this algorithm to be able to do more and more domains? There's a question about how do you get this technique to work for both poker and Go or poker and diplomacy or something like that? We developed techniques that work in Hanabi; we developed techniques that worked in diplomacy. One of the things that I was considering doing is just trying to get this algorithm to play as many games as possible, like trying to come up with an algorithm that would work for as similar as what was done in poker but be able to work more broadly. I think the diplomacy work actually convinced me that that's kind of the wrong way to think about it. You really need to start from the end, which is like, okay, we have this extremely general domain, and language is actually a really good example of this where you have such breadth. Instead of trying to extend a technique that worked in one domain to do more and more domains and eventually do everything, we should instead start from everything and figure out some way to scale test time compute. My guess is that initially, of course, it's not going to scale very well; it's not going to be a very good technique to scale test time compute, but then can you have it scale better and better? I think that change in mindset, I mean, the diplomacy work is really what convinced me to have that change of mindset because trying to take the techniques that we developed for poker and Go and apply them to diplomacy, we managed to apply it to diplomacy with some constraints on what it could actually do, and there was a ceiling to how much you could achieve. We actually only got to strong human-level performance in diplomacy, and it was pretty clear that if we pushed that paradigm a lot further, we weren't going to get to superhuman performance. So, to actually tackle the full game of diplomacy and reach superintelligent, like superhuman performance in diplomacy, it was clear we needed something that would actually just work for pretty much anything. So, I thought, okay, let's just jump to the endpoint and try to tackle everything."
"It's so interesting. You mentioned that you expect everything to converge on a single model. I guess in what time frame, like in the medium term, do you think that we have one model that rules them all? Obviously, there are lots of folks out there building specialized models for different use cases. Do you think building your own model makes sense? I guess there are folks building legal models or healthcare models or some of these things."
"It's a good question; I get asked this a lot. I don't have a great answer for this, but one thing I have been thinking about is, you can ask AGI to multiply two large numbers, and it can do it. It'll work through the arithmetic to figure out how to carry the digits and all that and actually multiply two large numbers and tell you the answer. It doesn't make any sense for it to do that. The optimal thing it should do is just call a calculator tool or write a Python script that multiplies the two numbers, runs the script, and then tells you the output. I think that that calculator tool is like one extreme end of the spectrum of very specialized, very simple, but very fast and cheap. On the other end of the spectrum, you have something like AGI that is very general, very capable, but also pretty expensive. I think it's quite possible that you'll see a lot of things that essentially act as tools in between those two extremes and that AGI or a model like AGI can use to save itself and save the user a lot of cost."
"Yeah, it's really interesting that the tools don't end up being capability-enhancing; they're more just to not require a massive amount of compute costs to solve something that could be much more easily solved."
"Yeah, it's also entirely possible that some of these tools just do a flat-out better job than AGI. So, I think the way I think about it is kind of the same way that I would think about how a human would act. You could ask a human to do something, but maybe they're better off just using a calculator or doing some other kind of specialized machine or something."
"Well, I guess on the AGI side, any, like you mentioned kind of your friend who's a professor using it, like any other kind of unexpected use cases that you've seen in the wild or personal favorites?"
"I think one thing I'm really excited for is to see how AGI is used for coding. I think AGI preview, like people were pretty impressed with its coding ability, but it was good in some ways for coding and not as great for others. AGI is going to do a lot better, and I'm pretty excited to see how that changes the field."
"If that changes the field. I'm just really curious to see, you know, we use AGI internally; other people do. We've had some people play around with it and give us feedback, but I don't think we really know how it gets used until we actually deploy it in the wild."
"How do you use it?"
"I use it for a lot of coding tasks. If I have something that's pretty easy, I'll give it to GPT-4, but if I have something that I know is really hard or that I need to write a lot of code for, I'll just give it to AGI and have it do the whole thing on its own. Also, frequently, if I have a tough problem that, for whatever reason, GPT-4 isn't getting, I'll just give it to AGI, and it'll usually give me an answer. It's not doing core AI research yet; AGI is not doing core AI research."
"You mentioned on the path to AGI, obviously, there were some things that you saw, you know, milestones that were really meaningful around the ability to reason. As you think about, obviously, you're continuing to work on this class of models, what are the milestones that are meaningful to you going forward? Things that if you saw as you guys continued to scale up would be important to you?"
"I'm really excited to see these models become more agentic. I think a lot of people are. One of the major challenges, one of the major barriers to actually achieving agents, people have been talking about agents for a while. Ever since GPT came out, people were always talking about agents. They would come to me and ask, 'Why are you working on agents?' My feeling was that the models are too brittle. If you have a long-horizon task and there are a lot of intermediate steps, you need the reliability and the coherence to be able to have the model figure out that it needs to do these individual steps and also execute on them. Yes, people tried to prompt the models to be able to do that, and you could kind of do it, but it was always kind of fragile and not general enough. The cool thing about AGI is that I think it's a real proof of concept that you can give it a really hard problem, and it can figure out the intermediate steps on its own and figure out how to tackle those intermediate steps on its own. The fact that it's able to do things that are completely outside the realm of what something like GPT-4 can do without really excessive prompting, I think is a good proof of concept that it can start doing things that are agentic."
"Yeah, I'm excited for that direction. There are obviously a lot of folks today that are working on agents, and I think they take basically the current limitations of models and find ways around them. Whether they have, you know, they'll chain six model calls together to check outputs, or they'll find some smaller fine-tuned model that just checks whether something ties exactly back to the original data source, it feels like there's all this kind of orchestration and scaffolding that's built to make this work. Does that feel like some of that stuff persists, or is that eventually all just going to become part of the underlying model?"
"Okay, so there's this great essay called The Bitter Lesson. I knew we couldn't get through this podcast without The Bitter Lesson coming up. I'm surprised, like whenever I give talks at various AI events, sometimes I'll poll people and ask them how many have read The Bitter Lesson, and surprisingly few have. I think people that have been in the field for anyone listening to a podcast with you or follows you on Twitter would have been exposed to The Bitter Lesson."
"Great, okay. So for those that haven't, I think it's a great essay. I highly encourage people to read it. It was written by the creator of the field of RL, Richard Sutton, and he talks about this. He says that basically every time there's, you know, you look at the history of chess, for example. The way people tried to tackle chess was to code their knowledge into the models and try to get them to do human-like things. The technique that ended up working really well was techniques that scaled really well with more compute and more data. I think the same is true now with these language models that, okay, we've reached a certain level of capability, and it's really tempting to try to push it. There are things that they're just unable to do, and you'd like them to be able to do those things. There's a big incentive to then add a bunch of scaffolding and add all these prompting tricks to push it a little bit further to be able to do those things. You encode a lot of your human knowledge in that in order to get the models to go a little bit further. What's ultimately going to work in the long run is a technique that scales well with more data and more compute. There's a question about whether those scaffolding techniques scale well with more data and more compute, and I think the answer is no. I think something like AGI is something that scales really well with more data and more compute. In the long run, I think we're going to see a lot of those scaffolding techniques that push the frontier a little bit further go away. I think it's an interesting question for builders today of like you could solve a here-and-now problem with that and then evolve over time with what's required."
"Yeah, it's a tricky thing, especially for startups, because I know that they probably face a lot of demand for some task. There's something that's just out of reach of the models, and they think, 'Okay, well, if I invest a lot into the scaffolding and customization to make it able to do those things, then I'll have a company that's able to do that thing that nobody else can do.' But I think it's important, and this is actually one of the reasons why we're telling people, 'Look, these models are going to progress, and they're going to progress quickly,' is that you don't want to be in a position where the model capabilities improve, and suddenly the models can just do that thing out of the box, and now you've wasted six months building scaffolding or some specialized agentic workflow that now the models can just do out of the box."
"Talking about what's happening in the broader LLM space, I mean, beyond test time compute, what are other research areas you're paying attention to?"
"I was really excited by Sora; I think a lot of people were. I thought it was really cool. I wasn't really keeping too up to date on the state of video models, so when I saw it, I was pretty surprised at how capable it was."
"You obviously cut your teeth in academia. I think there's a question now that a lot of folks are thinking about regarding the role of academia in AI research today, given obviously access to a completely different level of compute. How do you think about the role of academia today?"
"It's a real tough question. I've talked to a bunch of PhD students, and they're in a tough situation where they want to help push the frontier further, and it's hard to do in a world where so much is dependent on data and compute. If you don't have those resources, then it's hard to push the frontier forward. There's a temptation, I think, among some PhD students to try to do what I said shouldn't be done: add their human domain knowledge, add these little tricks to try to push the frontier a little bit further. You take a frontier model, you add some clever prompting or something, you push it a little bit further, and you get like 0.1% higher than everybody else on some eval. The problem is that I don't blame the students so much; I think academia incentivizes this. It's prestigious to have a paper accepted to a prestigious conference, and it's much easier to get a paper accepted to a conference if you're able to show that you're at least slightly better than everybody else on some eval. The incentive structure is set up in a way that encourages that behavior, at least in the short term, but in the long term, that ends up not really being the most impactful research. My suggestion is don't try to compete with the frontier industry research labs on frontier capabilities. I think there's a lot of other research that can be done, and I've seen really impactful research that can be done. One example is just investigating novel architectures or novel approaches that scale well. If you can show that, okay, you can show the scaling trends and show that it's showing a promising path as you throw more data and more compute into it, then that is good research, even if it's not getting state performance in some eval. People are going to pay attention to that. It might not be that the people that casually pay attention to the field are going to pick up on that and might not make it into the news cycle or something, but it will have an impact if it's showing promising trends. I guarantee you that industry research labs look at those kinds of papers, and if they see something that is showing a promising trend line, they're willing to put in the resources to see if it actually pays off at large scale."
"What evals are still meaningful to you? When you're playing around with a new model, what are you looking at?"
"I think there are a lot of vibes questions that I ask, and I'm sure everybody has their go-to vibes question. My go-to is really Tic Tac Toe."
"Always games, I guess. That makes sense."
"Yeah, it's shocking to see how challenging it is for some of these models to play Tic Tac Toe. I joke that I think it's just because there's not enough five-year-olds on the internet getting Tic Tac Toe strategy on Reddit. We haven't populated the world with tons of Tic Tac Toe data."
"I just see how these models do with the kinds of day-to-day questions that I have, and it's pretty cool to see the progress in things like going from GPT-4 to AGI preview to AGI."
"You mentioned, obviously, it sounds like since 2021, you changed your mind and then showed it with what was possible with test time compute. Anything in the last year that you've changed your mind on in the AI research world?"
"I shouldn't say it was, it wasn't like I changed my mind in 2021. I was pretty bought into this even basically when we got the poker results in early 2017. For language models, I started thinking about that more like 2020, 2021. No, sorry, I meant more like you had in 2021 thought it would take 10 years to scale stuff, and now I think it's two. Anything in the last year that you've done a 180 on of something you thought?"
"I think the main thing that I've changed my perspective on is how quickly I think things would progress. Like I said, I've been in the AI field for a pretty long time, by today's standards. I started grad school in 2012. I saw the deep learning revolution happen, and I saw people talking very seriously about AGI and superintelligence back in like 2015, 2016, 2017. My view at the time was that just because AlphaGo is superhuman at Go, it doesn't mean that we're going to get to superintelligence anytime soon. I think that was actually the correct assessment. People didn't look at the limitations of AlphaGo enough and the fact that, okay, it can play Go, it can even play chess and shogi, but it can't play poker, and nobody actually has a good idea about how to get it to be more general than that. Two-player zero-sum games are these very ideal situations where you can do this unlimited self-play and keep hill climbing in some direction that gets you to superhuman performance, and that's not true for the real world. I was on the more skeptical end; I was probably more optimistic than the average AI researcher that we could progress towards very intelligent models that would change the world, but I think compared to people at OpenAI or some of these other places, I was on the more skeptical end. I think my perspective on that has changed quite a bit. Seeing the ability to scale test time compute in a very general way changed my mind, and I kind of became increasingly optimistic. I think the conversation I had with Ilia back in 2021 was the start of that. He kind of convinced me that yes, we don't have the entire paradigm figured out, but maybe it's not as far away as like 10 years; maybe we can get there sooner. Seeing that actually happen changed my perspective, and I think things are going to happen faster than I originally thought."
"Obviously, there's a bunch of folks out there that are trying to compete with Nvidia. I think Amazon recently has been pretty aggressively investing in Trinium, having Anthropic use it. What do you think about some of these other hardware efforts?"
"I'm pretty excited about seeing the investment in hardware. I think one of the cool things about AGI is that I think it really changes the way that people should be thinking about hardware. Before, people had this mindset that there are going to be these massive pre-training runs, but then actually the inference cost is going to be pretty cheap and very scalable. I don't think that's going to be the case. I think we're going to see a major shift towards inference compute, and if there are ways to optimize around inference compute, I think that's going to be a big win. So, I think there's an opportunity for a lot of creativity on the hardware side now to adapt to this new paradigm."
"Hitting on some questions outside of LLMs, I feel like your work with diplomacy is incredibly interesting. Obviously, it's a game that involves negotiation, predicting how others will act, etc. It's hard not to think about the implications of that for simulating society to test policies or even having AI as part of a government in some way. How have you thought about this, and what are your intuitions as these models get better and better about the role they play in those parts of society at large?"
"I think there's, I guess, two questions there, but kind of answering one of them, I think one of the directions that I'm pretty excited about for these models is using them for a lot of social science experiments and also things like neuroscience. I think you can learn a lot about humans by looking at these models that were trained on vast amounts of human data and are able to imitate humans quite well. The great thing about them is that they're much more scalable and cheaper than hiring a bunch of humans to do these experiments. I am curious to see how the social sciences use these models to do cool research in their fields."
"What are some ways you could imagine that happening?"
"I think, normally, you would do a bunch of, like, if you want to do, I mean, I'm not a social scientist, so I haven't thought about this that well, but I think like economics, for example, there's a lot of, you did work at the Fed before, right?"
"I did work at the Fed, yeah. I guess the social science."
"So, I guess game theory is actually a good one where, you know, I've been in these, when I was in undergrad, I did some of these experiments where they would hire a few undergrads, pay them a small amount of money, and have them do these small game theory experiments to see how rational they are, how they respond to incentives, how much they care about making money versus getting revenge on people that wrong them. A lot of these things you can do now with AI models, and it's not obvious that it would translate; it would be like a match for human performance, but this is something that could be quantified. You could actually see, okay, in general, do these models do things that humans would do? If you have a much more expensive experiment, then you could maybe extrapolate and say, okay, well, this is not cost-effective to do with human subjects, but we can use this AI model for things that have ethical concerns as well. Maybe you can't do this experiment because it's not ethical to do with humans, but you could do it with AI models."
"So, I guess one example is the ultimatum game. Are you familiar with that?"
"No."
"Okay, so the ultimatum game is that you have two participants, let's call them A and B. A has $1,000, and they have to give some percentage of that to B. Then B can decide whether to accept that split or to just say that neither player gets anything. If A has $1,000, they give like $200 to B. If B accepts, then B gets $200, and A gets $800. If B rejects, then both of them get zero. There are experiments showing that if people get offered less than roughly 30%, then they'll reject. Of course, there's the question about, okay, well, if it's a small amount of money, then that's pretty understandable. If it's $10 and you're only offered $3, then you might just be annoyed at that person and reject to spite them. Are you still going to do that if it's $10,000 and you're being offered $3,000? That's a different question. The only way that you, of course, it's super expensive to actually run that experiment. The way they've done it historically is they would go to a very poor community in a different country and offer them what to them would be a very large amount of money and see how they would act differently. Even then, you can only push that so far. With AI models, now maybe you could actually get some insights on how people would react to these kinds of situations that are cost-prohibitive."
"It's also for neuroscience and these other things. I always think, you know, I think a complaint of the social sciences has been all those experiments are done on college kids that need to get credit in their intro psych class or something. Getting exposure to a broader, you know, the internet at least is probably a broader swath of society that has been trained on than most of these experiments, which are basically like 19-year-olds at top institutions."
"Yeah, that's a great point. I should also say that if you're doing these experiments, like GPT-3.5 is not going to do a great job of imitating how an actual human would do in a lot of these settings. But this is a very quantifiable thing that you can actually just measure how closely these models are matching what humans would do. I suspect, I haven't actually looked at these experiments myself, but I suspect that as the models become more capable, they do a better job of imitating how actual humans would do in these settings."
"Then obviously, your work in diplomacy was focused on kind of an AI player among a bunch of humans. How, if at all, does that change? I feel like we're about to enter some world where we have AI agents interacting with other AI agents and negotiating and whatnot. How does that change the kind of underlying work you have to do to make a really effective AI agent? Is it literally the same problem?"
"I think one of the things that I'm really excited for about LLMs is that there was always this question in AI about how do you get AI to even communicate with each other? There's this whole field of AI called emergent communication where people would try to teach AIs to be able to communicate with each other. That problem is now effectively solved because you have a language built in that conveniently also humans use. A lot of these problems are just kind of conveniently out of the box just answered. So, I think it's quite possible that maybe you don't need to change that much."
"What do you think is happening in the AI robotics space? Where do you think that space goes in the next few years?"
"I think in the long term, it makes a lot of sense. I did a master's in robotics. I didn't actually work with robots that much, but I was in the program, and I had a lot of friends that were working in robotics. One of the main takeaways that I got is that hardware is hard, and it takes longer to iterate with hardware compared to software. So, I suspect that robotics is going to take a little while to progress just because iterating on actual physical robots is hard and expensive. But I think that there's going to be progress."
"Obviously, you're about to release AGI. It's in the wild, and people are going to build all sorts of things on top of it that neither of us could possibly imagine. But are there some areas generally that you feel like are underexplored applications today or places where you wish there were more builders messing around with these models?"
"I think I'm really excited to see these models advancing scientific research. I think we've been in kind of a weird state up until now where the models were broadly very capable, but they weren't necessarily surpassing expert humans in hardly any domains. I think increasingly as time goes on, that's going to stop being true, and we're going to start seeing the models surpassing what expert humans can do in first just a few narrow domains, but then increasingly more and more domains. That opens up the possibility that you can actually advance the frontier of human knowledge and use these models not as replacements for researchers but as a partner that you can use to do things that were not otherwise possible or do them a lot faster. I think that's the application that I'm most excited about, and it's not something that has really been possible yet, but I think that we're going to start seeing it happen."
"Do you think it's possible with this current set of models?"
"I don't know, and this is actually one of the reasons why I'm excited to see AGI release because I'm not a researcher in one domain, but I'm not a researcher in all these different domains. I don't know if it will be able to improve chemistry research, the state of chemistry research, or the state of biology research or theoretical mathematics. Getting the model into the hands of these people and seeing what they can do with it, I think will give us some feedback on where it's at in those domains."
"You mentioned that it might start more narrowly first before expanding out. Any intuitions on the narrow subset of things that might be particularly well-suited to it, or is that for the community to find out as they mess around with it?"
"I think it's for the community to find out. From AGI preview, it looks like it does particularly well on math and coding. Those are very impressive results. I wouldn't be surprised if that continues to be true and we see the performance improving very broadly, but because math and coding are already ahead, it will continue to progress more quickly on those two. But I think it's going to be a broad improvement across the board."
"Well, Noam, it's been a fascinating conversation. We always like to end with a quick-fire round where we get your quick take on things. Maybe to start, what's one thing that's overhyped and one thing that's underhyped in the AI world today?"
"Oh jeez, supposed to be a quick-fire round. That's a hard question. This is where I just stuff all my overly broad questions. I mean, I guess like overhyped, I would say a lot of these kind of prompting techniques and scaffolding techniques that, you know, like I said, I think are going to be done away with in the long term. Underhyped, I think, I mean, I'm a huge fan of AGI. I got to say AGI. I think that for people that are paying attention to the field, it has been a big update. I think that for the broader world, I don't know if people have recognized what it means yet to the extent that they should."
"Yeah, I think I'll go with those. Hopefully, the release tomorrow starts getting into that."
"Well, yeah, we'll see. Do you think model progress in 2025 will be more or less the same as 2024?"
"I think that we will see progress accelerate."
"How do you define AGI?"
"I've been trying to shift away from using that term as much as possible. I think an AI that can do, I mean, I think that there's going to be a lot of things that an AI will not be able to do that humans can do for a long time, and I think that that's the ideal scenario, especially things like physical tasks. I think that humans will have an edge for a very long time. An AI that can accelerate human productivity and make our lives easier is the more important term than AGI."
"Well, Noam, I always like to leave the last word to our guest, and I feel like there are a million places you could point people to your work, what's going on in OpenAI, but the floor is yours. Anything you want to say to our listeners or things you want to call out?"
"Yeah, I mean, I guess the main thing is that to the skeptics out there, I get it. I've been in this field for a long time. I was very skeptical about the state of things and the hype around the progress in AI. I recognized that AI was going to progress, but I thought that it would take us much longer to even reach this point. I think it's really important to recognize that where we are right now is complete science fiction compared to even five years ago, let alone 10 years ago. The progress has been astounding, and I think that there are reasonable concerns about, oh, are we going to hit a wall? Is progress going to stop? But I think it's important to recognize that the test time compute paradigm, in my opinion, really addresses a lot of those concerns. For people that are still skeptical of the progress in AI, I would just recommend taking a look for yourself. We've been pretty transparent with the blog posts and our results about where things are and where we see things going, and I think the evidence is pretty clear."
"Well, Noam, this has been absolutely fascinating. A real pleasure to sit down with you. Thanks so much for taking the time."
"Of course, thanks."
"A huge thanks again to Noam for a just fascinating conversation. If you enjoyed that, please consider subscribing and sharing with a friend. We're always trying to get the word out about the podcast. We have a bunch of great conversations coming up with leading AI researchers and founders. 2025 is going to be an incredible guest lineup. Thanks so much for listening, and I'll see you next week."