Transcription
The talk about AI bubbles seemed very divorced from what was happening in frontier labs and what we were seeing. We are not seeing any slowdown of progress. We are seeing this very consistent improvement over many many years where every say like you know 3 4 months is able to like do a task that is twice as long as before completely on its own. It's very hard for us to intuitively understand these exponential trends. If you manage to make everybody in society 10 times more productive, you know what kind of abundance can we achieve? What will we be able to unlock in the next 5 years? I think we can go extremely far.
Welcome to the Matt podcast. I'm Matt Turk from First Mark. Today my guest is Julian Shitwezer, one of the world's most impressive AI researchers. Julian was a core contributor to DeepMind's legendary AlphaGo Zero and Muse projects, and he is now a key researcher at Enthropic. We covered the exponential trajectory of AI and his predictions for 2026 and 2027, the frontier in reinforcement learning and AI agents, and the science behind AI creativity and the famous move 37 from Alph Go. Please enjoy this fantastic conversation with Julian.
Hey Julian, welcome.
Hey Matt, thanks for having me. A couple of weeks ago, you wrote an incredible blog post that broke the internet entitled "Failing to Understand the Exponential Again." What is it that so many people are missing about the current trajectory of AI?
Yeah, it's funny that you bring up that blog post. I really didn't expect it to blow up that much. I actually had the idea when I was on holiday in Kyrgyzstan a few weeks ago on a very long car ride, and then I started thinking about this and like all the talk about, oh, AI bubbles I've seen on X and like, you know, this discussion, and it seemed very divorced from what was, you know, what was happening in frontier labs and what we were seeing. And that made me start to wonder a bit, like, is it that things are moving so fast that people maybe struggle a bit to extrapolate and understand intuitively? Oh, you know, maybe it's far away now, but, you know, it's doubling every so many months, which means that once it gets close to us, it's going to move past and become really good very quickly. And that reminded me a lot, in like a different way, but like what happened during early COVID, where we had a similar situation where, you know, at the beginning, it's like very few cases. It's like, wow, you know, it's never going to happen. It's only a few hundred people. Who cares? But if you understand the math and if you look at it, right, it's like, oh, it's going to double every, you know, week, two weeks. Clearly, it's going to be a massive scale. But it's it's very hard for us to intuitively understand these exponential trends because it's just not what we're used to in our normal environment. And so that's what got me thinking, oh, is something similar happening here with AI? Right? We are clearly, if if you're looking at many benchmarks, we have many evaluations, we have, we are seeing this very consistent improvement over many, many years where every say, like, you know, 3, 4 months, it's able to like do a task that is twice as long as before completely on its own. And so we, we can extrapolate this, right? And we see that, oh, in a year from now, maybe two years from now, the top models are going to be able to work completely on their own for like a whole day or more. Combined with this, combined with the fact that there's a huge number of knowledge-based jobs in the economy, knowledge-based tasks, and combined that in the frontier labs, we are not seeing any slowdown of progress. Just extrapolating those things together over a very short time, like, you know, half a year, one year, already that is enough to know that there is going to be massive economic impact. That means if you look at current, like, if you look at OpenAI, if you look at Enthropic, if you look at Google, those evaluations, those revenue numbers are actually fairly conservative. I think some more thoughts, some things more I've seen recently is that it's maybe actually even more interesting and more complex that, you know, while those frontier labs and frontier models are clearly very capable and on like an extreme trajectory, there are like a lot of other companies, right, that are trying to follow into the same AI sphere that may also have very high evaluation but not necessarily the revenues to support it. And so it's possible that there may simultaneously be like some sort of bubble in, you know, the wider ecosystem while at the same time the frontier labs on a very solid trajectory, having a lot of revenue, making a lot of money. I think that may be quite of an unusual situation. That in the past, you know, maybe in the dot-com bubble, maybe people were talking about like, you know, the railroad rush and stuff like this, we did not see this bifurcation. So I think it, yeah, I've been thinking about this more, and I think it's it's getting more and more interesting the situation.
Fascinating. So you alluded to some of your predictions or extrapolations for '26 and '27. Do you want to unpack that? You had, you had three of those.
Maybe, you know, calling it my prediction is giving myself too much credit, right? I will just say this. If you look, for example, at M-Eval and you very naively extrapolate the linear fit, that's what you would expect to happen. And so I'm just going to be humble, right, and say like, "Oh, most of the time, right, I'm not going to be smarter than statistical models, statistical extrapolation of past trends that have been very consistent." So I'm just going to be very humble and like, despite, you know, what I might know all about research and what's happening. Probably like, you know, the most likely, the best prediction I can make is actually just follow that data, that extrapolation and see where it's going to take us. And yeah, in that case, if you, if you roll this out, if you look at other benchmarks, I think we would have something like next year, maybe the models will be able to work on their own for a whole day's worth of tasks. If you think of software, right, you might say like, oh, implement this entire feature, build out this entire set of the app. If you think of knowledge work, right, if maybe do like a whole research report, this kind of scale. The reason I think why task length specifically is interesting is because that's what allows you to delegate more and more work to language models to agents. Even if you have a very clever model, but if it needs feedback or the interaction with you very often, then it really limits what you can delegate to it. If you need to talk to it every 10 minutes, right? Versus if you have something that can go for hours at a time, obviously, right? Then you cannot just have one copy of it. You can have a whole team that you delegate tasks to and manage them. And so I think that's why it's really critical that the models are actually smart enough, the agents are smart enough to work on their own, to correct their own errors, to, you know, iterate, because that's really what allows you to delegate indeed. Uh, task length and time to complete as the metric for progress. So by mid-2026, you mentioned agents can work all day autonomously. Late 2026, at least one model matches industry experts across many occupations. And then by 2027, models frequently outperform experts on many tasks. So it's more time running, and then generalization across the economy. And you mentioned GDP-Val, the OpenAI metric, as, um, as a benchmark to, to, um, uh, already see the progress towards multiple professions.
Yeah, I think the GDP-Val is like a super cool evaluation from OpenAI where they collected a lot of like, you know, real-world tasks from real domain experts to make sure it is actually representative of what you might do in the economy. And then they evaluated a lot of models on those tasks. They compared them against real experts' performance to give us like a really good indication of, you know, how close, how far are we from having significant economic impact. So I think that's that's like a super cool evaluation. The sort of obvious question is that GDP-Val and M-Eval are carefully, uh, designed benchmarks. How do they predict production value once you add compliance, liability, messy data, the messy world, tool friction, and all the things?
So I think like messiness and task length, like, you know, time duration that you're able to work independently are very similar or very correlated. So I think that's why it's interesting that M-Eval tries to measure how long can the model go on its own. Because if you know, if you think about like, you know, how do you come up with a task that, you know, takes a human 8 hours, 16 hours, right? You will have to include all these messiness and all this real-world mess to even be able to measure it. But I think, you know, ultimately, to go further, we really want benchmarks, we really want evaluations that come from the actual users, whether it's the industry, whether it's private users, because that's that's what ultimately matters, right? Like, is the model helpful to you? Do you get something out of it? Does your paperwork, helps you write something, fixes your code, helps you study, right? I think that's the real proof. If you release a new model, right, do people start using it more? Do they really enjoy it?
Is there anything that would change your mind? Any kind of signal, whether that's real-world adoption or benchmark performance, something that would make you more, um, cautious about, um, that exponential? Is there anything that would change your mind?
I mean, many things, yes. I think like, you know, many of these things are like, you know, internal only, right? I might look at our model pre-training, I might look at our fine-tuning, I might look at RL things, you know, how do new runs go compared to past runs? Do they match our expectations? The scaling continue? Then, you know, I might look at more public things of like, are people actually able to use those models to be more productive? For example, at the beginning, right, there's always some adaptation period of, oh, you have like a new tool like cloud code, takes you some time to figure out how to use it. But then in the medium term, in the long term, do people keep using it? Are they getting more and more productive using it? I think that's like, you know, one of the things I look at. Many, many signals. I think when you do RL, when you do research, I think you get very much in the habit of like looking for signals to prove yourself wrong. Because you know, you often have ideas that you get attached to, but that's not a good way to do research, right? You most of your ideas are not good and they're not going to work. So you really want to figure out as quickly as possible whether this idea is any good or whether it's actually wrong. So if you really get into this habit of like, oh, finding the fastest thing that will show that, oh no, this is actually not true.
So by 2026, 2027, in your extrapolation framework, AI becomes as good as humans. A key question of the moment is, um, can it be, to which extent can it become better than than humans? There's, um, uh, some chatter these days around, uh, move 37 and, uh, whether AI can create those like alien new paths to think and solve hard problems. So first of all, maybe remind the audience what move 37 is, and then do you think that AI, in its current state, uh, is going to be increasingly able to to provide move 37 type thinking?
Yes. So I guess, yeah, to give background, move 37, that was when we were building AlphaGo, AI program to play the game of Go. That was like in the year 2016, I think. And we're playing one of the best players in the world at the time. Because at that time, you know, no AI program, no computer program had ever beaten the top human players at Go. And it was considered to be, you know, one of the most difficult board games, sort of like, you know, a real test of intelligence. Move 37 happened during the second game of the 5-game match where AlphaGo played like a really unexpected, unconventional move that surprised many professional Go players. I think, you know, the commentator said it was, you know, truly creative, unexpected. And then ultimately, AlphaGo ended up winning that game. And so I think that was for many people an early sign that AI is not just, you know, purely calculating, following an optimal path, but it can also do something that is truly novel and creative that you might not expect, you know, just from imitating its training data. Yeah, I think I think that's very relevant in a modern context as well, right? Because as you alluded to, there is a lot of discussion of, oh, are LLMs just paring the training data? Can they actually do novel things? For me, as somebody who has been doing research a long time, I think it's pretty clear that these models can do novel things. And that's why they are so useful to many people, whether it's, you know, writing code for you, because obviously, right, you're not just writing code that you already have, that wouldn't be very interesting, or like helping you, you know, write a paper. Or the way those models are trained, they're literally trained to generate a whole probability distribution, which means that when we sample from them, we can generate, you know, infinite amount of novel sequences from them. For the question of something like move 37, I think there, it really comes down to like, you know, is it something that is sufficiently creative and impressive that we can easily recognize it? In the game of Go, right, that was pretty ideal conditions because it's, you know, very clean, very abstract, you can really, each move is very impactful, so you can really see it clearly. I think to have the equivalent for our modern models, you need the combination of a task that is like sufficiently difficult and interesting, and a model that is both able to create sufficiently diverse and creative ideas, and also able to evaluate accurately how good they are, so that it can go down, you know, increasingly novel paths while making sure that this novel path is actually, you know, interesting and useful. Creating novel things is actually very easy with language models. The hard part is creating novel things that are useful and interesting.
Extrapolating this further, there's the idea of creating novel science. So not just one move, but like a whole new idea, new concept. What's your current take on on this?
So I think AlphaCode and AlphaTensor proved that you can discover novel programs and algorithms. Uh, very recently, I think last week, there was some news of, uh, Google DeepMind and Yell, uh, in the biomedical field coming up with with brand new things as well. So do you think that's, uh, accelerating and that AI is in the process of discovering novel science?
So I think we're absolutely at the stage where, you know, it is discovering novel things, and we're just moving up the scale of how impressive, how interesting are the things that it is able to discover on its own. And so I think it's highly likely that sometime next year, we're going to have some discoveries that people pretty, you know, anonymously agree that, you know, this is super impressive. I think at the moment, we're more at the stage of, you know, oh, it came up with something, but there's debate about it. And like, but yeah, I'm not very worried because I see this process continuing, and then once it gets clear enough, there's less need to argue about it.
How far do you think we are from an AI winning the Nobel Prize?
Yeah, I think that's a really interesting question, right? Because we had a Nobel Prize for AI with AlphaFold, of course. And so I think the next very interesting point is going to be when can AI on its own make a breakthrough that is so interesting that it would win a Nobel Prize. And I think my guess for that level of capability might be maybe 2027. I think we're, we're probably not going to find out for quite some time afterwards because of the delay in getting prizes, but I think by 2027, 2028, I think extremely likely that the models will be smart enough and capable enough to actually have that level of insight on that level of discovery.
Amazing. I think, yeah, Nobel Prize, right? It's like the Fields Medal for math and all these kind of like advances. I think, you know, that's that's what I'm truly excited about, actually, is like AI that can help us advance science and really unlock, you know, both all the mysteries of the universe and all the improvements in living standards and abilities for us that we could have if we understood the world better.
All right. So extrapolating this even further, then we get into the AI 2027 thing that that you you you probably saw. So this general idea of, um, if AI, uh, can create novel science, then AI can create, uh, AI researchers and basically AI can create itself, which effectively leads to a discontinuity moment. So I don't know if in the blog post that's the singularity or whatever. Uh, but does that strike you as somebody who's, uh, you know, as deep in the field as possible, as something that is possible in the short term, or are there counterbalancing forces that make that path to discontinuity harder as you get closer?
Yeah, I think a true discontinuity is extremely unlikely. From, you know, obviously AI researchers are already using AI to accelerate themselves. And so what's what's already happening and like what is likely to continue to happening is that we see like a smooth improvement of productivity. And then the main open question is, how does the difficulty of improving AI keep scaling? Because a very common effect in a very common issue in many scientific fields is that we find all the easy problems first, and then, you know, as we continue exploring the problem, the field, it gets more and more difficult to make advances. So in my mind, the main question is, do these two trends balance each other out so that, you know, the AI makes us increasingly more productive, so that as it gets more difficult to make advances, we just sort of about stay on trend and then we keep improving roughly linearly? Or is it still too difficult, and then, you know, eventually after some time, we still see a slowdown? But it seems quite unlikely to me that we improve in productivity so much that we can actually accelerate. That would be very unlike any other scientific field. The normal course in many scientific fields is that we actually need to exponentially increase the research effort just to keep making progress and find new insights. For example, if you look at pharmacology, discovering new drugs, it's nowadays in the range of billions of dollars to discover a new drug versus, you know, maybe 100 years ago, a single scientist could discover the first antibiotic, right, by accident. It's not that we will be surprised by sudden takeoff in progress where, you know, oh, we're just doing our research and suddenly our model is like 10x better. We will be seeing advanced signs of, oh, we're making faster progress every single week. We can see something is happening. Maybe we decide to pause if we don't understand what's happening.
Do you think the current approach to modern AI systems, which effectively is pre-training plus RL, does that take us to where we want to be? Whether we call it AGI, ASI, it's unclear what any of the things mean. But, um, do you feel that this paradigm is the right one, or do we need to come up with a different architecture altogether, post-transformers, or otherwise?
I think that's a great question, and I think it hugely depends on what you mean by where do we want to be. So I think if you're thinking of, oh, we want some kind of system that can perform at roughly human level in basically all tasks that we care about productivity-wise, then I think, yeah, it's extremely likely that the current approach, pre-training or, you know, transformers, is going to get us there. If what you care about is, oh, we want to have a model of intelligence that is conscious in the same way we are, or, you know, more abstract qualities like this, I think that's maybe more uncertain, right? And I think this is where a lot of the confusion and disagreement comes from. As you alluded to, you know, AGI, ASI, like, you know, people talk about very different things and they have very different things in mind when they say, oh, the current paradigm is going to get there, it's not going to get there. I often, yeah, like to not use the term AI or ASI and just talk very concretely about, you know, what problem are we solving, what task are we solving, what quality are we interested in, because I find that often makes the actual disagreement much more obvious. But yeah, I think if you're just thinking of in terms of like, is this going to help us be massively more productive? Is this going to massively accelerate scientific progress? Then I think definitely the current approach will get there.
And given how extremely deep you are in, I cannot resist asking you sort of the, the, the, the trendy question, uh, duour, based on Richard Sutton's recent appearance on the Lex Fridman podcast. Do you think that the models of the future will be trained in RL from scratch, and that actually having pre-training, um, in addition to RL, is the wrong way to go?
Personally, I think that's unlikely. Not, not because pre-training is strictly necessary. I think we may well be able to train something completely from scratch, as we've been able to do in other domains. But more because pre-training on these vast datasets we have just brings us so much value that we would, from a practical point of view, not want to give it up. So we, you know, we might well do some agents that are trained from scratch out of scientific interest. It could be very interesting to learn about what would a non-human intelligence look like. But from a programmatic point of view, I definitely think we would keep using pre-training data. Not just from an efficiency point of view as well, but also I think there are interesting safety angles because by pre-training on, you know, all this human knowledge, we're implicitly creating an agent that has similar values as we do. And I think that is quite valuable for aligning, uh, you know, highly intelligent agents. If you already start out by caring about sort of, you know, the same rough set of values, that makes things much easier. Then you create an, you know, arbitrary alien intelligence that may have completely different values. Despite having, you know, done a bunch of from-scratch RL in the past, I think I'm often quite pragmatic about this.
I'd love to put a pin in that specific discussion about, um, alignment, and what we do to ensure safety, to later in the conversation because I think that's a super interesting, uh, vein. Um, but maybe to switch tacks for for a minute. Um, I'd love to go into a little bit of your, uh, story and then the monumental body of work that you've done at Google DeepMind before joining Anthropic around, uh, AlphaGo, AlphaZero, M0. So maybe, uh, just the 3-4 minute version of of your personal story from when you were a kid. What was the path that led you to become a world-class AI researcher?
Yeah, actually, when I was a kid, I didn't have any expectations of becoming an AI researcher. I was always very interested in computers. And I grew up in the Austrian countryside in a small village. So, you know, it's not like there was a huge amount of things happening, but computers were always very interesting to me. You know, it's like this connection to the wider world, to all these other interesting things. And I was very interested in computer games as well. And I think that's the first time I became interested in programming because I wanted to make my own games, which I think is very common in people who get into programming. But I somehow, I always got distracted by the technical aspect of, I'm going to build a very general game engine that, you know, can run any kind of game. And so I never actually ended up making any game. I learned a lot about making game engines and different technologies. And that's how I ended up studying computer science eventually in Vienna. Yeah, there was like a classical computer science degree. And then by chance, after my first year, in my first summer holidays, I had an internship with Google. And that's when I realized, oh, wow, these guys are doing really interesting things. They, you know, that's where their big clusters, the tens of thousands of machines are. That's first time I radically changed my plans from wanting to stay in academia. And, you know, I had originally thought, oh, maybe I'll do a PhD. That's when I changed, oh, no, actually, I just want to join these guys at Google. And I will finish my degree as quickly as possible. And so that's actually, yeah, when I got my full-time position at Google, finished my degree the next year, and then moved to London. So I was just working as a normal software engineer at Google, working actually like in advertising, which I wasn't super excited or interested in. So like the technology was interesting, right? Is like these huge systems, and Google has, you know, famously great technology. But actually, after, you know, a yearish of this, I was pretty done on board of advertising. And so I was actually planning to leave Google and thinking of maybe joining a hedge fund, going into finance. When by chance, I saw an email in my work inbox that this guy Demis was going to come to the office and give some talk about Atari and video games and AI. And it was actually a day off because I was visiting a friend somewhere else in England. But that email looked so intriguing that I was like, "Oh no, I'm going to have to like take the train back to the office right now and like see this talk." And yeah, like I'm really glad like that I saw this email and I did go back because that's like the moment where I decided, oh no, like no, I'm not going to go into finance. I'm going to move to DeepMind. I'm going to join these guys because this looks clearly, you know, super interesting, super amazing. They are doing really interesting research.
All right. Tell us a story of, um, AlphaGo, AlphaGo Zero, AlphaZero, MuZero, because it's, it feels like it's fundamental AI knowledge that everybody who has an interest in the space should know about, should understand the progression. In particular, so, um, starting with the beginning of AlphaGo, you alluded to it a second ago, but like, what did it do? What, how was it trained? And then how that, how did that evolve with each version?
AlphaGo, I think at that moment in time, Go in the machine learning community was this really big target where everybody felt like, oh, you know, it's this big unsolved challenge. ImageNet had just happened before, so clearly, you know, models were starting to do something with images and being able to recognize them and predict them. And if you look at the Go board, you know, the right way it looks a lot like one of those images that you classify. So there was a lot of momentum around using neural networks to somehow play Go. And then at the time, David Silver and Ashang Deep had been working on Go, I think like both of them had been working on Go for quite a while, had published some very interesting papers. And that's when the idea of using Monte Carlo research with deep networks came together. So the idea was that to train a deep neural network to predict which moves you might want to play and, you know, whether you're winning or losing the game, and then use the tree search to really make a big plan of what are all the possibilities in the game. How would it go for you if you chose a certain move or a different move? How would the opponent respond?
And to explain this in super plain English, uh, the term search in this case is, as you said, is research, is not what people normally think of search, which is searching a corpus. This is searching a series of options, effectively. Is that, is that the right way to think about it?
Yes. It's, it's quite literally what you might do when you play a game of chess, when you play any board game. It's quite literally thinking of, you know, what move am I going to do? What move is my opponent going to do in return? And then thinking about many possible moves like that and mapping out all the possibilities in the future.
So deep learning plus search. What was AlphaGo trained on?
Initial training phases of AlphaGo were on some human amateur games, if I remember correctly.
Mhm.
So basically just predicting, if you have humans playing many games of Go, try to predict at each turn in the game what move would they have played? And it turns out that, you know, if you train a deep network to do that, you can get something pretty decent, like amateur Go level.
Mhm. Mhm. But not good enough to actually beat a really strong player.
And by the way, just for the lore of it, uh, did you guys have any, uh, sense that, uh, was going to crush Lee Sedol? So, the the famous Go player that you mentioned earlier in the conversation? Was it, was it obvious before? Was that a surprise?
We thought we had a pretty good chance, but we were very nervous about like, you know, are we going to win? Are we not going to win? Are we going to lose? Yeah. We actually had some bets beforehand of like, how many games we're going to win or lose. Like, I think it was very ambitious to put the match as early as we did. If we had wanted to be a bit more safe, we may have like tried to do a few months later. And I think if you had done it a few months earlier, we would have probably lost. So it was very knife-edge of, I guess, which also made it much more interesting for us, right? Because it really means that each game is like a nail-biter of, oh, what's going to happen? Are we going to win? Are we going to play, you know, dumb move? What's going to happen? So that was very exciting.
AlphaGo Zero, which was, I believe, the year after. How was that different? What was the progression?
Main change between AlphaGo and AlphaGo Zero was to remove all the human Go knowledge. So instead of starting by imitating human Go games, we were training it just from scratch, playing only against itself, and rediscovering basically all of Go, completely figuring out from scratch how to play.
Did you give it the rules of the game?
We didn't give the rules of the game to the network per se, but we used the rules of the game to score the result. So basically, you know, it would play, and it would tell it, you know, who won, who lost, or, you know, you cannot make this move.
So the next hop was, uh, AlphaZero, which was a year or two later. How is that different?
So AlphaZero, the idea was, well, obviously Go is a really beautiful game, but ultimately, we would like to do something more general, right? So can we remove anything that is Go-specific and verify that the algorithm can actually solve more problems? And in that case, we did that by trying to solve both chess, Go, and Shogi, which is a Japanese chess, basically with the same algorithm, you know, the same network structure, just by running it in different games, and also making it, you know, much simpler, elegant, and faster. So basically, that was really laying the groundwork for applying the algorithms to solve real problems. And then the next, um, stop in the journey was MuZero. And and just to bring it home for, uh, people, you were, uh, I believe second author on AlphaGo Zero, and you were the lead author on MuZero, which, um, in the world of, of, I'm, I'm sure you're going to be very humble about it, but like, in the world of AI, is like as big a deal as it gets. So, um, I'll say it so you don't have to say it. Um, so MuZero, what was the next, um, what was, how was that different?
So the main motivation I had for making MuZero was that if you want to solve many real-world tasks, you have no way of perfectly simulating what's going to happen. And, you know, if you play a board game, obviously you know if you make this move, you know what's going to happen. It's like the piece is going to go there, it's going to take a piece, whatever, right? But if you actually want to solve something like a robotics task or anything more complicated, it's impossible for you to simulate what's going to happen accurately. And also, we as humans, we don't do this, right? We just imagine in our head of, oh, if I'm going to say this, then he's probably going to respond in that way. This meant that AlphaZero, as it was, could not be applied to such problems because it required some way of, you know, simulating the game, scoring the outcomes. And the idea with MuZero was that, well, we already have a deep neural network, right? These networks can learn a lot of things. So why not let it, why not teach it to predict the future of the environment, the future of the world? Why not make the model be able to learn for itself what is going to happen after each action it takes?
After that, you also, uh, applied this to, uh, code and math. So that was AlphaCode and AlphaTensor. So like zooming out a little bit, that evolution of, um, reinforcement learning in games, and then code, and then math. What did you learn about the general power of search and learning that is today relevant in modern agentic AI systems? How did that, that whole body of work translate to what, uh, you are doing today?
So games are a really good sandbox to learn very quickly about a lot of the reinforcement learning science. You know, the algorithms that work well, the kind of problems that we encounter, the even from a technical point of view, how do we build a learning system that spans, you know, many data centers, uses tens of thousands of machines, because games are very clean sandbox, very clean environments, so we can make many good experiments. And then now that we have a much more general model, right, the language models can do almost any task, but they're much more complicated. They're much slower to experiment with. We can apply those same lessons of, ah, you know, we know how to build a really robust reinforcement learning infrastructure. And now we can build the same one for language models, or like, you know, we know if you do this kind of RL, then the model will learn how to exploit the reward. And so we can apply the same lessons, the same mitigation techniques to the language models.
If I understand correctly, I think MuZero had, um, a learned world model.
Mhm. So basically, sort of rehearse the future, for lack of a better expression. Uh, so do, do modern LM agents have, have anything like that? Do they have an internal world model that lets them preview actions before they commit?
So I think yes, I would say that language models have an, not an explicit world model, but they do have an implicit model of the world. Because to be able to predict, you know, what is the next likely word in this sentence, how is this paragraph going to continue, they need to internally model, you know, what is the state of the world that makes this person say that thing. And so it's, it's actually somewhat similar to MuZero in the sense that MuZero also only had an implicit world model. You know, it was never trained to predict, you know, what does the screen actually look like if you take an action. It was also only trained to implicitly predict, if I take this action, you know, what is the next action I should take, or is it going to be good or bad for me. So in both those cases, you have an implicit representation of the world in your model that you can use to make predictions, but you're not actually reconstructing the full state of the world. Because reconstructing the full state of the world, you know, that can be very expensive and complex. If you think about, you know, super high-resolution video, audio signals, it's a very large amount of data that probably you don't actually need. If you think of human attention, we are only aware of a very small subset of what's actually going on all around us all the time, because that's, you know, the most relevant information that we actually need to make decisions. And that goes back to the prior discussion about so retraining. So the reason why pre-training and RL work well together is that you have that world model that's, um, implicitly embedded into the, the, the, the corpus. Although the argument against it is that it's what humans think the world model is, uh, as embodied by language versus what the world model actually is. And that's, that's my understanding of the, of the, of the debate.
I mean, for the debate, I think different people have different points of view, so I don't want to speak for anybody, but, yes, yes, yes. But yes, I think like pre-training on this rich knowledge gives you some representation of the world already, so that when you actually start to act and interact with the world, you can very quickly make, you know, meaningful decisions, meaningful actions. I, yeah, I like to think of it, you know, in a similar way. If, if you look at many animals when they are born, they very quickly know how to move, how to run, even, right? If you look at gazelles, for example, in the savannah, in a way that is like, you know, clearly they did not have time to really learn this from scratch, right? Few minutes or hours. And, you know, in their cases, they did not do pre-training, but they have some evolutionary encoded structure in their brain. Because clearly, it is very beneficial to have some sort of knowledge to make your learning more efficient.
Yeah, just RL in nature would, uh, would lead to not so good results. Like if you're a gazelle and like you have to A/B test whether to run towards the lion or away from the lion.
Exactly.
It's like the, you know, thousands of generations of gazelles acquired this knowledge over time.
Yeah. It was encoded in their genes and their brain structure in some way, right? And then you get to start on top of that. I think the main, you know, the main challenge or the main thing you need to watch out for is that you don't overencode or you don't restrict your search base too much. If, if your pre-training, if your prior knowledge prevents you from exploring something that might be the correct course of action, that will be bad. So there, you know, there is some danger there, you have to be aware of.
So this general idea of making pre-training and RL work together in modern AI systems seems to be the the big idea or topic of 2025. Although of course, I know it's, it's been years in the making. Uh, why did it take so long? U, it feels like RL, you know, progressed, uh, in, in, in its own direction, and then pre-training worked in its own direction, and those were slightly separate. Why did it take take so long to put them together? Is that just purely practical and economic, or or anything else?
Scaling up the language models to the massive degree that we scaled them up took a lot of effort on its own. And from a science point of view, from an engineering point of view, pre-training and supervised training is more stable and sort of easier to debug because you don't have this feedback cycle. You basically, you know, have a fixed target and you're trying to learn this target. And so then you can focus on like, you know, is my training working, and like, is my infrastructure working, and then, you know, does it scale as a fallover? Versus if you compare to RL, in RL you have this feedback cycle of, oh, I learn something, and then I use that to generate my new training data, and then I learn from that training data. And now, if you have, you know, something is not working, it's very hard to figure out where in this cycle your problem is coming from. No, maybe your training update was bad, and that's why you suddenly started behaving badly, or maybe the way you decided, you know, the way you select actions to behave is not correct, and so you generate bad training data, and that's what messed up everything. So it's just much more complicated to get working correctly. And so I think it makes a lot of sense to, you know, first scale up the pre-training, the architectures, figure out something that works pretty well, especially if you can already get pretty far by some fine-tuning, some prompting. And then when you know when it's clear that these models are really general, they are really useful, and we have them in a pretty stable state, then, you know, you can ramp up RL and take them even further. Even in our own work, right, if you look at AlphaGo, AlphaZero, we always followed a similar split as well. When we first set up the architecture of the network, the training using fixed supervised data. And only when we had that working really reliably, only then did we do the full RL loop and the full training. Just because like debugging all of it at the same time, you're just setting yourself up for failure. It's really useful to be able to isolate the component and say like, you know, I have known good data over here. I have a known good target there. If the thing in between is not working, I can isolate it. And then, you know, we can isolate all parts of the system.
How compute intensive is it to scale RL, and are there scaling laws for RL the same way you do in pre-training?
There's less published literature about it. But I think if you, if you look at all the RL literature over time, we see very similar returns on compute in pre-training and in RL, where we can invest exponentially more compute in RL and keep getting benefits. There's going to be some interesting research to come to figure out what are the trade-offs between pre-training and RL compute. We know how, what, what should be the split for a big model? For example, it could be 50/50, should it be like 1 to 10? Which way should it be? 1 to 10? So I think that's going to be extremely interesting. But so far, yeah, we definitely see good returns on both.
What's the latest state-of-the-art or or thinking, uh, in the field of, um, rewards? So in what you described, uh, for AlphaZero, AlphaGo, that was basically win/loss as a reward. Board. Then it sort of feels like we went into kind of like fuzzy human matching. This is good, this is not good. And now that we expand, um, as, as per the above, into more general fields where it's sort of unclear whether you win or lose, how does, how does that work? What, what parts of the evolution are you working on? Are you excited about?
Personally, I don't work that much on reward modeling. I mostly work on sort of reasoning, planning, search time, search compute, ways of making the model smarter by spending more computation. Yeah, thinking about rewards. I think the reinforcement learning process per se doesn't really care where the reward comes from. The algorithms are very happy to use any source of reward. Whether that's like a human feedback signal, it's like some automated signal from like, you know, winning, losing the game, or passing a test. Whether it's something more model-generated, for example, at Anthropic, we had this paper about Constitutional AI to, you know, have the model itself score whether you're following some guidelines.
Mhm.
So it can be very flexible to what kind of reward you follow. The RLHF, like all, all the things are those, those are at this
stage um stuff that that you see commonly common commonly uh used um any any many any thoughts?
>> Yeah, I think we're seeing like a huge mix of rewards and environments, and I think it's very much much people working very hard in figuring out what are the best reward sources and how do we scale it up and how do we, you know, get more rewards, more reliable rewards. That will be one of the key ingredients in scaling up RL further.
>> And so switching from from rewards uh, what is the latest thinking uh in terms of uh data training data for RL? Uh, again, following the uh evolution from like AlphaGo where it used to be human data and then like self-selfplay uh, what how does that uh how does that work? Where does the data uh come from and what kind of data works best to train modern RL?
>> Yeah, I guess the great thing about RL is that the data is generated by your model itself. So the smarter our models become, the better RL data we can generate, the more interesting and complex tasks they can solve, which then gives us more and more data that we can train on. Cuz like, you know, the more the more complex the task, the longer it takes to solve the task and the more data it generates that we can then use for training. I think part of the challenge is to find tasks that are really representative of what people actually want to do with the model because now language models are so general, people are using them for so many different things. There's more and more of a challenge of, you know, we need to cover as many of those as possible in order to make sure that, you know, the model is actually able to do this diverse set of tasks.
What uh matters more for training data? Is that quality? Is that quantity? Is that recency?
>> I think that's like a very interesting question that maybe doesn't have like a super clear answer yet, or it's maybe still interesting research to be done. I think we've seen papers arguing for different things or we've seen different benefits, right? Like clearly we see pre-training as we scale up the data, we can keep improving. But we've also seen very interesting fine-tuning results papers published for with a very small amount of examples, you can teach the model how to do an interesting skill. And I think we don't have any good scaling laws yet that tell us the trade-off. Especially, I think because it's very hard to measure what is the quality of a data point, right? Like how good is this example compared to this other example? Without being able to measure this, it's very hard to quantify the trade-off in any.
Okay, I think intuitively it's definitely true that if you have bad data, RL doesn't work that well, and if you have very high quality data, it becomes much more stable. For example, I think that was like a it's very clear in like Alpha Zero days where you we spend Alpha Zero spends a lot of computation. It does a lot of planning and search to decide which move to take. And so that generates very high quality data to train on, which then resulted in RL training that was incredibly stable. So, you know, you can run it across continents, take a long time to generate the data and then train on it, and it's very robust. Versus in modern RL with language models, you know, the difference in how good is the model and what data it generates that we then train on is not so large. Because we more directly sample from the model and then train it, which then results in reinforcement learning that is less stable. And so one direction of scaling RL and making it more stable is by improving this by, for example, putting more reasoning into your language model to generate much more high quality training data that can then give us training that is much more stable and that we can scale up much more easily.
I'd love to spend uh a little bit of time now on the general topic of RL and agents. So the famous agent AI that everybody's been talking about uh, you know, breathlessly for the last year. Uh, so for people listening and, you know, as as often in an effort to make this broadly accessible by by by, you know, a group of general people in tech, um, could you drive home the the uh sort of intersection and overlap between RL and and and agents? Does RL power agents? How does that work?
>> Yeah. So I guess like maybe first, let's take a step on what do we actually mean by agent?
>> Yes.
>> As compared to like as like a general language model, right? The second most debated question after AGIs is what is an agent? Yes.
>> I guess. Yeah, for our purposes, let's just say that an agent is an AI that can act on its own. You know, maybe take some actions on a computer, save some files, edit some files, send an email, whatever you want, right? But the main characteristic is that it doesn't have to interact with the user all the time. It can do things on its own. The reason why RL is very important for this actually connects back to pre-training because our pre our pre-training data is not very agent-like. If you think of the pre-training data, right, there is like websites and books and, you know, all kinds of recent text that has a lot of information, but it doesn't have a lot of actions. It doesn't really capture how the humans actually interact with the world. So if you take a raw pre-trained model, it's not a very good agent. You know, maybe you can prompt it a bit and like, you know, sort of push it in the right direction, but it's not going to be very good at interacting, and especially it's not going to be very good at correcting for its own errors because the pre-training data has no examples at all of how is our agent going to fail. And that's exactly where reinforcement learning comes in because in RL we can take our agent, let it interact with the environment, and then directly train on that interaction. So, for example, if the agent did well, we can reinforce those actions. And if the agent did badly, we can push it away from those actions. And if the agent sort of did badly at the beginning, but then recovered and managed to do well, then we can also reinforce that recovery. And so that's super important because it allows the agent to actually learn from, you know, its own distribution of behavior.
>> Mhm. And that just makes it much more robust because now he doesn't have to generalize to something he has never seen before. He can actually learn, you know, on the actual problem that he is trying to solve. And that's why, you know, RL is really unlocking so much agentic capabilities. Now.
>> If I'm an AI builder today, building an AI app, and I I build it on top of Anthropic. Uh, Anthropic is whatever model is going to come with some of this sort of batteries included. Uh, but as a builder on top, do I need to do my own RL? Uh, is there this emerging space of like RL as a as a service where, you know, for this task or that task that I build on top of a general model that sort of like offers you the ability to do RL? Or or can I do a lot of damage just through prompt or like maybe like supervised fine-tuning first?
>> I think nowadays with the capabilities of like, you know, Anthropic cloud models, top OpenAI GPT models, you don't need to do any fine-tuning. You can take the model as is, write your own tools, your own harness, and benefit from that agentic training because doing good agentic fine-tuning is actually very hard. And so it's it's quite hard to do better than the top frontier models that you might get. But on the contrary, coming up with good tools and a good representation of your task makes a huge difference. So like, you know, depending on how you express your problem for the model can make it way harder or way easier, and so you can get a lot of mileage out of that.
What's currently missing to achieve the big dream of agentic AI? Is it model capabilities at the core, or is it sort of like boring, quote and quote, engineering around reliability, tool use, safety? Uh, what needs to happen?
>> I think we have sort of there's basically improvements needed around the whole space. Make, you know, the model better able to correct its own errors. Make the model better able to continue going for long times without getting distracted. Making the model just smarter in general. Maybe making the model faster. Like there's basically like, you know, a whole set of things that we know that we can improve. There's probably, yeah, not one individual blocker, and that's why we will continue to see sort of smooth incremental progress over model releases. But sort of given how many things we know there are that we can do better on and improve, yeah, I'm quite excited about where models are going to end up. I think that's actually, you know, one of the reasons why AI is a very fun field is that there are so many low-hanging fruits that, you know, you can do much better on, but already the current models are so good that it's very fun to work on it. It's like, oh, I can fix this thing. It'll be even better.
>> Versus, you know, if you're in a place where everything has already been solved and it's like really hard to figure out how to make it better, it's a very different story.
Let's spend a minute on on eval, uh, and, um, we we touch upon this a little bit, but just to give it some some proper space. So there was, um, you know, in your blog post that we talked to at the very beginning of this conversation, this this concept of external benchmark, and then you quoted your piece, Goodhart's Law. Uh, so what does first of all, what is Goodhart's Law? And then, uh, how should labs compare results, uh, so that it doesn't end up with this kind of leaderboard theater that we've seen a little bit in the last, you know, couple of years?
>> Yeah. As Goodhart's Law basically says that any measure that becomes a target stops being a good measure. And, you know, you can think of that intuitively that if you start paying, for example, programmers based on how many lines of code they write, well, suddenly they will discover many ways to add more lines of comments, which is, you know, completely useless. And this is a very, yeah, general effect that obviously, right? If you give people an incentive that they should optimize, they will try very hard. Yes.
>> And we also see this with language model benchmarks. Of course, people want to get promoted. They want to launch their model. So any benchmark that is too easily measured or that has a lot of attention on it, people will optimize very hard for it. Which means that probably the model will look very good at that benchmark. But if you then use it for your own task, you might get different performance. Yeah.
You ask about like how what do we do about this? It's very hard to prevent people from optimizing on the benchmark. So one possibility is just periodically create completely new held-out benchmarks that nobody has seen before, and that gives you a fairly, you know, good estimate of model performance. So I know, for example, like a lot of researchers have their own toy problems that they use to test all the models precisely for that reason. So that, you know, this is a problem, a set of problems that nobody has seen. You have a pretty good guess that it's going to give you an unbiased estimate. If you're like, you know, an individual of your company trying to decide which model to use, it's probably, you know, something similar. Just make your own internal benchmark that really represents what you care about, and then measure on that. And I think that's likely to be the most objective, most accurate way of measuring.
Internally. What does that look like at a place like Anthropic or previously, uh, DeepMind? Uh, is there, um, I mean, I know there there are teams that are focused on on evals. How do you, how do you think about what works, what doesn't in terms of internal evals?
>> It definitely used to be easier to have good evals. You know, five years ago, the tasks were doing, I think it was easier to measure model performance. I think nowadays it's much more difficult, and I think we try to not over-rely on evals so much because it's quite hard, for example, to measure, you know, how good is this model really at writing code. Yeah, I think it's one of the big unsolved or very important problems in the field of, you know, making really good evals that are both cheap to run, reliable, and accurate. Because it's easy-ish to make an eval that takes one of those, but to get all three is quite hard. For example, like, you know, in the beginning, we were talking about open GBT eval, GBT eval, and that one is like, you know, it's very accurate and unbiased, but it's very expensive to because what it actually involves is like taking human experts, having them do the task, >> and then compare the model task to the experts, and like, you know, rate it with multiple people. So it's very accurate, but it's like extremely expensive to do.
And related to that topic of of evals, what what's the latest in terms of our ability, or I should say your ability, uh, to truly understand, uh, how models work? To the general field of mechanistic interpretability. You alluded to the fact earlier that, um, RL, if I understood correctly, sometimes makes it a bit harder because it does, uh, things occasionally in a more inscrutable way. My my words, maybe not not yours. Um, so what is the latest, and indeed, does RL make things harder or easier?
>> Oh, so what I meant before is that debugging RL in general, know, completely unrelated to interpretability, >> is harder because there are more moving parts. But it is also true that if you're not careful with RL, you can make interpretability harder. For example, one common thing with modern models is they they do reasoning with the chain of thought. You could look at the chain of thought to, you know, see what are the model's internal thoughts, and then you could also have a thought that, oh, maybe I should use that as a reward signal in RL and punish the model if it thinks the wrong thing. But then suddenly you completely destroyed your interpretability angle. So you sort of have to be careful that, yeah, you don't do RL on the signals that you want actually want to use to interpret what the model is thinking of doing.
That said, I think, yeah, there's some extremely exciting interpretability things happening, including mechanistic interpretability. I think like actually last year, I think before Johnic, maybe even, there was a super cool Golden Gate claw model where, you know, they found the neurons in claw that were responsible for the Golden Gate concept, and then modified them to make a version of claw that really loved the Golden Gate Bridge in San Francisco. And so that's like a really vivid example of, ah, you know, we really understand what's happening in this model, and like, you know, what better way is there to verify that understanding than actually changing the behavior of the model? And so I think that's like a super important direction for safety. As the models get smarter, we really need to be able to understand what is the model thinking internally. You know, what are the values it has? Is it lying to us? Is it actually genuinely following the instructions? And so I think like, you know, definitely extremely important area to invest in and work in. I think that especially if like, you know, people interested in working AI or doing AI research, I think interpretability is a great area to get into.
Yeah, perfect segue for last for the last part of this conversation. I'd love to to zoom out and and talk about, um, the impact of AI. So if we think that we are on the exponential and that things are going to only accelerate from here, what does that mean? Uh, and certainly safety and alignment, which is a core value at Anthropic, hopefully in other parts of the field as well, but like Anthropic is particularly uh vocal about safety and alignment, let's say. um, how how does that actually manifest? So we just talked about interpretability. What what, uh, for people who are concerned that this is going too fast and that we collectively are creating a a monster, quote unquote. Can can you give us a glimpse into the kind of work, uh, that is done for alignment and safety at a place like Anthropic?
>> Yeah, I think like the focus on sort of safety alignment pervades all of Anthropic, and there's very rigorous processes where we train a model whenever we want to release a model, both to, you know, analyze the capabilities of the model, verify the alignment of the model, ensure that it, you know, does not do harmful things on its own, ensure that it does not enable, you know, malicious users to do harmful things, and to the point where if we are unsure about the safety of a model, we will delay the launch and like, you know, until we're sufficiently sure that it is actually harmless, we will not launch and release a model, which, you know, may, you know, I guess shows that, you know, people take the safety much more seriously than any financial return or revenue. I think, yeah, also in terms of research and resources, the teams working on safety and interpretability are a big focus of the company, which, you know, gives me a lot of confidence that we're actually care about this and put a lot of effort into it.
And at a more technical level, and to, uh, tie back an earlier part of the conversation, um, around, uh, when when we're discussing the, you know, pre-training and and and and safety. So is safety and alignment an RL problem? Uh, and by that I mean, uh, the the beauty of having pre-training is that you import that world model as we're discussing, but arguably you also import into your brain, uh, a lot of bad stuff if you collect data from the internet, as we know, uh, there's good things, but also a lot of toxic content. So is, uh, alignment largely using RL to get rid of the bad stuff that is built into the pre-training?
>> We can definitely use RL to like shape the model behavior and ensure that, for example, given adversarial, given bad input, it sort of behaves safely or knows that it can refuse or is, you know, robust to attempts to hack the model. Yeah, I wouldn't view it alignment just like an RL problem. I think it sort of it goes throughout the whole stack. You might, you know, for example, filter the pre-training data in some way. You might after training, you might have classifiers that like, you know, look at the model, monitor the model behavior to ensure that it is actually aligned. You might, when you write the system prompt for the model that you use, you might put safety guidelines in there. So I think safety alignment, it really pervades the whole of research and the whole of, you know, product and deployment. It's not just isolated into any one part.
And then another super interesting topic in the same vein of like the impact of AI is obviously the discussion around jobs. So if as per the GDP discussion, the uh agents are becoming just as good or better than humans, obviously, what does that mean, uh, for all of us in terms of our jobs? What what have you learned, um, after the experience of Alpha Zero, Alpha Go, that that could give us a glimpse into, uh, what may happen once we all have super powerful agents do our jobs?
>> So I think the first thing that we didn't talk about yet so far is that artificial intelligence is quite, I mean, this may sound a bit simplistic, but it's quite different than human intelligence. So we can see that, right, that the model may be much better at us on some tasks, like, you know, calculation, obviously, and like much worse than us at other tasks. So it is not, I don't think it is at all going to be any like one for one replacement. It's going to be much more complimentary of, you know, the model is really good at something that maybe I really don't like doing or I'm not interested in or I'm very bad at, and then I'm much better than the model as a mother part. And so I think it's going to be like a gradual process of we're all going to incrementally start using models more and more to improve our own productivity rather than, you know, have a model that one for one is able to do exactly the set of things we can do. And so, for example, you know, I use cloud all the time to, for example, you know, refactor code or maybe write some front-end code that I don't want to write, or at the same time, there's other parts where I'm clearly much better putting than cloud still. So there is a synergy of, you know, use the best, most productive skills. I think I guess economists call it like comparative advantage, but like there is this, you know, long process of we'll both sort of improve our productivity incrementally. And I think that process is going to give us some time of figuring out politically and figuring out economically how do we want to, you know, benefit from this massive productivity increase. You know, even independently from AI, right? The promise of technology has long been that, oh, we're going to be all so productive, so wealthy that we need to work much less.
>> Yet, mysteriously, right, we all have like 40 hours working week for decades. And so, you know, I think it's much more like a political social problem of like figuring out how do we actually benefit from all these improvements and like, you know, bring the increases in wealth and productivity to everybody. And it's much less a technological problem.
>> Mhm. Also means that we can't really solve it with technology. We have to solve it at like a sort of a democratic political level. How do we spread these benefits?
>> Do you think that that, uh, increases inequality? So, uh, as you think about the impact of Alpha Go and and and and Mu Zero, what happened to the top Go players? And what happened to the top chess players? Did they do they disappear or did they get enhanced and better?
>> Yeah, I think at least in the case of chess and Go, there has been like more interest, and it has become much easier for people to study how to play Go, how to play chess, cuz now you don't need to find, you know, an expert tutor. You anybody can practice on their own, right? Spend a lot of time, and I guess like chess streamers are very popular on Twitch, right? Here. And similarly, right, like a lot of students are using language models to study. I think also for coding, right, cloud code, these agents, they raise the bar of what anybody who has an idea can accomplish on their own. I think, you know, the larger picture whether it increases or decreases inequality is quite hard to forecast. It both sort of raises the floor of what any person can accomplish, but it also gives very productive people an ability to be even more productive. It's possible that we see quite a difference between countries depending on the taxation, social redistributive system that they have, and whether inequality increases or decreases. For example, overall, I'm quite excited that it is very much non-zero sum. It is very much, you know, increases the total wealth available in society. I think if you think about progress, if you think about prosperity, that is the most important thing. Like redistributing the pie is kind of a loser's game. To get more wealthy, we really need to grow the pie. You know, if you think of the agricultural revolution, the industrial revolution, the reason why we have much better lives nowadays is because, you know, we're so much more productive. We have so much more wealth. And so that's the key step we want to unlock. If we manage to make everybody in society 10 times more productive, you know, what kind of abundance can we achieve? I think that's the key question, right? What advances does that unlock in medicine? You know, curing diseases, halting aging. What does it unlock in terms of energy? Obviously, right, we have like climate crisis. We need more energy rights to sustain our lifestyle. What advances in material science can we have? All of those are basically bottlenecked on how much intelligence we have access to and how can we apply it. So I'm, yeah, incredibly optimistic about like what we will be able to unlock in the next five years. I think we can go extremely far.
>> Well, that feels like a a wonderful place to leave it. Uh, thank you so much, Julian. This was absolutely fantastic. Thank you for spending time with us.
>> Yeah, thank you for all the exciting questions and, uh, giving me the time.
Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks, and see you at the next episode.