What the hell is going on inside neural networks? | Chris Olah

Hi, listeners. This is the 80,000 Hours Podcast, where we have unusually in-depth conversations about the world's most pressing problems, what you can do to solve them, and how come there's a cloud there, but not a cloud right there, just next to it?

I'm Rob Wiblin, head of research at 80,000 hours. I am really excited to share this episode because Chris Olah, today's guest, is one of the top machine learning researchers in the world. He's also really excellent at communicating complex ideas to the public, with his blog posts and Twitter threads over the years attracting millions of readers. A survey of listeners even found that he was one of the most followed people among subscribers to this show.

And yet, despite being a pretty big deal, this is the first podcast, and indeed the first long interview that Chris has ever done. Fortunately, I don't think you'll be able to tell that he hasn't actually done this many times before. We ended up having so much novel content to cover with Chris that we did more than one recording session. And the end result is two very different episodes that we're both really happy with.

The first one, this one, focuses on Chris's technical work and explores topics like what interpretability research is and what it's trying to solve; how neural networks actually work and how they go about thinking; multimodal neurons and the implications for AI safety work; whether the approach that Chris is taking can scale; digital suffering; scaling laws in machine learning models; and how wonderful it would be if all of the work that is described here could succeed.

If you find the technical parts of this episode a bit hard going, before you give up on this episode altogether, I'd recommend skipping to the chapter called Anthropic and the Safety of Large Models, or going to 2 hours and twelve minutes in to hear all about the really exciting project that Chris is helping to grow right now. You don't have to be an AI researcher to work at Anthropic, so I think people from a wide range of backgrounds could really benefit from sticking around to the end.

For what it's worth, though, I am far from being an expert on how AI works, but I was mostly able to follow Chris and so learn quite a bit about what's really going on with research into big machine learning models.

The second episode, which, if everything goes smoothly, we hope to release next week, is focused on Chris's really fascinating personal backstory, including how he got where he is today without having a university degree.

Oh, and one final thing: eagle-eared listeners will notice that our audio changes a couple of times here, and that's because, as is often the case with our longer episodes, Kieran has cut together sections from multiple different recording sessions to try and create a better final product.

All right, without further ado, I bring you Chris Olah. Today I'm speaking with Chris Olah. Chris is a machine learning researcher currently focused on neural network interpretability. Until last December, he led OpenAI's interpretability team, but this year, he has helped to launch a new AI lab known as Anthropic, which is particularly focused on the safety of very large ML models. Before Open AI, he spent four years at Google Brain developing tools to visualize what's going on in neural networks.

And Chris had a big impact at Google Brain. He was the second author on the launch of Deep Dream back in 2015, something which I think almost everyone has seen at this point. And he also pioneered feature visualization; activation analysis; building blocks of interpretability; TensorFlow; and even coauthored the famous paper Concrete Problems in AI Safety. On top of all of that, in 2018, he helped found the academic journal Distill, which is dedicated to publishing clear communication of technical concepts. And Chris is himself a writer who is popular among many listeners to this show, having attracted millions of readers by trying to explain cutting-edge machine learning in highly accessible ways.

In 2012, Chris took a $100,000 Thiel Fellowship, a scholarship designed to encourage gifted young people to go straight into research or entrepreneurship rather than go to a university. So he's actually managed to do all of the above without a degree.

Thanks so much for coming on the podcast, Chris.

Thanks for having me.

All right, I hope that we're going to get to talk about your new project, Anthropic and the Interpretability Research, which has been one of your big focuses lately. First off, it seems like you kind of spent the last eight years contributing to solving the AI alignment problem in one form or another. Can you just say a bit about how kind of you conceive of the nature of that problem at a high level?

When I talk to other people about safety, I feel like they often have pretty strong views or developed views on what the nature of a safety problem is and what it looks like. And I guess I feel like one of the lessons I've learned trying to think about safety and work on it and think about machine learning broadly has been how often I seem to be, in retrospect, I'm wrong, or I think that I wasn't thinking about things in the right way. And so I think I don't really have a very confident take on safety. It seems to me like it's a problem that we actually don't know that much about, and that trying to theorize about it a priori can really easily lead you astray. And instead, I'm very interested in trying to empirically understand these systems, empirically understand how they might be unsafe and how we might be able to improve that. And in particular, I'm really interested in how we can understand what's really going on inside these systems, because that seems to me one of the biggest tools we can have in understanding potential failure modes and risks.

All right. Well, in that case, let's waste no time on big picture theoretical musings and we can just dive right into kind of the concrete, empirical, technical work that you've been doing, which in this case is kind of reverse engineering neural networks in order to look inside them and understand what's actually going on.

The two online articles that I'm going to be referring to most here are the March 2020 article Zoom In: An Introduction to Circuits, which is the first piece in a series of articles about the idea of circuits, which we'll discuss in a second. And also the very recent March 2021 article, Multimodal Neurons in Artificial Neural Networks. Obviously, I'm willing to both of those, and they've got lots of beautifully and carefully designed images. So potentially worth checking out if listeners really want to understand all of this thoroughly. They're both focused on the issue of interpretability in neural networks, visual neural networks in particular. And I won't lie to you, those articles were kind of pushing my understanding and I didn't follow them all completely. But maybe Chris can help me get it properly here.

First, though, can you explain what problem this line of research is aiming to solve at kind of a big picture level?

Well, in the last couple of years, neural networks have been able to accomplish all of these tasks that no human knows how to write a computer program to do directly. So we can't write a computer program to go and classify images, but we can write a neural network to create a computer program that can classify images. We can't go and write computer programs directly to go and translate text highly accurately, but we can train a neural network to go and translate text much better than any program we could have written. And it's always seemed to me that the question that sort of is crying out to be answered there is, how is it that these models are doing these things that we don't know how to do? So the question that I study, and I think a lot of people maybe would debate exactly what interpretability is, but the question that I'm interested in is how do these systems accomplish these tasks? What's going on inside of them? And imagine if some alien organism landed and could go and do these things. Everybody would be going and rushing and falling over themselves to figure out how the alien organism was doing things. And you'd like, have biologists fighting each other for the right to go and study these alien organisms. Or imagine that we discovered some binary just floating on the Internet in 2012 that could go and do all these things. Everybody would be rushing to go and try and reverse engineer what that binary is doing. And so it seems to me that really the thing that sort of is calling out at all this work for us to go and answer it is what in the wide world is going on inside these systems.

Yeah. And what happens? What are the problems with not understanding it? Or I guess, what would be the benefits of understanding it?

Well, I feel kind of worried about us going and deploying systems, especially systems in high-stakes situations or systems that go and affect people's lives when we don't know how they're doing the things they do or why they're doing it and really don't know how to reason about how they might behave in other situations or in unanticipated situations. To some extent, you can get around this by testing the systems right, which is what we do. And we try to go and test them in sort of all sorts of cases that we're worried about how they're going to perform and on different data sets. But that only gets you to the cases that are covered in your data sets or that you explicitly thought to go and test for. And so I think we actually have a great deal of uncertainty about how these systems are going to behave. And I think especially as they become more powerful, you have to start worrying about what if in some sense, they're doing the right thing, but they're sort of doing it for the wrong reasons; they're implementing the correct behavior. But maybe the actual algorithm that is underlying that is just trying to get reward from you in some sense rather than trying to help you. Or maybe it's relying on things that are biased and you didn't realize that. And so I think that being able to understand these systems is a really important way to go and try and address those kinds of concerns.

Yeah, okay, so I guess, yeah, understanding how it's doing what it's doing might make it a bunch easier to predict how it's going to perform in future situations or if the situation changes; the situation in which you're deploying it. Whereas if you have no understanding of what it's doing, then you're kind of flying blind and maybe the circumstance could change, I think what do they call it? Change of domain.

Sure.

Yeah. So you might be worried about distributional shift, although in some ways I think that doesn't even emphasize quite enough what my biggest concern is or I feel like maybe it makes it sound like this kind of very technical, robustness issue. And I think that what I'm concerned is sort of something more like there's some sense in which modern language models will sometimes lie to you in that there's questions to which they know the right answer in some sense. Like if you go and pose the question the right way, they will give you the correct answer, but they won't give you the right answer in other contexts. And I think that's like sort of an interesting microcosm for a world in which these models in some sense they're capable of doing things but they're trying to accomplish a goal that's different than the one you want and may in sort of unanticipated ways go and cause problems for you.

I guess another way that I might frame this is sort of what are the unknown safety problems? What are the unknowns of going and deploying these systems? And if there's something that you sort of anticipate as a problem, you can go and test for it. But I think as these systems become more and more capable, they sort of will sort of abruptly sometimes change their behavior in different ways. And there's all these unknowns. And how can you hope to go and catch those? I think that's a big part of my motivation for wanting to study these systems or how I think studying these systems can make things more safe.

Let's just maybe give listeners a really quick reminder of how neural networks work. We've covered this in previous interviews, including a recent one with Brian Christian. But I guess basically you can imagine information flowing between a whole bunch of nodes and the nodes are connected to one another and which nodes are connected to which other ones, kind of like which neurons are connected to other neurons in the brain, is determined by this learning process. And I guess the weightings that you get between these different neurons is determined by this learning process. And then when a neuron gets enough positive input, then it tends to fire and then pass on a signal onto the next neurons, which is kind of, in a sense, in the next layer of the network. And so information is kind of processed and passed between these neurons until it kind of spits out an answer at the other end. Is that a sufficiently vague or sufficiently accurate description for what comes next?

Yeah, I mean, if I was going to nitpick slightly, usually which neurons connect to which neurons doesn't change; that's usually static and the weights evolve over training.

Okay, so the weights can just rather than disappear, the weights just go to a low level?

Yeah.

Okay. As I understand that based on reading these two pieces, you and a bunch of other colleagues at Google Brain and OpenAI have made pretty substantial progress on Interpretability. And this has been a pretty big deal among ML folks, or at least some ML folks. How long have you all been at this goodness?

I've been working on Interpretability for about seven years, on and off. I've worked on some other ML things sort of interspersed between it, but it's been the main thing I've been working on. And I guess originally it was a pretty small set of people who were interested in these questions. But over the years I've been really lucky to find a number of collaborators who are also excited about these types of questions. And more broadly, a large field has grown around this, often taking different approaches than sort of the ones that I'm pursuing, but there's now a nontrivial field working on trying to understand neural networks in different ways.

What different approaches have you tried over the years?

I've tried a lot of things over the years, and a lot of them haven't worked. And honestly, in retrospect, it's often been obvious why. I tried at one point this approach of going in the sort of topological approach of sort of trying to understand how neural networks bend to data that doesn't scale to anything beyond the most trivial systems. Then I tried this approach of looking at how sort of trying to do dimension I reduction representations. That doesn't work super well either, or at least didn't for me. And to cut a long story short, the thing that I have sort of ended up settling on primarily is this almost stupidly simple approach of just going and trying to understand what each neuron does and how all the neurons connect together and how that gives rise to the behavior. And the really amazing thing is that as you start to understand what different neurons are doing, you actually start to be able to go and read algorithms off of the weights. There's a slightly computer science heavy analogy that I like, which might be a little bit hard for some readers to follow, but I often think of neurons as being like variables in a computer program or registers in assembly. And I think of the weights as being like code or assembly instructions. And you need to understand what the variables are storing if you want to understand the code. But then once you understand that, you can actually sort of see the algorithms that the computer program is running. And that's where things, I think, become really exciting and remarkable. I should add, by the way, about all of this, that I'm sort of describing this, and I'm the one here who's talking about it, but I should say that all of this has been the result of lots of people, and I'm just incredibly lucky to have had a lot of really amazing collaborators working on this with me.

Yeah, no doubt. So what can we do now that we couldn't do ten years ago, thanks to all of this work?

Well, I think we can genuinely understand how large chunks of neural networks work. We can actually reverse engineer chunks of neural network and understand them so well that we can go and handwrite weights. So you just take a neural network where all the weights are set to zero, and you write by hand the weights. And you can go and re-implant a neural network that does the same thing as that chunk of the neural network did. And then you can even splice it into a previous network and replace the part that you understood. And so I think that is really a very high standard for understanding systems. And I guess I sometimes like to imagine interpretability as being a little bit like, I don't know, maybe like cellular biology or something where you're trying to understand the cell. And I feel like maybe we're starting to get to the point where we can understand one small organelle in the cell or something like this, and we really can nail that down.

Okay. So we've gone from kind of having a black box to having a machine that you could potentially kind of redesign manually, or at least parts of it.

Yeah.

I think that the main story, at least from the circuits approach, is that we can take these small parts and they're small chunks, but we can sort of fully understand them.

Yeah. So a really key concept or two really key concepts here are kind of features and circuits.

Yeah. Can you explain for the audience what those are and how you pull them out?

Yeah. When we talk about features, we most often, though not always, are referring to an individual neuron. So you might have an individual neuron that does something like responds when there's a curve present or responds to a line or responds to a transition in colors. Or later on, we'll probably talk about neurons that respond to Spiderman or respond to regions of the world. So they can also be very high level. And a circuit is a subgraph of a neural network. So the nodes are features. So the node might be a curve detector and a bunch of line detectors. And then you have the weights of how they connect together. And those weights are sort of the actual computer program that's running that's going and building later features from the earlier features.

Okay, so features are kind of smaller things like lines and curves and so on. And then a circuit is something that puts together those features to try to figure out what the overall picture is. A circuit is a partial program that's building the later features from the earlier features. So circuits are connections between features and they contribute to building the later features from the earlier features.

I see. Okay. So I can imagine within, say, a layer of a neural network, you've got different neurons that are picking up features and then kind of the way that they go on to the next layer, the combinations of weights between these different kind of features that have been identified and how they push forward into features that are identified in the next layer. That's a circuit.

Yeah. Although often we'd look at a subset of neurons that are tightly connected together. So you might be interested in how, say, the curve detectors in one layer are built from features in the previous layer, and then there's a subset of the features at the previous layer that are tightly intertwined with that. And then you could look at that, and then you're looking at a smaller subgraph, and you might also go over multiple layers. So you might look at how there's actually this really beautiful circuit for detecting dog heads. And I know it sounds crazy to go and describe that as beautiful, but I'll describe the algorithm because I think it's actually really elegant. So Inception V1, there's actually two different pathways for detecting dogheads that are facing to the left and dogheads that are facing to the right. And then along the way they mutually inhibit each other. So at every step it goes and builds a better doghead facing in each direction and has it so that the opposite one inhibits it. So it's sort of saying a doghead can only be facing left or right, and then finally at the end it goes and unions them together to create a doghead detector that is pose invariant, that is willing to fire both for a doghead facing left and a doghead facing right.

Okay, so I guess you've got neurons that kind of correspond to features. So a neuron that fires when there's what appears to be fur and then a neuron that fires when there appears to be a curve of a particular kind of shape, and then a bunch of them are linked together and they either fire together or inhibit one another to indicate a feature in the next. At a higher level of abstraction, like, this is a dog head and then more broadly you say like, this is this kind of dog. It will be like the next layer along. Is that kind of right?

Yeah, that's structurally right.

Okay. Yeah. Interesting. Was it hard to kind of come up with the technical methods that you use to identify what feature a neuron corresponds to and kind of how do you pull together all of the different connections and weights that should be seen as functioning as a circuit?

Yeah, so there's this interesting thing where neural network researchers often look at weights in the first layer. So it's actually very common to see papers where people look at the weights in the first layer. And the reason is that those weights connect to red, green and blue channels in an image if you're say doing vision. And so those weights are really easy to interpret because you know what the inputs to those weights are and you almost never see people look at weights anywhere else in the model in the same kind of way. And I think the reason is that they don't know what the inputs and the outputs to those weights are. And so if you want to study weights anywhere other than the absolute input or maybe the absolute output, you need to go and have some technique for understanding what the neurons that are going in and out of those weights are. And so there's a number of ways you can do that. Like one thing you could do is you could just look about what we call data set examples, just like feed lots of examples through the neural network and look for the ones that cause neurons to fire. That would be a very maybe neurosciencey approach. But another approach that we found is actually very effective is to optimize the input. We call this feature visualization. And you optimize the input, you just do gradient descent to go and create an image, say, that causes the neuron to fire really strongly. And the nice thing about that is it separates correlation from causation in that resulting image. You know, that everything that's there is there because it caused the neuron to fire. And then we often use these feature visualizations both they're a useful clue for understanding what the neuron is doing, but we also often just use them sort of like variable names. So rather than having a neuron be mixed four C, four, seven, we say we have this image that stimulates it and it's a car detector. So you get an image of a car and that makes it much easier to go and keep track of all the neurons and reason about how they connect together.

Okay, so it seems like you've got this method where you choose a neuron or you choose some combination of connections that forms a circuit and then you kind of try to figure out what image is going to maximally make that neuron or that circuit fire. And you said gradient descent, which kind of, as I understand it, is like you choose like a noise image and then you just kind of inch bit by bit to get it to fire more and more. And more and more until you've got kind of an image that's pretty much optimally designed through this iterative process to really slam that neuron really hard. And so if that neuron is there to pick up fur, then you'll figure out the archetypal fur thing that this neuron is able to identify. Is that right?

Yeah, and then once you understand the features, you can use that sort of a scaffolding to then understand the circuit.

I see. Okay. So, yeah, I think I didn't realize this until I was doing research for this episode. But you've got this microscope website, at least at OpenAI, they're hosting it where you can kind of work through this big image recognition neural network and kind of pick out I don't know whether it's all of them, but it's like, many different neurons. And then see what are the kinds of images that cause this to fire? And you'll be like, oh, that's something that's identifying this kind of shape. Or this is a neuron that

Corresponds to this kind of fur color or whatever else. And you can see how this flows through the network in quite a sophisticated way. It really is pulling apart this machine and seeing what each of the little pieces does.

Yeah, the microscope is wonderful. It allows you to go and look at any neuron that you want. And I think it actually points to a sort of deeper underlying advantage or thing that's nice about studying neural networks, which is we can all look at exactly the same neural network. And so there can be these sort of standard models where every neuron is exactly the same as another model, as the model that I'm studying. And we can just go and have one organization, in this case OpenAI, go and create a resource once that makes it easy to go and reference every neuron in that model and then every researcher who's going and studying that model can easily go and look at arbitrary parts of it just by navigating to a URL.

Yeah. By the way, why? So people have probably seen Deep Dream, and they might have seen, if they've looked at any of these articles, kind of the archetypal images and shapes and textures that are firing these particular feature neurons. But the colors are always so super weird. It produces this kind of surrealist Deep Dream thing. I would think if you had a neuron that was firing for fur then it would actually look like a cat's fur, or would it be a shop sign? If it's a shop sign, urine. But in fact, they kind of look like this crazy psychedelic experience. Is there some reason why that's the case?

After we published Deep Dream, we got a number of emails from, I think quite serious neuroscientists, asking us if we wanted to go and study with them whether we could explain experiences that people have when they're on drugs using Deep Dreams. So I think that this analogy is one that really resonates with people. But why are there those colors? Well, I think there's a few reasons. I think the main one is you're sort of creating a super stimulus. You're creating the stimulus that most causes the neuron to fire, and the superstimulus. It's sort of not surprising that it's going to push colors to very extreme regimes because that sort of is the most extreme version of the thing the neuron is looking for.

There's also a slightly sort of more subtle version of that answer, which is suppose that you have a curve detector. It turns out that actually one thing the neuron is doing is it's just looking for any change in color across the curve because if there's sort of different colors on both sides, that sort of means that it's stronger. In fact, this is just generally true of line detectors. Very early on in neural networks, the line detectors align, fires more if there's different colors on both sides of the line. And so the maximal stimulus for that just says any difference in colors. It doesn't even care what the two colors are. It just wants to see a difference in colors.

Okay, so the reason that they can look kind of weird is just that these feature detectors are mostly just picking up differences between things and so they want like a really stark difference in color, and maybe the specific color that the thing happens to be doesn't matter so much. And so it doesn't have to look like actual cat fur. It's just kind of the color gradient that's getting picked up. Well, it depends on the case, and it depends a lot also on the particular neuron.

There's one final reason, which is a bit more technical, which is often neural networks are trained with a little bit of Hue rotation, sometimes a lot of Hue rotation. Which means that the images, the colors are sort of randomly shuffled a little bit before the neural network sees them. And the idea is that this makes the neural network a bit more robust, but it makes neural networks a little bit less sensitive to the particular color that they are seeing. And, yeah, you can actually often tell how much Hue rotation a neural network was trained with by exactly the color patterns that you're seeing when you do these features.

Okay, so what concretely can we learn about kind of what or how a neural network system like this is thinking using these methods? Well, in some sense, you can sort of quite fully understand it, but you sort of understand a small fraction. So you can understand what a couple of neurons are doing and how they connect together. You can get it to the point where it's just sort of your understanding is just basic math, and you can just sort of just basic logic can reason through what it's doing. And the challenge is that it's only a small part of the model, but you literally understand what algorithm is running in that small part of the model and sort of what it's doing.

It sounded like from those articles that you think we've potentially learned some really fundamental things here about how neural networks work and potentially not only kind of how these thinking machines work but also how neural networks might function in the brain and maybe how they always got them necessarily have to function.

Yeah. What are those things that you think that we might have learned? And what's the evidence for them?

Well, one of the fascinating things is that the same patterns form and the same features and circuits form again and again across models. So you might think that every neural network is its own special snowflake and that they're all doing totally different stuff, and that would kind of make for a boring or a very strange field of trying to understand these things. Because you'd imagine another analogy I sometimes like is to think of interpretability as being like anatomy, and we're, like, dissecting a little bit strange because I'm a vegan, but it's grown on me over time. We're dissecting these neural networks and looking at what's going on inside them. Like an early anatomist might have looked at animals. And if you imagine early anatomy and just like every organism had a totally different anatomy and there was no similarities between them, there's nothing like a heart that exists in lots of them that would create a kind of boring field of anatomy. Like you probably wouldn't want to invest in trying to understand most animals. But just like animals have very similar anatomies, I guess in the case of animals, due to evolution, it seems like neural networks actually have a lot of the same things forming, even when you train them on different data sets, even when they have different architectures, even though the scaffolding is different, the same features and the same circuits form, and actually, I find the fact that the same circuits form sort of like to be the most remarkable part where, I don't know, the fact that the same features form is already, I think, pretty cool. The neural network is learning the same sort of fundamental building blocks of understanding vision or understanding images, but then it's actually like literally, even though it's scaffolded on differently, it's literally learning the same weights, connecting the same neurons together. So we call that universality, and that's pretty crazy. And it's really tempting when you start to find things like that to think that, oh, maybe these same things form also in humans. Maybe it's actually something fundamental. Maybe these models are discovering sort of the basic building blocks of vision that just slice up our understanding of images sort of in this very fundamental way. And in fact, for some of these things, we have found them in humans. So some of these lower-level vision things seem to mirror the results from neuroscience. And in fact, in some of our most recent work, we've discovered something that was previously only seen in humans, these multimodal neurons. We'll talk about multimodal neurons in just a second.

So it sounds like you're saying you train lots of different neural networks to identify images, and you train them on different images, and they have different settings. Are you putting them in? But then kind of every time you notice these common features, you've got things that are identifying particular shapes and curves and particular kinds of textures. And I suppose also things are organized in such a way that there are kind of feature-detecting neurons and then these organized circuits for moving from lower-level features to higher-level features. That just happens every time, basically. And then we have some evidence you think that this is matching things that we've learned about how humans process visual information in the brain as well. So this is like there's a prima facial case, although it's not certain that these universal features are kind of always going to show up.

Yeah. And it's almost tempting to think that there's something like, I don't know, that there's just like, fundamental elements of vision or something that just sort of are the fundamental things that you want to have. If you reason about images that like a line is in some sense a fundamental thing for reason about images, and that these sort of fundamental elements or building blocks are just always present.

Is it possible that you're getting these common features and circuits because we're kind of training these networks on very similar images? It seems like we love to show neural networks, cats and dogs and grass and landscapes and things like that. So possibly it's more of a function of the images that we're training them on than something fundamental about vision.

Yeah. So you could actually look at models that are trained on pretty different data sets. Like, there's this data set called Places, which is sort of recognizing buildings, basically, and scenes, and those don't have dogs and cats, and a lot of the same features form. Now, it tends to be the case that as you go higher up, the features are more and more different. So in the later layers, you have actually these really cool neurons related to different visual perspectives. And what angle am I looking at a building from, which you don't see in ImageNet models, and vice versa? ImageNet models have all these features related to dogs and cats that you don't see in the Places model. But there do seem to be a lot of things that are fundamental, especially in earlier vision that form in both of those data sets.

Yeah. So I guess yeah, a question would be if aliens trained a model or if we could look into an alien brain somewhere, would it show all of these common features? And I suppose maybe it does just make a lot of intuitive sense that a line, an edge, a curve, these are, like, very fundamental features. So even on other planets, they're going to have lines. But maybe at the high-level stuff, some of the features and some of the circuits are going to be going to correspond to particular kinds of things that humans have and that exist in our world, and they may not exist on another planet.

Yeah, that's exactly right. And I think it's the kind of thing that is purely supposition but is really tempting to imagine, and I think a kind of exciting vision for what might be going on here. I do find that the stories of investigating these systems does feel just like, also emotionally more compelling when I think that we're, like, actually discovering these, like, deeply fundamental things rather than these things that just happen to be arbitrarily true for a single system.

Yeah. You've discovered the fundamental, primitive nature of a line.

Yes. Or, I mean, I think it's more exciting when we discover these things that people haven't seen before or that people don't guess these things called high-low frequency detectors that seem to form in all models. They're just what the name suggests. They look for high-frequency patterns on one side of the receptive field and low-frequency patterns on another side of the receptive field.

Yeah. I didn't initially understand what those were. What do you mean by high frequency and low frequency here, so a pattern. For instance, with lots of texture to it and sharp transitions might be high frequency, whereas a low-frequency image would be very smooth or also an image that's out of focus will be low frequency.

Right, so it's picking up how sharp and how many differences there are within that section of the image versus like another part of it, exactly. It seems like the models use these as part of boundary detection. So there's a whole thing of using multiple different cues to do boundary detection, and one of them is these transitions in frequency. And so, for instance, you could see this happen at the boundary of an object where the background is out of focus and it's very low frequency, whereas the foreground is higher frequency. But you also just see this maybe like, just if there's two objects that are adjacent and they have different textures, you'll get a frequency transition as well. And in retrospect, it makes sense that you have these features, but they're not something that I ever heard anyone predict in advance. We'd find. And so it's exciting to discover these things that weren't predicted in advance and that both sort of suggest that you might have said, oh, these attempts to understand neural networks they're only going to understand the things that humans guessed were there. It sort of refutes that. But it also suggests that maybe there are just like, all of these we're digging around inside these systems, and we're discovering all of these things, and it's just like this entire wealth of things to be discovered that no one's seen before. I don't know. Maybe it's like we're discovering mitochondria for the first time or something like this, and they're just there waiting for you to find them.

Yeah, I guess. The universality kind of conjecture that you're all toying with would be that at some point we'll look into the human brain and find all of these high-low detector neurons and circuits in the human brain as well. And they're playing an important role in us identifying objects when we look at in the world.

Yeah, that would be the strong version. And also just that every neural network we look in that's doing sort of vision, natural image vision is going to go and have them as well. And they're really just this fundamental thing that you find everywhere.

Yeah. Something I just thought of as a way that you could potentially get a bit more evidence for this universality idea is obviously humans have some bugs in how they observe things. There's ways in which they can get tricked visually. And maybe we could check, do the neural networks on the computers also have these same errors? Do they have the same ways in which they get things right and get things wrong, and then that might suggest that they're perhaps following a similar process?

Yeah, and I think there's been a few papers that are exploring ideas like this. And actually, in fact, when we talk about the multimodal neuron results, we'll have one example.

All right, well, that is a perfect moment to dive into the multimodal neuron article, which just came out a couple of weeks ago, and I guess is dealing with more kind of state-of-the-art visual detection models. What did you learn in that work that was different from what came before?

So we're investigating this model called CLIP from OpenAI, which you can roughly think of as being trained to caption images or to pair images with their captions. So it's not classifying images. It's doing something a little bit different. And we found a lot of things that were really deeply qualitatively different inside it. So if you look at low-level vision, actually a lot of it is very similar, and again, is actually further evidence for universality. A lot of the same things we find in other vision models occur also in early vision in CLIP. But towards the end, we find these incredibly abstract neurons that are just very different from anything we'd seen before. And one thing that's really interesting about these neurons is they can read. They can go and recognize text and images, and they fuse this together. So they fuse it together with the thing that's being detected. So there's a yellow neuron, for instance, which responds to the color yellow, but it also responds if you write the word yellow out, it will fire as well. And actually, it'll fire if you write out the words for objects that are yellow. So if you write the word lemon, it'll fire, or the word banana, it'll fire. And this is really not the sort of thing that you expect to find in a vision model. It's in some sense, it's a vision model, but it's almost doing linguistic processing in some way, and it's fusing it together into what we call these multimodal neurons. And this is a phenomenon that has been found in neuroscience. So you find these neurons also for people. Like, there's a Spider-Man neuron that fires both for the word Spider-Man as an image, like a picture of the word Spider-Man and also for pictures of Spider-Man and for drawings of Spider-Man. And this mirrors a really famous result from neuroscience of the Halle Berry neuron or the Jennifer Aniston neuron, which also respond to pictures of the person and to drawings of the person and to the person's name. And so these neurons seem, in some sense, much more abstract and almost conceptual compared to the previous neurons that we found. And they span an incredible wide range of topics. In fact, a lot of the neurons, you just go through them, and it feels like something out of a kindergarten class or an early grade school class. And you have your color neurons. You have your shape neurons. You have neurons corresponding to seasons of the year and months to weather, to emotions, to regions of the world, to the leader of your country, and all of them have this incredible abstract nature to them. So there's like a morning neuron that responds to alarm clocks and times of day that are early and to pictures of pancakes and breakfast food and all of this incredible diversity of stuff or season neurons that respond to the names of the seasons and the type of weather associated with them and all of these things. And so you have all this incredible diversity of neurons that are all incredibly abstract in this different way. And it just seems very different from the very relatively concrete neurons that were seeing before that often correspond to a type of object or such.

Yeah. So we've kind of gone from having an image recognition network that can say, recognize that this is a head, to having a network that can recognize that something is beautiful, or that this is a piece of artwork, or that this is an impressionist piece of art or something. It's a higher level of abstraction and grouping that lots of different images might have in common of many different kinds in many different places.

Yeah. It's not even about seeing the object. It's about things that are related to the object. For instance, you see a Barack Obama neuron in some of these models, and it, of course, responds to images of Barack Obama, but also to his name. And a little bit for U.S. flags and also for images of all these things that are associated with him. Also cause it to fire a little bit.

Yeah, when you put it that way, it just sounds like it's getting so close to doing what humans do because what we do, we're going about the world. It's just like we've got all of these associations between different things. And then when people speak, the things that they say kind of draw to mind particular different things with different weightings. And that's kind of how I feel like I reason, and language is very vague. Right. And lots of different words, they just have different associations, and you smash them together, and then kind of the brain makes of it what it will. So it's possible that it's capable of a level of conceptual reasoning, in a sense that is like, approaching a bit like what the human mind is doing.

Yeah. I feel really nervous to describe these as being about concepts because I think that's sort of a charged way to describe it that people find people might strongly disagree with, but it's very tempting to frame them that way. And I think that there's a lot of things about that framing that would be true.

Yeah. Interesting. Do you know off the top of the head kind of what is the most abstract category that you found? Is there anything that's particularly striking and amazing that it can pick up this grouping?

One neuron that jumps to mind for me is the mental health neuron, which you can roughly think of as firing whenever there's a cue for any kind of mental health issue in the image. And that could be body language or faces that read as particularly anxious or stressed. But for lots of words like the word depression or anxiety or things like this, the names of medicines associated with mental health, slurs related to mental health. So words like crazy or things like this, images that sort of seem like sort of, I don't know, like psychological sort of themed images a little bit and just this incredible range of things. And it just seems like such an abstract idea. There's no single very concrete instantiation of mental health, and yet it sort of represents that.

How is this model trained? I guess it must have a huge sample of images and then just lots of captions. And so is it using the words in the captions to recognize concepts that are related to the images and then group them?

Yeah, it's trying to pair up images and captions. So it'll take a set of images and a set of captions, and then it'll try to figure out which images correspond to which captions. And this has all sorts of really interesting properties where this means that you can just write captions and sort of use that to program it to go and do whatever image classification task you want. But it also just seems to lead to these much richer features on the vision side. And I think sometimes people are tempted to say that, oh, what it's doing? Is it like these neurons correspond to particular words in the output or something. But I think that's not actually going on. So, for instance, there's all of these region neurons that correspond to things like countries or parts of countries, or some of them are really big. Like there's an entire Northern Hemisphere neuron that responds to things like coniferous forests and deers and bears and things like this. And I don't think that those are there because they correspond to particular words like Canada or something like this. It's because when an image is in a particular context, that changes what people talk about. So you're much more likely to talk about maple syrup if you're in Canada or if you're in China, maybe it's much more likely that you'll mention Chinese cities or that you'll go and you'll have some of the caption be in Chinese or things like this.

Okay, so the training data was a set of corresponding images and captions. And it's developing these multimodal neurons in order to kind of probabilistically estimate or to prove its ability to tell what the context is, in order to figure out then what words are most likely to appear in a caption. Is that broadly right?

Yeah, that would be my intuition about what's going on.

What implications, if any, does the discovery of these kind of multimodal neurons have for safety or reliability concerns with these neural networks?

Well, I guess there's a few things. The first one sort of speaks to this unknown unknowns type concern where if you look inside the multimodal model or you look inside CLIP, you find neurons corresponding to literally every trait or almost every trait that is a protected attribute in the United States. So not only do you have neurons related to gender, related to age, related to religions, there's neurons for every religion. Of course, these region neurons connect closely to race. You have a neuron for pregnancy. You have a neuron for sort of parenthood that's looking for things like children's drawings and things like this. You have a neuron for mental health and another neuron for physical disability. And so almost every single protected attribute there is a neuron for. And despite the machine learning community caring a lot, I think, about bias and machine learning systems, it tends to be very alert and on guard for bias with respect to gender or with respect to race. And I think this is just that we should be looking for a much broader set of concerns and that it would be very easy to imagine CLIP going and discriminating with respect to parents or with respect to mental health or something like this that I think you wouldn't really have thought that previous models could. And I think that's kind of an illustration of how studying these things can surface things that weren't worried about before and perhaps should now be on the lookout for.

Yeah, I'm guessing if you tried to tell CLIP, naughty CLIP, we shouldn't be having multimodal neurons for race or gender or these other characteristics and then you said so you kind of struck them out, I bet when it was getting trained right, it would kind of shift those concepts. It would blow them into something else, like a geographic neuron or something else about behavior or personality or whatever. And so it would end up kind of accidentally building in the recognition of

Gender and race but just in a way that's harder to see. Yeah, I think it's tricky to get rid of these things. I think it's also not clear that the thing you want to get rid of—that you actually want to get rid of—is representation of these things. There's another neuron that sort of is like an offensiveness neuron. It responds to really bad stuff like swear words, obviously, racial slurs, pornography—yeah, really offensive things.

And on the one hand you might say, oh, we should get rid of that neuron. It's not good to have a neuron that responds to those things. On the other hand, it could be that part of the function of that neuron is actually to go and pair images that are offensive with captions that rebut them. Because sometimes people will post an image and then they will respond to that image. And that's the thing that you would like it to do. Like you would like it to be able to go and sort of recognize offensive content and rebut it or something like this.

And so my personal intuition is that we don't want models that don't understand these things. We sort of want models that understand these things and respond appropriately to them. I see. So it's better to kind of have it in the model and then potentially fix the harm that it would cause down the track using something that comes afterwards rather than try to tell the model not to recognize offensiveness. Because then you're more flying blind, or… It's not like we would say we should prevent children from ever finding out about racism and sexism and all of these things. Rather, we want to educate children that racism and sexism are bad and that's not what we want. And on how to sort of thoughtfully navigate those issues; that would be my personal intuition.

Yeah, that makes sense because that's what we're trying to do with humans, and maybe that's what we should try to do with the neural networks, given how human they are—they are seemingly becoming, I guess, just…

Sort of continuing on the safety issue. This is a bit more speculative, but we're seeing these emotion neurons form, and something I think we should really be on guard for is whenever we see features that sort of look like theory of mind or social intelligence. I think that's something that we should really keep a close eye on. And we see hints of this elsewhere. Like modern language models can go and track—track multiple participants in a discourse and sort of track their emotions and their beliefs and sort of write plausible responses to interactions.

And I think having the kind of faculties that allow you to do those tasks, having things like emotion neurons that allow you to sort of detect the emotions of somebody in an image, or possibly incorrectly reason about them, but attempt to reason about them, I think makes it easier to imagine systems that are manipulative, or are not sort of incidentally manipulative, but are sort of more deliberately manipulative. And it just would be—once you have social intelligence, that's a very easy thing to have arise. But I don't think the things that we're seeing are quite there. But I would predict that is a greater issue that we're going to see in the not too distant future. And I think it's something that we should have an eye out for.

Maybe there's a few other small things maybe to say. So one other thing is just—I think one thing that one should just generally be on guard for doing interpretive research, especially if one's goal is to contribute to safety—is just the fact that your guesses about what's going on inside a neural network will often be wrong. And you want to let the data speak for itself. You want to look and try to understand what's going on rather than assuming a priori and looking for evidence that backs up your assumptions. And I think that reach and neurons are really striking for this. Like about 5% of the last layer of the model is about geography, and if you'd asked me to predict what was in CLIP beforehand and what was like significant fractions of CLIP, I would not have guessed in 100 years that a large fraction of it was going to be geography. And so I think that's just another lesson that we should be really cautious of.

Yeah. So you're saying once you're trying to figure out how things might go wrong, you would want to know what is the network actually focusing on? And it seems like we don't have such great intuitions about that—the network can potentially be focused on features and circuits that are potentially quite different than what were anticipating beforehand.

Yeah. Or just an approach that I sometimes see people take to try and understand neural networks is they guess what's in there and then they look for the things they guessed are there. And I think that this is sort of an illustrative example of why—especially if your concern is to understand these unknowns and the things the model is doing that you didn't anticipate—and that might be problems. Here's a whole bunch of things that I don't think people would have guessed and that I think would have been very hard to go and catch with that kind of approach.

Yeah. Okay, so that's two different ways that the multimodal neural network work has been potentially relevant to safety and reliability. Yeah. Are there any other things that have been thrown up?

One other small thing which maybe is just less of a direct connection but maybe is worth mentioning is I think that a crux for some people in whether to work on safety or whether to prioritize safety is something like how close are these systems to actual intelligence, and are neural networks just fooling us? Are they just sort of illusions that sort of seem competent but aren't doing the real thing, or are they actually in some sense doing the real thing and we should be worried about their future capabilities? And I don't know. I think that there's a moderate amount of evidence from this to update in the direction that—no—neural networks are doing something quite genuinely interesting, and correspondingly, if that's a crux for you with regards to whether to work on safety, this is another piece of evidence that you might update on.

Yeah. I guess just the more capable they are and the more they seem capable of doing the kinds of reasoning that humans do, then that should bring forward the data which we would expect them to be deployed in potentially quite important decision-making procedures.

Yeah. Although I think a response that one can have with respect to evidence in the form of capabilities is to imagine ways the model might be cheating and imagine the ways in which the model may not really be doing the real thing, and so it sort of appears to be progressing in capabilities but that's an illusion—some might say it actually doesn't really understand anything that's going on at all. I mean, I think “understand” is a charged word there and maybe not very helpful, but the model in some sense is just an illusion and just a trick and won't get us to things that are genuinely capable. Is that right?

Well, I think it's hard to fully judge, but I think when we see evidence of these systems sort of implementing meaningful algorithms inside them, and especially when we see evidence of things that sort of have been perceived as evidence of human-like—a sort of neural evidence of humans understanding concepts—I think there's somewhat of an update to be had there.

Yeah. One other safety concern that came up in the article, at least as I understood it, was that people might be familiar with adversarial examples where you can potentially get a neural network to misidentify a sign as a bird or something like that, by modifying it. And by having these higher-level concepts like “electronics” or “beautiful” or whatever, it seemed like you could maybe even more easily get the system to misidentify something. So if you had the example where you had an apple and it would identify it as an apple, and then you put a Post-it note with the word “iPod” on it and it then identified this as an iPod—is that an important reliability concern that's thrown up? That maybe as you make these models more complicated, in fact, they can fail in ever more sophisticated ways?

Well, it was certainly a fun thing to discover. I've noticed it's blown up quite a bit, and I think maybe some people are overestimating it as a safety concern, where I wouldn't be that surprised if there was a relatively easy solution to this. The thing that to me seems important about it is—well, there's just this general concern with interpretability research that you may be fooling yourself that you're discovering these things, but you may be mistaken. And so I think whenever you can go and turn an interpretive result into a concrete prediction about how a system will behave, that's actually really interesting and sort of can give you a lot more confidence that you really are understanding these systems. And I think it is, at least for these particular models, without prejudice to whether this will be an easy or hard problem to solve, there are contexts where you might hesitate to go and deploy a system that you can fool in this easy way.

Yeah. Are there any examples where you've been able to use this interpretability work as a whole to kind of make a system work better or more reliably or anticipate a way that it's going to fail and then diffuse it?

Well, I think there are examples of catching things that you might be concerned about. I don't know that we've seen examples yet of that then translating into really compellingly ameliorating that concern or making changes to a system to make it better.

Yeah, I guess one hope you might have for a neural network is that you could sort of close the loop and make understanding neural networks into a tool that's useful for improving them and sort of just make it part of how neural network research is done. And I think we haven't had very compelling examples of that to date. Now, it'd be a bit of a—a little bit of a double-edged sword. Like, in some ways I like the fact that my research doesn't make models more capable or generally hasn't made models more capable and sort of has been quite purely sort of safety-oriented and catching concerns.

So when we saw the sort of questions from the audience for this interview with you, one person asked, why does Chris focus on small-scale interpretability rather than figuring out the roles of larger-scale modules in a way that might be more analogous to neuroscience? Seems like an appropriate moment, maybe, to ask this question: What do you make of it?

I think ultimately the reason I study—and I think it's useful to study—these smaller chunks of neural network is that it sort of gives us an epistemic foundation for thinking about interpretability. The cost is that we're going—and we're talking at these small parts—and we're setting ourselves up really, for now, a struggle to go and be able to build up to understanding large neural networks and make this sort of analysis really useful. But it has the upside that we're working with such small pieces that we can really objectively understand what's going on because at the end of the day, it sort of reduces just to basic math and logic and reasoning things through. And so it's almost like being able to go and reduce complicated mathematics to simple axioms and reason about things. And there's just, I think, a lot of disagreement and confusion, and I think it's just genuinely really hard to understand neural networks and very easy to misunderstand them, that having something like that seems like a really useful thing to have.

I feel like if maybe the radical success for circuits isn't that everyone talks about neural networks all the time in terms of circuits, or that's fundamentally how we analyze neural networks, but that it sort of takes on the role that having axioms does in mathematics—that it's kind of this foundation that everything can potentially be reduced down to.

Okay, so this work kind of shows that we can maybe—with a bunch of time—kind of go in and mechanically laboriously understand neural networks at this kind of circuit level, especially older ones that were a bit smaller. We can plenty understand a large fraction of those models, but neural networks have become a lot larger over the last few years and they're going to become a lot larger still before we actually start deploying them to important problems. Like modern language models are way bigger than these image recognition models are. That creates this challenge that maybe we can understand all these tiny circuit subcomponents, but there's just going to be so many of them that is going to be far beyond the capacity—know—Chris Olah or maybe even a whole team with you to understand any kind of meaningful fraction of these neural networks, which I guess you could kind of call this analysis scaling problem.

Yeah. What are the plausible ways in which we could scale the analysis so that we could actually understand these bigger models or maybe work around the problem in some way?

Yeah, I think this is a very reasonable concern and is the main downside of circuits. So right now, I guess probably the largest circuit that we've really carefully understood is about 50,000 parameters, and meanwhile the largest language models are in the hundreds of billions of parameters. So there's quite a few orders of magnitudes in difference that we need to get past if we want to even just get to the modern language models, let alone future generations of neural network. Despite that, I am optimistic. I think we actually have a lot of approaches to going and getting past this problem.

What are a couple of them? I guess? How many do you want to go through?

I think there's a good four or five that are worth going through.

All right, let's do the first one. So maybe actually the best place to start is just sort of the most naive thing, which is—if we were to just try and take the circuit approach sort of as is and try to scale it, do we have any hope of doing that? And it's not the thing that I would bet most on, but I actually think that it's more plausible than people might think. And there's two reasons for that. So the first reason is just that as neural networks become larger—there's more circuits to study, there's more features to study—but often the features and circuits in some ways become crisper and easier to understand.

So you're saying as the models become more sophisticated they have more parameters, but for that reason they're somewhat better at their job and so they are classifying things in a way that's clearer and more coherent to people.

Exactly. I think that sometimes when you have really weak models they're just representing things in a very confused way that entangles lots of things together and that actually makes them pretty hard to study. And it seems like often as models become stronger they actually sort of become, in some ways, crisper and less entangled and cleaner.

Is that in part because I suppose they don't have as many parameters as they might like to have? And so they have to cram a whole bunch of different concepts into kind of the same circuit and make it like double work.

Yeah, I think that is probably part of it. There's this problem we call polysemy, which is when a neuron is sort of fulfilling multiple roles like that. I think also they just may not be able to represent the actual abstractions that sort of cut a problem apart. They just literally don't have the computational capacity to build the right abstractions. And so they're working with suboptimal abstractions. There's sort of this interesting thing—it's a little bit different—but there's a paper by Jacob Hilton where they explore if you train models on progressively more diverse data, the features become sort of more interpretable as you do that. And so that's another thing that I think is sort of maybe the most rigorous result pointing in this direction that sort of suggests that there's some way in which as you have, in some sense, better models, they become more interpretable.

Okay, so they're more and more interpretable, but you're going to say something after that.

Well, just that sort of is a countervailing force against the problems of scaling. And then of course, there's a second thing, which is there's this really interesting thing we call a motif. And this is actually an idea we've borrowed from systems biology—there's Yuri Allen and a number of other people have really pioneered this approach to understanding biological networks in terms of these recurring patterns that they find. And it turns out we can find similar recurring patterns in neural networks, and those can really—they can actually simplify circuits by orders of magnitude.

Okay, so the idea here would be, even if it's very big, if it's just lots of recurring things that are basically all the same, then you can potentially understand a whole lot of it all at once. But yeah, what's an example of one of these motifs that kind of recurs through a network?

Well, a really simple one that we see is say, unions, where you have two different cases, and then you get a neuron that unions over those two cases, but that one doesn't give you a huge amount of traction in understanding things. The one that we've got sort of the most juice out of is what we call equivariance. And this is when neural networks have symmetries in them, where you have a feature and there's actually lots of copies of that feature that are transformed versions of the same feature. And if you have a whole bunch of features that have this property where, say, they're all rotated copies of the same feature and you have that across multiple layers, then actually the circuits themselves begin to have symmetries and you can understand—you can sort of simplify them by large integer factors. In the case of the curve circuits work really—you got a 50x simplifier, which is really nice.

So you're saying if there's lots of circuits that are recognizing the same thing in lots of different colors and lots of different rotations, then you could potentially look at all of those and just say, well, this is the curve-recognizing thing. And then they all end up spitting out at the same thing at another level, all saying, “this is curve.”

Yeah. They all connected together in the same way. And so you can just understand at once—and in one fell swoop you've actually understood a much larger amount of stuff. And so if you think about this happening to bridge many orders of magnitude, it's actually really encouraging. When you see things that are not just incremental improvements but actually are sort of these order-of-magnitude improvements, it sort of suggests that actually it's not completely a fool's errand that you might hope to go and bridge several orders of magnitude.

Okay. Yeah. Okay, so that's two ways that the problem of scaling might not be so hopeless. Yeah. What are some other approaches that you might have to bridging the gap?

Yeah, so I'd put both of those sort of in like the broad category of approaches where you're still sort of trying to make the basic circuit style approach work, and the rest of the ideas are not quite going to be that—they're going to be a little bit more different. What was our motivation for studying circuits?

Well, I think a big part of our motivation for studying circuits is to kind of be this epistemic foundation, and that doesn't mean that everything that we study needs to always be in terms of that foundation. Rather, the benefit is that it's giving us a way to sort of frame anything else we ask about in a very rigorous way. And actually when you study neural networks, you often see that there's these larger-scale structures. There's ways in which there's clusters of neurons that do very similar things, or there's ways in which actually you see parts of neural networks where all of the weights sort of have a very systematic pattern—sort of almost in a way that feels like a tissue in biology or something like this. And so there's all of these hints that there actually is a ton of large-scale structure, and you could imagine some future approach to interpretability where we study things in terms of this much larger-scale structure. And then perhaps when you find interesting things or things that are safety relevant in terms of that large-scale structure—like maybe you find some cluster of this larger thing that is involved in social reasoning or something like this and you're, like, worried that the model is perhaps manipulative—then you could look at that much more closely, and that could give you the ability to cut through many's orders of magnitude by going and just looking at very small parts that your large-scale analysis has told you are particularly important.

So if I understand you're saying—so maybe it's the case that kind of—again, using the analogy with the body—if circuits are kind of like cells, perhaps, but you'll notice that those cells are organized into other structures like tissue. And so you will be able to notice ways that circuits all aggregate together into some broader structure within the network. And then maybe there'll be a high-level structure again you'll see, which is perhaps like organs, and then you'll be able to work up. So you'll always be able to zoom in more into circuits and then features. But then you're just at the beginning of this process, and you're also going to identify other structures that will allow you to understand the full span of all of the orders of magnitude of size, rather than just these lower-level ones.

Yeah, that's exactly right. And I don't know if we're making analogy to medicine—maybe as an area where science actually has to be useful to people and solve problems—being able to understand things. Maybe you want to understand things in very fine grain—be able to go and understand things in very fine-grain detail. But if you know that there's a problem with the heart, you don't have to go and carefully analyze what's going on in the foot. And so you sort of both want the ability to look very closely at things and understand them very carefully, but also the sort of higher-level overview that you can sort of use to reason about what parts you need to pay attention to for particular kinds of problems and what parts you don't need to pay as much attention to.

Right. So if you're like, okay, we've got this enormous language model, but the part that we're really concerned about is the social manipulation part, so that we're going to really zoom in a lot of detail and understand it. And then the part that's just recognizing different pieces of fruit, we can let that one go because it's not a central…

Exactly. Or you might even imagine that there's problems that you can see in this larger-scale view, but that you are sort of able to develop this larger-scale view in a way that you trust because you had this foundation to build upon. And so you're able to use that to sort of reason about larger-scale things and to know that the things that you were talking about actually do map to what's going on—really going and genuinely occurring—in the model.

Yeah. While we're on this body thing, I thought one objection you might raise to this objection is it'd be kind of stupid to say, well, all you're doing is studying individual cells. The body is so much bigger than an individual cell. Like what can you learn about disease or what can you learn about the human body just by studying cells? Because the thing is the cells are replicated. The whole thing—other than bones, I suppose—it's made of cells—to be like, well, if we can understand the way that an individual cell functions and the way that it messes up, then we have really learned something about the whole because it's all this just recurring pattern.

Yeah. And also when you ask questions about tissues, if you're confused about what's going on—again, I'm not a biologist, so I'm sort of using all these biology analogies, and I might be distorting them a little bit—but I imagine it's really useful to be able to go and say—to ultimately be able to ask—if you think something is an issue with a tissue, what is going on at the cell level? Do we understand what's going on there? Or if we have some theory, can we validate it at the cell level? May be very helpful for places where you're confused.

Yeah. Another approach that people might suggest is—well, if we don't have enough humans to go and be analyzing all of these circuits—maybe we need to automate it and create new ML systems with their own circuits that analyze the circuits of other things and recognize what they are and go around them. What do you make of the idea of kind of automating this process somehow?

Yeah, I think that's totally an option. It's not my favorite option—at some level it's just aesthetic. I really like this approach of humans understanding things. I guess I also think that just a lot of…

The proposals, like you're for automating things, I don't yet fully understand them. And I think they often, in some ways, a lot of them sort of are sort of borrowing ideas from alignments that I think are not yet that mature, and then trying to combine them with ideas from circuits which are also not that mature.

And I guess I have this just nervousness that when you take a bunch of ideas that aren't mature, you're sort of making yourself even more vulnerable to things failing because you now have many points of failure. And so perhaps I'll be more excited about automation in the future when it seems like more of these ideas are figured out, then I can reason more carefully about how you'd imagine connecting them together. But I think it is really a good sort of fallback.

Yeah, makes sense. All right, are there any other approaches dealing with the scaling issue that we should talk about, or is that kind of the top few options for now?

Well, I think there is one more, which is just throw more humans at the problem. I don't know. We don't think about this right now, and if this really is an important problem and at some point we're really deploying neural networks where there are high stakes, imagining throwing 1000 people at going and working on systematically auditing things doesn't seem entirely crazy.

Yeah, well, I mean, more than that, I guess if you got like ten now, why not 10,000? Why not 100,000? If these networks are making up a large fraction of the economy, then you might think that could kind of naturally happen, at least if it helped to make them function better.

Yeah, and if you look at how many people analyze the security of the Internet or things like this, that's got to be some pretty nontrivial number of people. Or design cars to not crash and not hurt people.

Yeah, I guess in some ways that's actually another reason why I study circuits, is just trying to demonstrate that it's possible at all to understand neural networks, just as a way of justifying to society that this is worth investing in and worth trying to figure out how to scale.

Yeah, I guess that's another sort of implicit motivation. I have you've kind of alluded several times to the fact that there's a bunch of disagreement or maybe a lack of consensus about interpretability within the field. What are the disagreements or different interpretations that people have of interpretability?

Yeah, I guess there's two things. So one is just I think within interpretability, people mean lots of different things, and there isn't really consensus about what interpretability is or what it means to understand a model. And then I think outside of interpretability, there's a nontrivial amount of skepticism from some other members of the ML research community. Once, many years ago, a colleague of mine who I respected told me that they thought all interpretability research was bullshit. And I think that's maybe stronger than the typical view. But I think that there are some people who are skeptical, and I think that probably to some extent, they're picking up on the fact that maybe things aren't fully figured out about what interpretability means.

Do they still think it's bullshit? Because I just don't understand how someone can look at these articles. Like, maybe you think it's not the most important thing, or maybe there's mistakes being made, but I don't understand how you could think that this isn't legitimate research into how these systems work.

Well, I think actually another motivation for trying to be so rigorous and trying to build this foundation is in some ways to address this concern. Because I think actually, if you believe that all these attempts to understand neural networks don't make sense and just are fundamentally flawed, that's a very different view from how I see the world. And I think it really changes whether you think it makes sense to pursue this as an approach to safety or it actually is a very fundamental crux, I think. And so I've sort of seen a lot of my work as trying to sort of create something that sort of is very objectively correct with circuits.

Yeah. Maybe being more concrete or specific, kind of what are the different takes that people have on what interpretability is or how it should be construed?

So I guess I should caveat that and say that both. I'm obviously biased towards my own work, and I'm just not sure that I'm able to fully, fairly represent the views of everyone in the space. But one way that I think about this, and I really owe this view to Tom McGrath, is that I think of interpretability as being a pre-paradigmatic science in the sense of Thomas Kuhn and the Structure of Scientific Revolutions.

So I guess the idea here is, know, usually sciences have a paradigm where everyone agrees on sort of what the important questions are and how you answer those questions and how you tell if an answer is a valid answer to that question and what are the topics of investigation and things like this. And I think that interpretability doesn't have that. There's no shared set of answers to that question. And so I think that the discourse that's going on interpretability right now is sort of it's a very common pattern in early fields of science where you don't have consensus on these questions and people are proposing different answers and sort often having trouble communicating with each other and there's sort of a lot of disagreement and confusion. And in fact, I think that we're falling into another very common pattern, which is often when a field is in this state, practitioners will try to lean on existing disciplines to go and answer these questions for them. And in this case, I think we see practitioners largely lean either on machine learning to try and answer this question or on HCI to answer these questions.

What's HCI? Human-computer interaction.

So I think the ML people sort of want in some ways to have a metric for what it means for things to be interpretable. And I think the HCI people want to do user studies and ask, know, was this explanation useful to you? And maybe ask does the person make better predictions after seeing an explanation? Or things like this? And I think these are both sort of valid answers to go and have. I tend towards a different answer, which is to see interpretability as being like an empirical science. It's like the biology of neural networks or something like this. And so the question isn't whether something is useful, the question is whether something, a statement is true and whether it's true in the sense of falsifiability. And you certainly see this view in other areas of machine learning and people studying neural networks. I think you don't see it as much for people who are doing things that might be articulated as interpretability. I think it's often because people see visualization as less scientific. But yeah, to me, really the goal of circuits is to sort of try to show how interpretability can be an empirical science.

Yeah, maybe I've just been influenced so much from reading the articles you've contributed to writing and your framing of it all. But to me it does just seem like you're making specific claims about the functions of different circuits. And like, well, if you take this cluster of neurons and weights and so on, this is what it does. And so you're right, you could say, well, should we look into this? It will depend on, is this useful? Is this helping us make better predictions about stuff that we actually care about in terms of what it does? But yeah, just like you can look at a machine and say, well, this is what a gear does. It seems like you can say that about circuits. And having that level of understanding, at least that's like, one sense of interpretability is like, I look at this machine and I understand what each part is contributing to the whole. That seems very natural to me.

Yeah. And I would describe this as being like mechanistic interpretability. Or some people would call it transparency, the sort of like we are understanding this as a mechanism. We're understanding what causes it to work. And you can have other types of work that are maybe a little bit less focused on this. So there's a lot of work on saliency maps which try to highlight, say, you have an image classifier or what parts of the image were important in going and classifying in the final answer the model gave you. And for that you might want to ask that more in the question of like it turns out it's actually very difficult to sort of ask that question in a really rigorous way because these functions are so nonlinear and complicated. And for that it might make more sense to ask it in this sort of HCI type lens of is the explanation useful? Is it causing users to make more accurate predictions?

Okay, so it sounds like at a high level, you think there's disagreement in part because people just don't agree on what it is to understand a neural network, what it is to have interpretability. And maybe over time we should expect that with other fields before it, people will, well, maybe there'll end up being multiple different conceptions of interpretability, but people will understand them more crisply and say, well, we have like this sense, but not this sense. Or maybe people will converge on kind of a common idea of what interpretability is. That is probably most likely the most useful one for the actual work that you're trying to do.

Yeah, I think that's right. I think you could either have multiple fields form or often, at least according to Thomas Kuhn, one paradigm will eventually win out. I guess what I find really interesting about Kuhn's description of this is I think that the thing that he sort of thinks is central to a paradigm isn't the particular theories that somebody has. You develop these different schools that have different ways of thinking about a problem with electricity. We ended up with. There were multiple sort of schools that were some were more focused on how charges repel and some were more interested in current. And I think the current one sort of won out. And I think to Kuhn, the important thing isn't the particular theories they had, but the phenomena they were choosing to pay attention to. And so from that lens, maybe the thing that is sort of central to the Circuits paradigm to going and approaching this and I should say I think there's probably other work that sort of embodying a similar paradigm. I don't want to claim it entirely for us, the core of that is paying attention to features and how they connect to each other and having that sort of be the phenomena that you focus on.

Okay. Yeah. So yeah, the articles that we've been talking about are just packed with lots of images that I imagine it took a whole lot of time to make. And maybe that's an integral part of interpretability is that people have to be able to grasp it and so they have to be able to see it. Do you think that kind of visualization is going to be a core part of interpretability going forward? Or is that just maybe specific to these cases?

I think a lot of people conceive of science as being about studying summary statistics because a lot of science, we have a couple of numbers that we can boil things down to that are really important and we are able to study how those interact. I think people are often pretty surprised when they see our work because they're used to research, and especially machine learning research involving studying a bunch of summary statistics and creating line plots and that sort of being what they feel like science should look like. Or you can get the bottom line from a paper by looking at a graph that shows the accuracy or something over time as you add more neurons or something like that.

Yeah, and I think it feels very rigorous to them; that's what they expect science to look like. But it's this really interesting thing where summary statistics can actually kind of blind you, where I guess there's this famous example Anscombe's quartet; have a bunch of sets of 2D points that have the same mean and standard deviation but are totally different when you actually look at them. And that's for two summary statistics that we really understand very well, means and standard deviations. And there's just so much going on inside neural networks that if you just try to boil it down to a single number and especially if you don't understand what's going on first, you're losing sight of almost everything important. At least that's how it seems to me. And so a lot of our work is just trying to show some fraction of all of the intricate structure that exists within these models. And I guess I often feel a little bit like we're like somebody looking through a microscope for the first time, and you're seeing cells for the first time. And sometimes people are like, is that scientific? Well, it's a qualitative result, not a quantitative result. But seeing cells was a qualitative result, not a quantitative result. But it was really important. But it was very important. And so, yeah, I think that if you want to understand these models and you want to really understand everything that's going on inside of them, you need to be looking at this fine-grained structure, and you need to be trying to get access to all of this. And that sort of inevitably pushes you towards visualizations, because data visualization is just really a tool for displaying lots of data and getting access to lots of data and communicating it. And the fact that we're used to sort of a small set of visual forms that are the ones that are effective for communicating certain kinds of data that we often work with doesn't really change the fact that we're going to want to work with other visual forms for understanding the large amount of data we get from neural networks.

Yeah, maybe if I could try putting it another way and see whether you agree. You're like trying to get people to understand, people who've never seen a car, get them to understand how a car works. And maybe you can't get that level of intuitive appreciation for how a car functions without getting your hands a little bit dirty, without actually maybe playing with the different pieces and seeing how they connect together and looking at the schematics. You couldn't just present people with a graph or like a bunch of numbers and be like, that this summarizes the car. And maybe it's a little bit more like engineering maybe, than coming up with some natural law.

Yeah, I mean, imagine if you tried to describe cars with five numbers. Maybe there are certain ways that it's useful to describe cars with five numbers, like horsepower and size and number of gallons consumed per hour or something like this, but you couldn't go and build one with just but you couldn't go and build one. And if you want to understand how a combustion engine is working, you probably need to go and look a little bit more closely than those five numbers.

Yeah, that makes sense. Another concern that people might have about interpretability research is just that, as we've been noting throughout this conversation, it seems very similar, or very analogous in some ways to, neuroscience. And neuroscience, over many decades, despite having a pretty large research community, hasn't really cracked understanding of the human brain. It's like it's made some progress, but it hasn't made nearly as much progress as we would need probably to be comfortable deploying a really important ML system. Does that make you at all pessimistic or concerned about the prospects for understanding neural networks as much as we need to?

Yeah, I think it's a real concern. The community of people working on neuroscience is orders of magnitude larger than the community of people working on circuit-style interpretability of neural networks. And so it does seem like a pretty good case for being concerned. But I do think that there are actually a number of very large advantages that interpretability has over neuroscience, which level things out a little bit. So just at a high level, I actually wrote a short note about this recently, but some of the advantages are that you could just get access to the responses of every neuron. I guess in neuroscience, you'd normally only be able to go and access, record a couple of neurons, and you'd be able to record them for a limited number of stimuli. But here you can go and get it for all of them. You not only have access, I guess a lot of neuroscientists are trying to get the access to the connectome of how all the neurons connect the human brain, for instance. But in neural networks, not only do you have the connectome, but you have the weights that connect every neuron. And you know what computation every neuron does, and you just have access to the entire thing. And that's what makes circuits possible that we can go and look at how the weights connect neurons.

Yeah, there's also the fact that weight tying can dramatically reduce the number of unique neurons that exist in neural networks. What can, sorry. So there's this, for instance, in vision models, you actually have all of these replicas of the same neuron where you go and you run them at every position in the image. So you're like, if I have a line detector, it's useful to go and run that line detector at every position in the image. And the result of this can be that you have maybe like, 10,000 times fewer neurons than you would if you were studying a comparable biological neural network or fewer unique neurons. And so that's the difference between it not being possible to go and just look at every single neuron and it being easily possible to go and look at every single neuron. And we've done this for some vision models, and so that's a huge advantage.

Yeah, it sounds like maybe the advantages are so great that you'll end up learning more about the human brain by studying neuro. Maybe then you can understand the human brain by studying the human brain. Just because the underlying data is so hard to access within the human brain. It's like it's tied up in all this physical stuff that's super hard to play with.

Yeah, I mean, I'm not a neuroscientist, and I wouldn't want to opine too much, but it seems very plausible to me that perhaps studying artificial neural networks could teach us a lot about neuroscience. And I should say that actually, that's just partway; there's a number of other advantages that I haven't mentioned. So I think that we do have a pretty significant number of advantages.

Okay, yeah. We'll stick up a link to that note that you recently published, or I guess you lay out the other benefits that you have. We got a ton of audience questions where we mentioned that we're going to be interviewing you. Maybe I'll just chuck in a couple of these now.

Yeah, one that I was interested in, I'm not sure whether this is one that you can really tackle because maybe it's just too philosophical, but someone pointed out, like Chris, you're finding all of these discoveries inside neural networks that mirror neuroscience. As we were just saying, people like Brian Tomasik and others who are very concerned about suffering in the future have raised concerns that potentially neural networks or artificial intelligence or reinforcement learners might be able to suffer when they're running on computers or I guess potentially feel pleasure as well. This person asked, yet, do you think your findings should make us more or less worried about that?

Well, I'll just start by saying that I don't feel super qualified to comment on this because I'm not a moral philosopher or philosopher of mind or anything like that. And I really don't know how you should think about whether something has moral patienthood or not.

Yeah, honestly, I'm not sure whether super qualified, but sorry, go on.

But my instinct is that this is a pretty serious concern, and I think it's something that people think it's funny to talk about or that you're kind of being a crank if you talk about it. But, yeah, I'm worried. This is a pretty serious issue. And I guess a big part of the reason why I'm worried about it isn't even so much the probability that they might be agents that are entitled to moral patienthood, but that it's so invisible. So I care a lot about animal rights. I'm a vegan, and I think one thing that makes animal rights so insidious is that people don't see the suffering that's going on in factory farms. It's hidden from us. It's invisible. And how much more invisible would the suffering of neural networks be if they were to suffer, which I don't know that they will, but if they were, how much more invisible would it be if they're running on a server where no one can see them and there's no visible output of their suffering? And a couple of clicks of a computer button, of buttons on my keyboard, go and scale up the number of neural networks that are experiencing this, I think that really increases the risk that we could have a moral catastrophe.

Now, I don't know. Is there anything useful that can be said about that I can say about whether models might be suffering?

Well, I do think these multimodal neurons should be a little bit of a red flag for us. We have something that previously only existed in humans, and now we're observing in artificial neural networks. I'm not saying that necessarily should be a big flag. I think other people would be better positioned than me to think about that. But I do think it's the sort of thing that I wish were systematically that was causing systematic reviews and sort of systematic check-ins when we find things like that, because I think each one sort of seems to increase the risk that maybe we're dealing with moral agents.

Yeah. So I guess the concern is, well, we think that humans can suffer, probably. And so if these neural networks just keep adding in more and more components, functional components that humans have, then maybe they'll develop the ability to suffer as well. Because that's one of the things that's in our minds. It seems a little bit like an image recognition system. It's not obvious what is analogy to a preference, a desire, and suffering here. Maybe you need to have a more agenty neural network before it can suffer, or that feels intuitive to me, but I suppose we will have such systems.

Right. Well, I think a lot of people really focus on reinforcement learning as the thing that must be the difference between a neural network suffering or not suffering. And I'm pretty skeptical that's actually the central issue, like a reason why you might not think that is you could train a neural network with imitation learning instead to go and mimic an actor. And I think that if it led to a neural network with the same behavior, I'd probably think that they were equally likely to suffer. I have a bunch more technical reasons why I think that reinforcement learning is probably not the central thing. But I don't know, maybe the more important point here is just I think the level of discourse here is so low and so minimal that just, like, really basic points and thoughts haven't been laid out. I think just because it feels sort of like a crankish thing to be talking about, especially if you're like somebody who's sort of a serious researcher. And I think that's an issue. I think that we should be trying to think these things through carefully, and it's a really hard topic to think about, but I wish that there was more careful discourse.

Yeah. Another listener wrote in and wanted to ask, maybe we can use this interpretability research to understand neural networks that are doing things that we ourselves can do and that we ourselves can understand. But if you're having a superhuman system, a super intelligent system, it might be doing tasks that are too complex or difficult for us to do and therefore maybe too complex and difficult for us to understand.

Yeah. Do you think that could possibly interfere with our ability to use these interpretability tools on far more advanced systems than what we have today?

Yeah. So I split this question into two parts. One is you could just generically be worried about the scaling of neural networks and whether we'll be able to keep up with studying larger neural networks. And one is you might have specific concerns that regard to super intelligence and systems being smarter than us. In particular, you might think that there's ways in which you can study very large models, very powerful models. And then when those models become smarter than you because they're sort of just intrinsically because they're thinking about problems in a way that's smarter than you and is more sophisticated than you, that maybe that's a special point where you get screwed up. So I think we've already talked about the scaling part earlier and really sort of it's more these super intelligent specific concerns that we should focus on here.

Yeah, that seems right. I guess one thing that seems a little heartening here is we already have models that are smarter than us in narrow ways. Like ImageNet models are better than me at recognizing different kinds of dogs. And with effort we can look at these models and we can actually learn from them and make ourselves smarter. There's this really lovely talk

By Bret Victor. Media for thinking the unthinkable. And he says some things in it which strike me as being very profound, sort of in general. And actually, I think that this kind of tools for thought type thinking is something that is underappreciated in the EA community, but I think is actually extremely relevant to neural networks. And maybe I'll actually just quote it for a second because I think it is a really powerful way of thinking about this.

Yeah, go for it.

So he starts by quoting Richard Hamming. And he goes, Richard Hamming goes, "Just as there are odors that dogs can smell and we cannot, as well as sounds that dogs can hear and we cannot, so, too, are wavelengths of light that we cannot see and flavors we cannot taste. Why then, given our brains are wired the way we are, does the remark ‘perhaps there are thoughts we cannot think’ surprise you? Evolution so far may possibly have blocked us from being able to think in some directions. There could be unthinkable thoughts." And I think in some ways this gets to the root of the concern you might have about trying to understand a superintelligent system. You might think that there are thoughts which are impossible for humans to think that this system is thinking, and therefore we're screwed.

Yes.

And so Ficter responds as follows: "The sounds we can't hear, the light we can't see—how do we even know about those things in the first place? We built tools. We built tools that adapt these things that are outside our senses to our human bodies, our human senses. We can't hear ultrasonic sound, but you can hook up a microphone to an oscilloscope, and there it is. You're seeing that sound with your plain old monkey eyes. We can't see cells and we can't see galaxies, but we build microscopes and telescopes, and these tools adapt the world to our human bodies, to our human senses. When Hamming says ‘there could be unthinkable thoughts,’ we have to take that as yes, but we built tools to adapt these unthinkable thoughts to the ways that our minds work and allow us to think these thoughts that were previously unthinkable." And I think that there's actually a really deep analogy to what we're doing with interpretability. Michael Nielsen and Sean Carter have a very nice idea, a nice essay on what they call artificial intelligence augmentation. But I think the high-level idea of all of this is that we can try to build tools to help us be smarter and to think these thoughts. And in fact, if we have systems that are reasoning in these ways, they can teach us the abstractions and maybe even be themselves the tools that allow us to go and think these previously unthinkable thoughts.

Yeah.

So a concern might be that while maybe you could break these unthinkable thoughts into pieces that we can digest, maybe given enough time and effort, perhaps that's going to be really quite hard because we're going to be dealing with abstractions that are very foreign, very alien, very difficult for humans to grasp. And so it could be a much more laborious thing than understanding that this is a dog recognizing a circuit.

Yeah, that could be true. Although I don't know if you look at—well, it depends on the human. But I think if you look at many humans who I think are really smart, they actually are very good at their thinking, seems very clear and actually often quite easy to understand. It could also be that you could think of the curve of how understandable a neural network is, and you could imagine that there's some valley where it's very confused and has really bad abstractions, and then it gets crisper and crisper and becomes easier to understand. And then perhaps there's some threshold at which it starts to have just sufficiently alien thoughts that we can't understand it anymore, past human capacity. And there's an empirical question about where that is. But if we can get to the point where we're dealing with human-level systems and somewhat stronger than human-level systems and understand them and really be confident in their safety, that would actually seem like a pretty big safety win to me, even if there's some point where these things fundamentally become so alien that we just can't understand them at all.

Yeah. All right.

Yeah. Another listener wrote in saying, "Chris seems to be extraordinarily good at kind of pulling apart and understanding the inner workings of his various models, including even complex ones and fairly large ones." And they're asking, "Does—how does one presumably—them continually kind of take their ability to explain and understand models to the next level?"

Well, that's very kind of them. I think probably the biggest thing is just practice and spending lots of time staring at neural networks and trying to figure out what's going on. But let's see, what else can I suggest? Well, I think at the risk of being slightly self-promotional, just reading the Circuits thread may be quite useful. Like, it walks you through a lot of examples of things that neural networks do and examples of how they do it and the circuits that are implementing it. I think just having a library of concrete examples that you understand is actually very helpful.

Yeah, I feel like I benefit a lot from that. I think that a useful exercise also could just be to create toy problems for yourselves and ask yourself, "How would I implement this behavior in a neural network with this architecture?" I often am sort of playing with things like this when I can't sleep at night, and yeah, I don't know, I feel like I come to much more deeply understand what the constraints of an architecture are and what kinds of things are easy and hard.

Makes sense. Sounds like practice. Practice, practice, practice. I think just like spending time with a model and just poking around a lot is really helpful. And then maybe one thing that I think people might find surprising is just building some basic data vis fluency and getting some competency with tools that allow you to just get lots of data on paper and get neural networks out there, organize them with feature visualizations, go in and have simple interfaces that allow you to navigate that because the ability—there's just so much data inside these models. The ability to have tools that allow you to navigate it and sift through it—it's the thing that prevents you from needing to resort to summary statistics that will mislead you.

Yeah, I think I failed to say earlier that this microscope tool that OpenAI is offering, which I think from memories at Microscope.OpenAI.com—yeah, it's just like—it's beautiful to play around with, even the people out there who aren't into ML, even the people out there who aren't actually trying to learn how to do this for their job. I recommend go and check it. Like, I was blown away. This is something that, you know, shouldn't this be the way that you understand it? And then I was like, "Oh, my God, I clicked through the articles. Like, this exists now." Presumably that kind of experience might help people.

Yeah. Another question that actually multiple listeners raised was, most of what we've been talking about so far is about these visual models, things like CLIP, which are trying to figure out, classify images or take an image and figure out what the caption should be, or take a caption and then design an image around it. But there's probably going to be other models that are maybe more important that might be getting deployed and that are more capable, that are doing more relevant stuff. I guess obviously models like GPT-3, which would take an opening paragraph and then write an entire essay from it—seems like perhaps more work looks like that than classifying images. Is this interpretability work going to just extend naturally from the image stuff to the natural language interpretation? Or might there be more fundamental differences that mean that the analogy somewhat breaks down and it's harder to do or you have to kind of reinvent things?

Yeah, well, I guess something I should say before diving into this is a lot of interpretability work does get done on language models. It's often a pretty different style of interpretability work than the things that I sort of feel really excited about. But a lot of work does get done there. So I'm going to interpret this question as like, "How optimistic should we be that circuit-style interpretability work can sort of thrive in the context of language models?" And I think to some extent that's a hard question to answer before we really try. I feel optimistic. I don't really see any fundamental reason why it shouldn't work, but I think it's reasonable for people to be skeptical until we demonstrate that. I actually switched to doing work on language models very recently, and so all I've done so far is just some very preliminary stuff. But we've actually reverse-engineered a couple of simple circuits for doing actually some things related to meta-learning in language models, and it's actually seemed quite easy. So my preliminary position is that not only do I not see any fundamental reasons why these things shouldn't work, but I feel quite optimistic.

You mentioned earlier that some kind of different interpretability work has been done on language models.

Yeah. What's been turned up by that? And maybe what does that look like?

I think there's a lot of work that's broadly as a flavor of, "There's some linguistic feature we think is important. Can we determine the extent to which that's represented at different points in CLIP?" So I might describe this as slightly like top-down interpretability, where you have a hypothesis about something and you try to see if you can predict it from some layer. There's also some sort of more visualization-oriented approaches where there's one paper by Google that I really liked where they were just sort of visualizing how the embeddings of various words evolve as they interact with the context around them. And they looked at the word "die," and they found that in some contexts it's like a German article and that's like one cluster, and in another context it's death, and that's another cluster. I think that the sort of like more visualization-oriented stuff is cool. There's been some nice work visualizing attention patterns. So yeah, I think that there is some interesting work in this space. I don't really know that any of it has quite got to the point of sort of getting to mechanistic explanations and this sort of bottom-up sort of understanding of features that I sort of feel like I most trust. But also I'm entering a new space and I'm presently an amateur in that space, and so I wouldn't want to comment too confidently.

Yeah. Okay. So this kind of top-down interpretation where people might approach the model and say, "Well, it's got to be able to predict whether a noun or an adjective should go here, so we should expect to find some part of it that's doing this prediction of what kind of word should go next." And then we're going to go scout around and see if we can find it. I guess you prefer it maybe where we're like, "We start with a neuron or we start with a cluster and then work up and then figure out bottom-up, like, what is this thing doing?" without bringing too many preconceptions to the table. But I suppose, yeah, it's natural to do both. Maybe that would be the thing that I'd be maximally excited about. But I'm excited to have a wide range of people trying things. And I think it is really exciting that there's a significant community of people asking questions in the space, and there's, like, this whole area that people call "Bertology," I guess, like biology, but for BERT, which is a language model. And, yeah, I think it's wonderful that is a significant thing.

Is there anything more that you can say about what would be the equivalent of a circuit in a language model? I suppose it sounded like with the example of "die" or "d," you could imagine that there's circuits that detect, "Is this a sentence about this kind of thing? Is this a sentence about a person or is this a sentence about a machine?" Or maybe I guess you would have to start with something that's maybe even simpler than that, but that might be like some kind of equivalent where it's like figuring out the broad cluster of the topic.

Yeah, well, in my conception of this, just like we have features in vision models, we also have features in language models. They live in MLPs, so you have these blocks that are just regular neural network layers with neurons like anything else in transformers. And then the thing that's different is you also have these attention heads, which are very different from anything we see in convnets. And so the thing that we have to do is sort of adapt our sort of framework of thinking about things in terms of circuits to also include these attention heads, and then hopefully we can use that to go and understand some larger-scale mechanisms that are going on.

Yeah, makes sense.

Are there any sorts of ML models, maybe models apart from language ones that exist now or might exist in the future, where you worry that these interpretability methods wouldn't—guess, you know, examples of other kinds of neural networks—AlphaZero that learns to play lots of games, I guess MuZero. We've got, I guess Facebook's recommender algorithm, I imagine is a neural network. It's trying to perform a different kind of task. Are there any ways in which there could be a class of neural network that's just really hard to understand?

I mean, I think there's lots of types of neural networks where I haven't thought really hard about what understanding them would be like and what trying to do circuits-type analysis on them would be like. I do think that models that are so-called model-based reinforcement learning, where they're sort of in some sense like anticipating futures or something like AlphaGo that unrolls lots of possible futures and then has that influence what it does, would be very interestingly different to study. I don't know that I think they'd be harder, but I think they'd be very interestingly different than the models that I presently study, I guess.

Yeah. Over this conversation, we've raised a couple of different objections or doubts that people might have about this interpretability research agenda and how useful it will be. Are there any other skeptical arguments that trouble you at all? Like any ways that perhaps this could turn out to be a dead end or not be so useful that I guess ones that keep you up at night or ones that maybe should keep you up at night if you weren't as optimistic a person?

Well, I think the scaling one is probably the thing that I'm most—sort of. I feel relatively optimistic, but when I say I'm relatively optimistic, I don't know, maybe that means, like, I'm greater than 50% optimistic. I think there's still a lot of room for things to fail, and I'm the person who should be most—I'm sort of probably slightly delusionally optimistic about it. It's a bit of a selection process that would make it surprising. If you thought this was a bad plan and you were the main person leading it, that would be a little bit worrying. But I think this is why you want to have lots of different agendas, and hopefully there's sort of a portfolio where you hope that one of them eventually pans out and you aren't necessarily counting on any single thing to succeed.

In terms of worries that haven't been voiced as much here, there's this issue of polysemanty where a neuron responds to multiple things, and there's a lot of theories about why this happens. Like, one theory is that there isn't enough space in some sense to include all the concepts. I think we just don't know why it happens. But it makes it a lot harder to understand neural networks, especially if you're trying to look at things in terms of neurons.

Yeah, just to explain for people, I guess, this polysemanty, this is the thing where you have a circuit that is simultaneously seeming to do two very unrelated things. And you're like, "This is a circuit that detects cars and the Queen of England." And you're like, "Why is this circuit doing this double duty?" And it seems like it makes sense that this happens when the neural network is small and it has to try to cram as many functions into very few neurons. But I guess you're saying this seems to happen even in very big networks sometimes.

It certainly happens, at least in medium-sized networks. And one answer you could have is, "Well, as you make the model larger, there's also a larger set of things that the model could represent that are useful. And so the number of neurons that are available to go and store things is increasing, but also the set of things you want to store is increasing."

I see. So I guess it's got this trade-off. It can either have fewer concepts so it says, "No, this is going to do only cars and not the Queen," or it can accept, I guess, some frustration, some technical challenges with having a circuit that's doing two things and might misfire, but in return, it gets two concepts that it can cram into there now. So it's got this trade-off all the time. Like, even as it gets bigger, as long as there's, like, more concepts that aren't represented, then it still has that trade-off.

Yeah, that's exactly right. And I think we don't know what the asymptotic behavior of that is. Like, as you make models very large, does polysemanty go away or do we need to go and do something else? Like, is there some way that we can sort of unfold a neural network into a larger neural network which doesn't have polysemanty, for instance? Or can we just try to very carefully reason through circuits and despite there being polysemanty, be able to sort of very carefully reason things through and it doesn't end up being that big an issue?

Okay. Yeah. And the reason this troubles you is just that when you come up with a polysemantic circuit, it's just much more of a pain in the ass to understand it. So it slows you down. And I think we're much less confident that we understood everything about it.

Okay. Yeah. What else is hidden in there? What else is hidden in there? And you could even imagine there's things that are very small in every single neuron but are sort of hiding between the neurons. They're sort of orthogonal, sort of almost all of the neurons. And so it's very difficult to see them but might still be being represented in the model. And so it's very hard to catch from this kind of lens. This is actually why earlier you asked me what a feature was, and I was like, "Well, usually we think of features as neurons, but the reason that we don't just say that features are neurons is that there seem to be some cases where the features are actually, like, stored in combinations of neurons or where a neuron corresponds to multiple features." And the ideal thing would be to find some way to go in and represent things so that's all separate. This is related to—there's a literature in machine learning where people talk about disentangling representations. It's a little different because usually then they're talking about a couple of features that they know, like they care about gender, maybe, and, like, hair color or things like this, and they want to find a way to disentangle things, whereas we don't have that benefit here. But it's a closely related problem.

Yeah. All right. Yeah. We've covered quite a bit of the stuff that was in these articles and I guess in these circuit series more generally. Is there anything that you think is important and interesting that the audience maybe should know that we haven't managed to cover so far?

I mean, I guess we've talked a lot about the ways this could fail, and I think there's maybe just something that's worth saying about how wonderful it would be if this could succeed. It's both that it's potentially something that makes neural networks much safer. But there's also just some way in which I think it would aesthetically be really wonderful if we could live in a world where we have—we could just learn so much and so many amazing things from these neural networks. I've already learned a lot about silly things like how to classify dogs, but just, like, lots of things that I didn't understand before that I've learned from these models. And you could imagine a world where neural networks are safe but where there's just some way in which the future is kind of sad, where we're just kind of irrelevant and we don't understand what's going on and we're just like humans who are living happy lives in a world we don't understand. And I think there's just potential for a future even with very powerful AI systems that isn't like that and that's much more humane and much more a world where we understand things and where we can reason about things. And I just feel a lot more excited for that world. And that sort of is part of what motivates me to try and pursue this line of work.

Interesting. Yeah. So the idea there is in the future, it might be that neural networks are doing just a lot of the stuff that people used to do that they are kind of calling the shots in some sense, although hopefully ideally aligned with our interests. But it might be a very alienating world if they are these black boxes that we just can't comprehend and we ask them to do stuff and then they do it, but we don't have any understanding even of the process. But if we can look at them like machines, like cars, where you can Google it and then find out how a car works, and you're like, "Okay, the world is comprehensible to me," and that makes me feel better about it. It makes me feel like less of a useless just being waited on by these neural networks. And I guess we can also potentially learn things about the world from these neural networks. Like, one reason why they're able to do things that we can't do or do things better than us is that they've got features inside their minds, they're thinking about things a particular way, and maybe we could learn to understand, possibly even kind of copy the clever things that these neural networks are doing.

Yeah, there's this idea of a microscope AI. People sometimes will talk about agent AIs that go and do things and oracle AIs that just sort of give us wise advice on what to go and do and another vision for what a powerful AI system might be like. And I think it's a harder one to achieve than these others and probably sort of less competitive in some sense, but I find it really beautiful as a microscope AI that just allows us to understand the world better or shares its understanding of the world with us in a way that makes us smarter and gives us a richer perspective on the world. This is the kind of thing that this would enable.

Yeah, it's something that I think is only possible if we could really succeed at this kind of understanding of models. But aesthetically, I just really prefer it. Yeah. So that people have maybe a clearer picture in their head of what's going on. Can you describe a possible story in which this interpretability research would help us to foresee a failure mode of a neural network when it's deployed and then kind of diffuse it ahead of time? Is there any kind of toy story that we have of that?

Yeah, so I think there's a number of different stories that sort of—maybe you can sort of think about them from the world where a neural network or circuit-style interpretability sort of most dramatically succeeds and how that contributes to safety, to progressively worlds where it succeeds less, but still contributes to safety in some meaningful way.

Yeah, go for it.

Maybe on the most extreme side, you could just imagine us fully, completely understanding transformative AI systems. We just understand absolutely everything that's going on inside them, and we can just be really confident that there's nothing unsafe going on in them. We understand everything. They're not lying to us; they're not manipulating us. They are just really, genuinely trying to be maximally helpful to us. And you could sort of an even stronger version of that—we understand them so well that we ourselves are able to become smarter, and we sort of have a microscope AI that gives us this very powerful way to see the world and to be empowered agents that can help create a wonderful future.

Okay, now let's imagine that actually interpretability doesn't succeed in that way. We don't get to the point where we can totally understand a transformative AI system—that was too optimistic. Now what do we do? Well, maybe we're able to go and have this kind of very careful analysis of small slices, so maybe we can understand social reasoning and we can understand whether the model—we can't understand the entire model, but we can understand whether it's being manipulative right now, and that's able to still really reduce our concerns about safety. But maybe even that's too much to go and ask for. Maybe we can't even succeed in understanding that small slice. Well, I think then what you can fall back to is…

Maybe just you have, with some probability, you catch problems. You catch things where the model is doing something that isn't what you want it to do. And you're not claiming that you would catch even all the problems within some class. You're just saying that with some probability, we're looking at the system and we catch problems.

And then you sort of have something that's kind of like a mulligan; you made a mistake, and you're allowed to start over. Where you would have had a system that would have been really bad, and you realize that it's bad with some probability, and then you get to take another shot. Or maybe as you're building up to powerful systems, you're able to go and catch problems with some probability. And that sort of gives you a sense of how common safety problems are as you build more powerful systems.

And maybe you aren't very confident that you'll catch problems in the final system, but you can sort of help society be calibrated on how risky these systems are as you build towards that. Yeah, I guess this is a model where neural networks that are getting deployed are kind of like cars, or maybe it's deploying a new drug or a new pesticide. And it's not that we want zero failures. It's just that we want to have better systems to be more likely to find problematic side effects or to find ways that they're going to fail as often as possible. Because the more we manage to foresee problems and prevent them, the better.

I suppose some people who are operating more within a very rapid takeoff of AI—we're going to reach superintelligence really quickly, and any errors that slip through risk catastrophe—that's kind of a very different vision, maybe, for how AI could end up affecting the world. I suppose from that point of view, just being like, well, we found 80% of the problems, maybe it wouldn't put people's minds at ease so much. Do you have any thoughts on that? Or maybe it's just that you think that this more gradual deployment of ever more powerful neural networks is just by far the most likely scenario to play out.

I mean, first I'd say I still am aspiring to catch all the problems. I don't know this approach is going to succeed, but I do think that, I guess in some ways, the way that I'm answering this question is in progressively worse worlds, do we contribute to safety? Like how do we contribute to safety? And as we succeed less, our contributions to safety are smaller. Hopefully, they're still helpful. I do think that people—even people who have a relatively fast takeoff worldview, unless they have a very fast takeoff worldview—should be kind of excited about anything that might help society become more calibrated on whether there are risks from AI systems. Because the kind of coordination that you'd want to have about deploying a transformative AI system seems much more likely if you have concrete examples and have concrete examples of systems being unsafe. It seems really hard to coordinate people around a completely abstract problem. And so even if it's with much weaker systems, if you can have compelling examples of safety problems or of systems in some sense, especially being manipulative or being treacherous in some way, it seems to me much more—I feel much more optimistic about going and persuading people to take action.

Yeah, that makes sense. So what's the future for interpretability research? Are there any kind of really promising horizons that we can expect to see over the next couple of years? I think the question isn't are there promising horizons, but I don't know—of the immense number of promising horizons, I feel like this must be what it's like to be involved in a very early scientific field where there's so much low-hanging fruit. It just seems to me that you can go in a million directions and you'll find amazing things to discover about neural networks in every single one of those directions.

Yeah, okay, so there's just a fertile unfarmed ground everywhere that I can see. Everywhere that I can see. I mean, I think you can just sort of crack open up a neural network, and if you're looking carefully with high probability, you can discover something that no one else has seen before. And maybe there's particularly high-impact things like, I think discovering motifs or discovering larger-scale structures or making progress on language models or things like this are higher impact maybe than other things. But if you imagine this is like we're in the early days of the biology of deep learning, there's just every direction—there are amazing things to find.

Nice. All right. Well, with that in mind, what are kind of the first steps that people should take if they're really keen to get into this interpretability agenda and contribute to it? Suppose I'm thinking we've had other episodes; we've talked about how do you pursue a career in machine learning in general, but maybe for someone who is maybe already heading towards a machine learning or like an AI safety career, how do they push themselves in the interpretability direction?

Well, again, I'll caveat that there's lots of different things that people are doing within the interpretability space. And I think that my advice is only very useful if you're interested in this kind of like circuit-style interpretability. I think my first piece of advice would just be to read the circuit thread carefully. I think it's the most concentrated example of trying to do this particular style of interpretability research. Again, it's my own work, or I'm involved in it, so I'm kind of biased. But I think if you're excited about this kind of work, that's probably a good starting point. I think just trying to really deeply understand neural network architectures, trying to reason about how you would implement different kinds of things in them, spending time poking around with them yourself—I think all of that is really useful. I think a lot of standard advice about how to get into machine learning and build up skills of general relevant software engineering and relevant understanding of theory of neural networks is all very applicable. I'd also try to develop some very basic data visualization skills. I think that's something that will serve you really well if you have it.

Are there any kind of conferences or like social events? People, they'll read all these things, but probably eventually they'll want to talk to people within the field in order to move further on. How might they be able to do that? Yeah, again, I think this particular approach is a pretty niche thing. There is a Slack community that you can access if you read the circuits thread. And also a number of us are pretty active on Twitter. I've really wanted to organize some kind of social event around this, but COVID has made that a hard thing to do. Yeah, but I guess people can follow you on Twitter. And I guess presumably, probably the subfield is going to grow, and eventually there will be events and things people can go to network and keep up to date. I suppose you probably present at the standard ML conferences, right?

Yeah, well, I sometimes give talks at conferences. I mostly publish in Distill since we sort of have this venue that allows for interactive visualizations. And so I think for this particular niche, a lot of stuff is sort of concentrated around Distill and around the Distill Slack community and things like this.

Beautiful. All right, I guess that brings us to an end—our very long discussion, substantial discussion there of interpretability. Before we stop talking about technical issues though, boy oh boy, did our listeners want me to ask you about scaling laws? It seems to be really the talk of the town among people who are following OpenAI or following ML. I actually don't know all that much about scaling laws, so maybe you can explain to me why everyone is asking me to ask you about scaling laws. Maybe what I, and they, should need to know about these things.

So before I say anything, I should say I have a number of close collaborators who work on scaling laws. I personally haven't done any work on scaling laws, and so I'm really not an expert, and I will try to answer—sort of talk about this as intelligently as I can. But if I say anything that's really clever, it's probably due to them and due to experts that I get to talk to about it. And if I say anything really stupid, it's probably just that I misunderstood something. So that's just a sort of caveat that I want to put on anything I say about scaling laws?

Yeah. All right, I guess. Yeah. First of, maybe we should explain what scaling laws are.

Yeah. Basically, when people talk about scaling laws, what they really mean is there's a straight line on a log plot. And you might ask, why do we care about straight lines on log plots? Might ask, Chris, what's on the axes? Well, it depends. So there's different scaling laws for different things, and the axes depend on which scaling law you're talking about. But probably the most important scaling law, or the scaling law that people are most excited about, has model size on one axis and loss on the other axis. Losses, like performance losses.

Yeah, it's like a high loss means bad performance. Okay. And so the observation you have is that there is a straight line where as you make models bigger, the loss goes down, which means the model is performing better, and it's like a shockingly straight line over a wide range. Okay, so if I increase the model size, I'm just remembering from the economics that I did that if it's a log-linear graph, so linear on the x and then log on the y, then a 10% increase is always as valuable. But if it's log on the x axis and log on the y axis, then if I increase the model size 10%, then what happens to the loss of performance? I don't really have an intuitive grasp of what a log graph is.

Yeah, well, honestly, I actually often think about this in a sort of log-linear way, because I think often the units on one axis are actually kind of natural to think about in terms in log space, and then you can just think about it as an exponential. But if you want to think about how both variables relate, you get something of the form y equals a x to the power of K. So, for example, y equals x squared would be a power law, or y equals the square root of x would be a power law. And we see these all the time in physics. You see, like, one of Kepler's laws relating the amount of time it takes for an object to orbit around the sun and the length of the longest axis of its orbit. That's a power law. It's related by something like that. And so then if you have a say, you double one thing, then you have this power, and you increase the other variable by that power. So if it was square root, for instance, and you did ten x, and you'd go and change one by the square root of ten.

Okay, so it's some sort of polynomial. I'm not sure I completely followed that. But the bottom line is that people have updated in favor of the view that we don't hit declining returns as much as we expected to from having bigger models. So it's just like increase the number of neurons, increase the number of connections, increase the number of parameters, and the performance just keeps getting better and it keeps getting better at a decent clip, even for models that were already very big. Is that kind of where this bottoms out?

Yeah. And I think maybe it's not just that you're not hitting diminishing returns. And of course, all of this that has been observed so far, this could be potentially to change. It could be that the next increase in model size, suddenly the scaling law breaks and it no longer performs this way. But the fact that it's such a straight line sort of is very surprising and makes it tempting to go and extrapolate further. And perhaps you can reason about models to some extent that are larger than the models you can train right now. I think that is why people are excited about scaling laws.

Okay? So they're using it to project forward, saying, well, in 2022 we'll be able to make a model that's this much bigger, and then we should expect this level of performance. And like, wow, isn't that like a great level of performance?

Yeah. And there are other scaling laws, by the way. So there's ones with respect to model size. There's also ones with respect to the total amount of compute you use, because as you make models larger, you also have to train them for longer. And so you can ask, like, if I have a given compute budget, what is the optimal allocation of compute between model size and training them longer? And then what loss do I get given that optimal allocation? And that also follows a scaling law. There's also a scaling law as you increase the amount of data that you train a model on. And there's sort of an interaction between all of these. And so there's a sense in which I think one analogy people sometimes make for thinking about this is that it's kind of like statistical physics. There's sort of this amazing fact about gases where you could try to reason about a gas in terms of every gas particle, but it turns out that there's like a few things like temperature and pressure and volume and entropy that really tell the story. And similarly, there seem to be a few variables, maybe for neural networks that also tell a large, at least high-level story of neural networks. And this is in some ways, like, almost the opposite of what we're talking about with circuits, where we are going and really getting into the details. Here you have this extremely abstract view where there's just a few variables that matter, and you go and you have them interact.

I see. Okay. So one reason that people might be excited about this is that maybe we're learning from all of this experience that we're building up some very big-picture, fundamental claims about the relationship between the amount of compute that you have in training, the number of parameters, and the amount of data that you feed into it and how intelligent, how high is the performance of a system? I suppose that's, like, amazing in itself. We're like learning perhaps some fundamental laws about the nature of intelligence and information processing. Then I guess it's even more exciting when you kind of project forward and, like, well, in the future, we'll have this much data and this much compute, and this will be able to support a model with this many parameters. And it probably helps with our ability to forecast where we'll be in the future. And it seems like every couple of months, there's, like, new, interesting scaling laws that allow you to reason about different things. So just a few weeks ago, a new paper came out on scaling laws for transfer learning. And transfer learning is this popular thing where oftentimes one person will train a large model, and then other people who can't train large models themselves and don't have a lot of data, but have small amounts of data, will fine-tune that large model to their task. So they go and train it very briefly, starting with a trained model that somebody trained on different data and then going and training it a little bit on their data. And it turns out that there is actually a scaling law. There's sort of a scaling law that very similar to, at least for how model size affects us and sort of affects there being, like, in some sense, an exchange rate between the data that you trained on and the data that you're testing on. This is some really lovely work by Danny Hernandez and his—yeah, Danny Hernandez has been on the show. We'll stick up a link to that episode.

I know it's not your area, but what do you think these people asking about scaling laws maybe should know about them? And is there any way, perhaps, that they might misunderstand them?

Yeah, well, I think that they are probably focused on them because of what they say about neural network capabilities in the future. And I think that is a really important reason to care about scaling laws. I think that's something that people maybe underappreciate is that scaling laws are much more general than that and actually may be very useful for reasoning about the properties of neural networks that we'll build in the future more generally than just sort of their loss. And the really exciting version of this to me—and here I'm just really inspired by things that Jared Kaplan has mentioned to me and probably just giving a worse version of his thoughts—Jared is one of the people who's done a lot of leading work in scaling laws—is that maybe there's scaling laws for safety. Maybe there's some sense in which whether a model is aligned with you or not may be a function of model size and sort of how much signal you give it to do the sort of human-aligned task. And we might be able to reason about the safety of models that are larger than the models we can presently build. And if that's true, that would be huge. Where I think there's this way in which safety is always playing catch up right now. And if we could go and create a way to think about safety in terms of scaling laws and not have to play catch up, I think that would be incredibly remarkable. And so I think that's something that, separate from the capabilities implications of safety laws, is a reason to be really excited about them.

Interesting. So the idea there would be that on the x axis we have something like how much human feedback have we given the system in order to train it to make sure that it understands our preferences. And then on the y axis you've got how often does it mess up and totally misunderstand our preferences. And I suppose if you know what the relationship between those two things is—and I suppose you'd also need to throw in probably there's going to be other factors that affect this relationship like how big is the network, how complicated is the task or something—but that would allow you to project how much input are you going to need for some future model that maybe doesn't even exist yet.

Yeah, or like one really crazy picture you might have is like maybe you sort of have two axes and then you have at every point sort of how aligned the model is with you or something like this. And maybe that's a function of the amount of sort of human preference signal you gave it and the model size or something. And maybe there's in some sense like a phase change somewhere where there's the aligned regime and the unaligned regime or something like this. This is sort of extremely speculative, but if something like that was true, that would be really remarkable, I think.

Are there other things that you think people should know about scaling laws or perhaps focus more on? I guess one other thing is that scaling laws show these really smooth trends for the overall loss if you make them larger, but they actually show discontinuous trends if you try to do the same kind of analysis. And you look at very specific capabilities, like how well the model does certain kinds of arithmetic, you'll actually see these sort of discontinuous jumps. And so there's sort of these—I don't know, it's tempting to think that there's maybe some kind of phase change or there's something discontinuous going on at those points. I think it connects a little bit to why it's important to think about interpretability and to think about these systems and to sort of look carefully at them, because it actually sort of suggests that these kind of predictions don't—at least presently—seem to tell us the whole story of what's going to happen with these larger systems. And in fact, there's something complex going on, at least in terms of these finer-grained things that we'd still like to understand.

Okay, yeah, so the fact that we've seen at least some discontinuities so far means that, well, maybe there'll be discontinuities in the future. We shouldn't rule that out and assume it's all smooth.

Well, and it sort of suggests that the smooth loss is actually like, maybe the result of many discontinuous. There's lots of different small capabilities or something that are all improving at different points, and they're sort of discontinuously improving, and that sort of in aggregate creates this smooth transition. But that's maybe for more specific things. In fact, the typical case may be this more discontinuous thing, and it just means that there's a lot more room for us to suddenly be surprised by the capabilities of larger models than you might think. And I think that's a reason why we should be looking very carefully at what's going on inside large models.

All right, I'll fish again. Is there anything else that people maybe should know about or give more emphasis to with regards to scaling laws?

Well, maybe one other thing is that this allows us to have a much more sort of systematic and sort of rigorous engineering approach to working with neural networks, where rather than sort of taking shots in the dark, we're sort of developing this picture of this much more systematic way to think about how we should expect neural networks to perform and what really matters. And I think there's a way in which I feel safer in a world where that's how designing neural networks is approached. And this is sort of like, much less an argument about a concrete way that it's going to intervene and more just that's the flavor of research and the flavor of AI development that I would feel best about if we could have it. And so that's another thing that I think feels important to add about scaling laws.

So I think one thing that people might be slightly sad about with these scaling laws is that they kind of suggest that bigger is better—bigger models, more compute. It all makes a big difference to performance, I guess. Most people obviously don't have access to the biggest models or the most amount of compute. There's only a few centers that have that kind of massive funding. What does that imply to people who are doing research but have smaller budgets and access to fewer resources? Can they still contribute or are they at risk maybe of getting a little bit cut out of things?

Yeah, I think machine learning sort of as a field has been structurally in a very strange position for the last number of years where quite a large number of actors could go and do relatively state-of-the-art work. And I think that's not true in many other fields. If you're like doing aerospace engineering, the number of people who can go and build a full-scale rocket ship to go and test is presumably a very small subset of the people working in aerospace engineering that's relevant to that. And if you're working on particle physics, CERN and the LHC are these enormous things. And it's hard for me to compete particles. So, yeah, your basement cyclotron probably isn't going to keep up. What is one to do? Well, I think one answer, and maybe this is a little bit of an answer that I have a sort of selfish interest for promoting, is that interpretability doesn't require this. Interpretability allows us to train models once and then potentially for many people to go and study it and try to understand these models. And so I think it's possible that maybe rather than having everybody be training a million models, we might have a smaller number of actors training models and people studying them. Although I think there still are questions about in what ways can we responsibly and safely provide all those models to everyone? I guess another answer is just to try to do small-scale work that allows us to understand this bigger-scale picture or to understand this larger-scale picture that we're starting to see through scaling laws. And I think it's probably possible if you're rigorous enough and careful enough to be able to say useful things. In fact, I think we've seen a number of scaling laws papers that are sort of operating in smaller-scale regimes and seem quite interesting.

So one listener wrote in a question that was kind of along the lines of, it seems like a lot of progress to date has been people using the same algorithms or reasonably similar algorithms over time, but throwing much more data at them, much more compute at them, and then seeing how much can their performance go up. Yeah, that would—like, have there been interesting advancements in the underlying algorithms? And are the algorithms very different? And maybe how much of the overall progress that we've seen has been driven by algorithms—not data and compute increases?

There certainly have been algorithmic improvements. I guess the transition to Transformers is probably the most striking one in recent years. And actually, we're just talking about Danny Hernandez all the time today, but he has a blog post on the OpenAI blog about the efficiency of training neural networks over time and how algorithmic improvements have increased that over time. And of course, there's been lots of really exciting work in other domains. I think progress on graph neural networks is really cool. I think just...

In general, machine learning is going in all sorts of directions, and it's very interesting. But I kind of want to push back against the premise of this question that there's something like—I think in some ways this question is sort of saying that it's sort of boring if what's going on is increases in compute leading to greater capabilities, or that sort of being the story.

And I actually think that it's not at all boring and in fact is really beautiful that maybe a small number of things are really driving the big picture story here, as scaling law sort of suggests, and then giving rise to this immense amount of structure that we can look at. It's sort of like saying, "Oh, the universe is boring because the universe runs on really simple physical laws," or "evolution is boring. It just cares about survival of the same thing. It's just the same thing all over all those millions of years." But I think it's not boring at all, and I think it's part of the beauty of it. And yeah, I think the things that we are observing that result from this are really gorgeous. Something you're saying. It's like it's another beautiful example of being able to get unlimited complexity and accomplish so much with just very simple underlying processes.

Yeah, I think that's the story that is emerging for me. And who knows if it's what I will believe in a couple of years or whether anyone agrees with me, but that's my take. All right, let's push on and talk about something pretty exciting that's happened recently, which is that you and some of your colleagues have recently left OpenAI to start a new project, which you've decided to call Anthropic, that was announced publicly just very recently. And I see you've now got a website up with a bunch of vacancies being advertised. To set the scene for everyone, though, what's the vision for how Anthropic is going to contribute to the AI and AI safety space?

Yeah, Anthropic is a new AI research company focused on the safety of large models. We're trying to make large models reliable, interpretable, and steerable. And so, concretely, that means that we're training large models, studying interpretability and human feedback in the context of large models, and thinking a lot about the implications of that for society.

Okay, so I guess one of the most distinctive things about Anthropic is that you're focusing on the safety and I guess, construction of large models and I guess maybe the largest models that you can manage. And I guess some people might be worried that since you're working on those large models, which are, I guess, the models of greatest concern, you might also be accelerating progress towards larger and therefore more dangerous models. Yeah. What do you think about that?

Yeah, I have a lot of sympathy for that concern. I think that large models are probably the greatest source of AI risk in the foreseeable future. And so it's pretty reasonable to be concerned about things that might accelerate that. But that's exactly why I think it's really important for us to be working on their safety, and we should try to be responsible and thoughtful about working with large models. But yeah, I think if you know that something is coming and you think that it might be dangerous, then you probably want to do safety work focused on that.

Okay, so it sounds like your general take is that it is possible that this kind of work might accelerate the development of larger models, but that is kind of outweighed—that we should do it anyway because those are the things that might do the most damage. Working and understanding them and figuring out how to make them safe is the most effective way to work on safety. And that's kind of the dominant consideration.

Yeah, so I think to be a little bit more precise, I actually think that there's sort of one primary argument and then there's actually two secondary arguments that are also worth considering. So I'll give you the primary argument first: it seems to me like large models are sort of something really dramatic happening, basically inevitable at this point. Society seems like we're sort of on a pretty straightforward path of people going and building larger and larger models. And at the same time, it seems like these large models are pretty qualitatively different from smaller models. So a concrete example of that is some of the largest language models will, in some sense, lie to you. You go and you ask them questions, and they demonstratively know the answer to the question, but they still give you the wrong answer when you ask them. Maybe it would be more accurate to characterize that as bullshit rather than lying, but they're in some sense not giving you—they're sort of not giving you true answers that they know. And that's not a problem you can observe in smaller models because smaller models struggle enough with saying something coherent that you don't have any problems with lying.

And so if you believe then that large models are inevitable and that large models are qualitatively different enough that the most valuable safety research is going to need to work on them to go and be effective, then I think you're sort of left with two options. Behind door one, you can sort of go and do safety research on models that other people have produced. And that means that you probably have, depending on the exact setup, something like a one to three-year lag on the models that you're working with. And in that world, I think safety research is sort of perpetually playing catch up where it's working on these older models and then trying to do safety research with models that are several years behind. And the second option is that you try to go and create a scaling effort and do scaling research that's very tightly integrated with safety such that you can be doing safety on the largest model.

So I guess I feel like a concern that I sometimes hear in the safety community is: what if the important safety research can only be done at the end? Someday we're going to build really powerful systems, really transformative AI systems, and what if the valuable safety research can only be done in those systems? Well, I feel like if you're worried about that kind of thing, then you should be pretty excited. And it seems really important to go and be able to shave off a couple of years, and that sort of is effectively buying us more time to work on those systems. So that's the primary argument.

Okay, so a key motivation is that at the moment, safety research is always behind the cutting edge because the cutting edge is these largest models, and the safety research isn't necessarily tied into the stuff that's really at the cutting edge of capabilities. And by making Anthropic focused on the largest models, you're keeping it up to pace so that if there are new considerations raised by the latest advances in capabilities, you'll be there with a safety mindset in order to try to apply safety thinking to that. I guess to some extent it has to be an open or an empirical question whether this is net beneficial. At least if you think that advancing the rate of progress within the largest models is harmful, then it's kind of an empirical question: how much do you speed those up versus how much do you make them safer? And one has to make a judgment call there. But I guess it sounds like you think the effect that you have on accelerating the progress or accelerating the size is relatively modest compared to the gain an insight that you have to how they work and how to make them work better.

Yeah, I mean, I would actually probably frame potential harm, not as like—I don't think that I see capabilities progresses as intrinsically harmful. But I do worry a lot about races and accelerating races and creating incentives where people are going to cut corners on safety. But yeah, I think we can be pretty responsible about that. And I also think that at least if you're sort of taking an empirical approach to safety, it's really hard to make progress if large models are just really qualitatively different and you don't have access to them. And so yeah, I kind of think that safety research playing catch up is a losing game, and we need to go and work on large models.

You said that if kind of you're not working with the largest models, then plausibly, you could be kind of one to three years behind the cutting edge or playing catch up on that kind of timeline. Where does that number come from?

Yeah, so this is just my estimate, and I think it depends a lot on the exact institutional setup. So I think if you're at a company that's training large models but aren't really closely integrated with the scaling efforts in particular, then you might be looking at something more like a one-year delay, maybe even more like six months. And that's due to your need for this specialized infrastructure for working with these models, due to the fact that these models cost a lot to be able to do anything with, just like they cost a lot to train. And from the fact that expertise may only reside with the people who are training these models and so you might not know how to work with them. And just that it's really hard to integrate with a project while it's actively being worked on. So you end up having to wait for it to be polished and then kind of hand it out in a way that it's possible for external people to use. And that happens at some substantial delay. And you probably need to go and get engineers who have expertise working with these models to help you build the tooling you need to do useful work on top of them. On the other extreme, if you're at an academic lab, it seems like that gap is growing pretty dramatically. And my guess is that a lot of academic labs only have a couple of GPUs and don't have expertise training even moderately large models and are doing work with models that's very far behind the state of the art. And I think there's a spectrum between there. And so I think that's where my one to three-year estimate came from.

I see. Yeah. Is it a problem in itself that kind of academia is being—it sounds a bit like—pushed out or elbowed out by the fact that the funding requirements have grown so large that they're kind of beyond the normal size of academic grants?

Yeah. I think it's a really tricky problem, and it might be interesting to look to other areas of science that have higher capital costs to learn about how they operate. So I think if you were, like, designing airplanes, you wouldn't expect research labs to be able to go and build their own airplane, or certainly—there's many—something like CERN or the Hubble Space Telescope isn't something that an individual lab can do. And these fields develop mechanisms for being able to go and coordinate and go and do high capital cost experiments. And so I think it's a sort of painful thing that we're going through right now as we make that transition from a field that's more like mathematics or traditional computer science, where every PhD student has sort of the maximum amount of useful resources, and you don't really need anything beyond—it's just your own ability to think—to a field that is more like some of these other fields where you have really high capital costs and you need to organize things differently.

Okay. Earlier you said there were three different lines of argument that you were going to use to explain why this was net beneficial. And that was the first one. Yeah. What's the second?

Well, I think somebody might say, "What if AI systems are just so dangerous and safety is so hard that even if you're careful and you try to work on safety, just like deep learning is just bad for the world?" So let's assume for the sake of argument that's true. That just—I'm pretty optimistic that's not what the world looks like. But let's assume for the sake of argument that we're in a world where deep learning is just intrinsically dangerous and really large models at some point become harmful, and it's very difficult to avoid that. Well, it seems to me that in such a world, we probably want to cause a pretty dramatic slowdown in work towards large models. Maybe we should pass a moratorium on models above a certain size or something very dramatic like that. And I don't see that happening by default. My guess is that the only way that's going to happen is if we have really compelling evidence that large models are very dangerous and necessarily so. And necessarily so. And I don't see a way that you're going to get that kind of evidence if you aren't working with large models. And so I think even people who even hold that view should be excited to go and do safety research and really critical analysis of large models, because I think that's the only way that you're—I think the effect of this research is going to be pretty—to any acceleration, is going to be kind of marginal. There's already lots of actors working in this space. And I think the main thing you should be optimizing for, if that's your view, is the possibility that you can get the evidence that could build consensus around a really dramatic slowdown.

Yeah, okay. That one makes sense to me. What's the third one?

Well, I guess my final argument is: we live in a world with a lot of ongoing moral catastrophes—factory farming, global poverty, neglected diseases, and also oncoming disasters like climate change or potential sources of existential risk. And those are extremely tough problems. But I think once we're supposing we're in a world where this kind of dramatic risk from AI exists, I think you also have to be supposing that you're in a world where you have AI systems that could potentially really dramatically improve these problems if they were aligned. And so just as much as I feel an obligation to worry intensely about AI risk and ways that AI could go wrong, I think we also have a great responsibility to make sure that we aren't missing the opportunity for AI to dramatically improve the world. I think that could be equally bad and an equally big failure.

Yeah. So this is an argument that's been around for a while, which kind of runs: even if you're someone who thinks that faster advances in artificial intelligence would be dangerous because they're happening so fast, that we won't necessarily have time to fully prepare for them and figure out all of the additional safety requirements that we need for more advanced, larger models, it could nonetheless come out safer on net, all things considered. If those new technologies—if those massive advances in artificial intelligence—could also reduce the risk of nuclear war and reduce the risk of bioweapons being used or a terrible pandemic or just a war between countries. And depending on kind of the ratio you see in the risk of this new technology versus all of the other risks that technology might then obviate and supersede, then even if it's dangerous to go faster, it can come out positive on balance because you're going to spend less time in the meantime with all of those other concerns—swords of Damocles hanging over you. I think we could actually put a link up to a little mathematical model that Nick Becksted made a couple of years ago to try to see how large is this effect.

Yeah, that's exactly right. Although I also want to highlight—and I think this depends a lot on your moral views—but it's also important for me to weigh not just existential risks that might harm the far future, but to also take seriously present ongoing harms that could be mitigated. And I understand that from a lot of you utilitarian views, that might be the only thing that should matter, but honestly, I just feel really upset about factory farming and about global poverty specifically. I think in some ways I sort of emotionally wish I could work on those instead of AI if I thought that was the optimal thing for me to do. Yeah. And this isn't to say that it should be the dominating concern, but I don't think it should be forgotten.

Yeah. I think part of the reason the argument is often phrased that way, just like comparing risk against risk, is that it's so clean when you're just always dealing in the same currency. You don't have to make any conversions from one thing to another because plausibly—depending on what parameters you stick in—you could just say, "Well, this argument is actually self-undermining. There's no benefit in any currency," so it's very convenient. But it also would be very nice to solve all of these other problems that bedevil us all the time, and I guess—but potentially live a lot longer because we'll make huge advances in biomedical science and so on.

I guess I thought, yeah, you might lean even more heavily on the argument that Anthropic is kind of a drop in the ocean of all AI research and that really—how much is it going to be doing to actually spur larger models than have been created elsewhere and especially like more dangerous larger models? It seems like you're potentially going to be a much larger fraction of the safety-focused thinking and safety-focused research than you would be of large model research, which is going on in many places.

Yeah, I think the arguments here are actually pretty subtle and nuanced. So first we could try to make an argument on ratios, and you'd have to account for three things. The first, as you say, is sort of like maybe the fraction of Anthropic's effort that goes into safety or capabilities, although I think—I don't really buy that. I think that's maybe not the right distinction to make if the question you're trying to answer is: how do you make large AI systems safe? Those actually are very intertangled things that aren't so easy to pull apart. But yeah, I think safety is probably a bigger part of our mission than most organizations. That's sort of the reason we exist. But I think there's two other things that maybe are even larger factors. So one is: if you believe that the most effective safety research can only be done on large models, then you have to multiply the amount of safety effort that's going in by the effectiveness multiplier. And I think that effectiveness multiplier is like—I don't know—suppose that you're like comparing Anthropic with a lab that does 5% safety research, and let's say that Anthropic, even though I sort of think this isn't quite the right way to think about things, is doing 50% safety research or 70% safety research or something like that. But then you think that it's actually like a 20 to one effectiveness multiplier or something like this. That might be the swamping thing. The other thing is: if the thing that you're really worried about is whether you're accelerating races or causing the field to race, I think it's not how much of your effort are you spending on training large models or something like this that's relevant. It's actually like really nuanced things about how you conduct yourself in the field. Yeah, I think it matters a lot exactly what kind of research you do. I think it matters a lot if you're doing models that you think are just below the largest models that are being trained or that are larger than them. I think actually something that people really underrate is that I think now that we're getting into the regime where AI experiments cost potentially many millions of dollars, I think that marketing matters a lot. I think that the kinds of things that cause organizations to be willing to go and invest huge amounts of resources in something—a flashy demo—may go a long ways further than a very dry reference to a technical result. And so I think that there's actually a lot of detail there. So in any case, stepping—zooming out again. So we have perhaps this growth—like how much of your effort is going to one thing or another, which may not quite be the right framing if you're sort of entirely focused on building safe systems or if you're sort of really tightly integrating those things. But then you also have to multiply by the effectiveness multiplier you're getting on safety and also by, I don't know, the carefulness multiplier on how you're affecting race dynamics. And my guess is the one that for me is the largest one is my belief about the effectiveness multiplier on safety. And I think we already have some examples of that, right? Like the multimodal neurons work that I did wouldn't have been possible without access to—this is work that I did while at OpenAI, but it wouldn't have been possible to do without access to state-of-the-art models and being able to go and tightly integrate with that effort. Or the learning from human preferences paper again relied on access to large models to be able to go and demonstrate that. And so yeah, I think a lot of safety research can get a large multiplier if you have access to these models.

Yeah, we're talking about Anthropic, and this has been a big diversion into this question of safety and large models. Before we go back to our topic, I just want to, I guess ask one more question, which is just—maybe this is going to show my naivete—but it seems like just making the models larger, doesn't that just mean like adding a whole lot more compute to the thing? It's kind of the same thing, but you are spending more money on the hardware and spending more money on the electricity. Maybe if there's no amazing insight here that you're demonstrating by just running large models, then maybe it doesn't really help or it doesn't really promote larger models just to be doing it. Maybe even there's a chip shortage at the moment, right, and there's only so many ability to make these chips. So if you just buy up the chips then—I guess, or there's only so much compute to go around. So maybe whoever's doing it doesn't really make that much difference to the total new AI safety agenda—go and buy all GPUs and use them for crypto mining. There we go. And then use the crypto mining to buy more GPUs.

There was this work by my colleague Danny Hernandez where he just made a graph of the amount of compute used to train the largest models as a function of time. And he found that the amount of compute that people use on the largest models doubles every three and a half months or so. And I think to some extent marginal actors probably only affect that a little bit. It's sort of this strong trend that's a function of lots of people. But at the same time, I think that every time people see people using more compute, that sort of reinforces that trend and can sometimes accelerate it a little bit more.

Okay. Yeah, that makes sense. All right, let's go back to Anthropic as an org. We've been talking about large models as kind of one distinctive aspect of the research agenda. Is there any other notable aspects of it that are worth highlighting for people?

Yeah, I guess being focused on safety is pretty unusual, and I think us making a big bet on interpretability is also a pretty unusual thing to be doing. I guess another slightly unusual thing is just also being really focused on thinking about how this work affects society and trying to tightly integrate things, and I guess actually just really trying to tightly integrate all these pieces together. So trying to integrate the scaling work and the interpretability work and the human feedback work and the societal impacts work and sort of unite those all in a single package and in a tightly integrated team is pretty unusual. I think something that people probably underappreciate is: if you want to do any kind of safety research on a large model, it's not just a question of having access to that large model, but these models become very unwieldy and very difficult to work with because they're just so enormous that a lot of the infrastructure that you'd normally work with just doesn't work. And so for whatever kind of safety research you want to do, you actually probably have to do a lot of specialized engineering and infrastructure building to enable safety research on those large models.

I had no idea.

Yeah, and you need a lot of expertise from people who have experience working with large models. And so I think there's a lot of things that can only happen when you really tightly integrate things. I think people maybe don't appreciate the extent to which scaling is really two separate things. There's training large models, but there's also this work on scaling laws of going and predicting them. And one of our big focuses is sort of taking scaling laws to every domain. So for interpretability, we know that there's these points where if you look at the trend for very specific tasks, you get these abrupt transitions where a model goes from—a smaller model can't do arithmetic, and then suddenly a larger model can. And we think that probably is a phase change in the underlying circuit. So can we study that? Or for human feedback, can we understand how sort of the scaling of Goodhart's law—what happens if you go and if you have models of different sizes and you…

Also, have different amounts of human feedback? Can you understand empirically, or can you understand in terms of something like scaling laws, whether a model will correctly generalize to the human feedback or sort of goodhart itself?

And so those are questions that sort of bringing this kind of thinking about scaling hopefully allows us to think about it and maybe allows us to start doing work that isn't just about making the present model safe, but even is about making future models safe. And so all of that requires you to be tightly integrating the expertise that you generate by working on scaling with safety.

Okay.

Yeah, just to paint a picture for people, kind of how many staff are at Anthropic now and do you have an office, are you based in any particular location?

Yeah. So we're based in San Francisco. We are 17 people right now, and I hope that we will soon have an office. We do not yet.

All right.

Yeah, well, I guess this is a good time to be remote.

Yes. I think still to this day, most of the people I know in the UK are working remotely, but I think it's going to be any month now that we're going to be back in the office properly.

Who's funding the whole operation?

Yeah, Anthropic's series A was led by Jan Talon, who I guess cares a lot about AI safety and was joined by a number of individual funders rather than VCs.

Yeah, I guess given that you're working on these large models, imagining that one of your primary expenses is just buying compute and hardware and all of that kind of thing.

Yeah, that's right. I think that actually, in a lot of ways, AI is maybe starting to look more like other industries where you have really intense capital costs for research, like biotech, and you have to spend lots of money on reagents. And similarly, we have to spend lots of money on compute for our experiments. And so I think that's in some ways, an industry that sort of is. Natural to compare us to, yeah, less like Etsy. More like manufacturing cars.

What's the business structure? Is this kind of for-profit or more of a non-profit model?

Yeah, Anthropic is for-profit. Technically, we're a public benefit corporation, but that just means that we are not legally obligated to maximize shareholder value. I think, for what it's worth, that might be true of all corporations, or at least when this has been taken to court. Corporations have a lot of yeah, I think in practice it's very difficult to go and do this, but a public benefit corporation makes sure, yeah, makes for sure.

Yeah. And I'm not super familiar with public benefit corporations, but do you end up with something else in your charter, like in addition to making profit that's kind of a specified goal of the organization?

Yeah, so you go and you put a mission in your charter that you're allowed to go and prioritize over shareholder value.

Okay. And I guess here it'll be something like developing AI for the benefit of everyone and sharing the spoils, something to that extent.

Yeah.

Okay. So as a public benefit corporation, you can take investment from investors who are aiming to make some money back. You can also sell products and use that to fund the growth. It's just that you also have other goals in addition to making money at the end of the day. And I guess the board and so on can have a vision for the company that's not just maximizing dividends or maximizing shareholder value.

Yeah. So how does one kind of monetize the kind of research that you're doing? How do you actually kind of get the revenue in order to keep growing?

Yeah, well, right now and for the near future, we're just focused on research. But I think that we're kind of in an interesting point in the trajectory of AI where I think in the past, the economic value of AI has been bottlenecked on models being capable at all and being able to do anything useful at all. And it seems like we're maybe transitioning out of that phase a little bit, that actually now we can build. And I think a lot of organizations can build models that are in some sense capable of doing useful things, but they aren't reliable, they aren't trustworthy. Nobody understands how they work, and that's becoming the bottleneck on their use and on economic value from them. And so I think that makes us hopeful that there will be opportunities if you can become really good at building systems that are safe and reliable and trustworthy that there will be economic value from that.

Yeah, so I guess I can see multiple different business models here. So one would be like actually designing these systems that are very high stakes yourself because you have the expertise in certifying that they are safe and are going to act as desired and predicted. Another one might be kind of doing consulting, using that expertise to help other companies fix their models. And I suppose another might be selling tools, that interpretability tools and so on, that will allow everyone else to look inside their models and help to make them safe themselves. I guess probably all of these are plausibly on the table for the future at the moment.

Yeah, I mean, we're super early stage. I think the most likely thing would be training large models ourselves and making them safe. We think that it's probably important to integrate safety and the design of models. I should also say that we plan to share the work that we do on safety and share that with the world because we ultimately just want to help people build safe models and don't want to hoard safety knowledge or something like this. Keep it all for yourself.

All right, so I guess I'm imagining that Anthropic, a pretty new organization, is on the hunt for people to fill all kinds of different roles.

Yeah. Are there any particular roles that you're hiring for at the moment that are worth highlighting?

Yeah, well, I think there's three roles that are in particular really high priority for us, and they're maybe not the roles that people might expect. I think there's often an assumption that the highest impact and most important and in-demand roles at a machine learning research organization are going to be machine learning research roles. But in fact, the things that we really need are there's two engineering roles that are really important, and there's a security role that I think is absolutely essential that we find someone good for.

Okay, yeah, let's take those one by one. What's the engineering role? And I guess why is it important enough to really want to highlight it?

So really at the core of Anthropic's ability to go and do our research, to go and explore safety in these large models is the ability to work with large models. And at the core of that is our cluster, basically a supercomputer that we're using to do all of our work on. And so there's really two roles related to that are central to our ability to go and do productive research. The first one is what we're calling our infrastructure engineering role. And this is really someone who's going to be responsible for going and keeping that supercomputer running and having the tooling we need to interact with it. And I think there's two ways you could sort of think about why it's such a high-impact role for us. One way to think about it is that right now all of that work is being done by researchers. I was listening to one of the old ADK podcasts a while back where you were discussing operations, I guess it was, with Tanya Singh, and she commented on, know, by doing operations work, she was able to go and free up for each hour that she spent. She was able to free up more than 1 hour of researcher time per hour of operations time that she did. And I think that's absolutely something similar would be true here. That somebody who took on this infrastructure engineering role would be freeing up way more than 1 hour of researcher time for every hour they spent on it. And the reason there is just that they'll actually have the specialist skills to know how to keep a supercomputer, actually have the special skills. Yes, they will really know what they're doing instead of being experts in a different topic who are playing at knowing what they're doing.

Yeah, so that's exactly right. There's another way you could think about it, which is we are spending tens of millions of dollars a year on this cluster to allow us to go and do our safety research on large models. And sometimes the cluster is unusable or things break unnecessarily. And we think that it's quite likely that somebody who took on this role could increase the uptime and the reliability by more than 10%. And so in some sense, that would be equivalent to going and giving millions of dollars to an organization that's focused on the safety of large models. And so if you're excited about making large models safe, that could be a really high-impact thing to do.

Yeah, so I don't really know how you network computers or make a supercomputer, but I'm guessing it's a little bit technical. Are the people in the audience who might actually have the skills to operate a computer of this scale, are they going to know who they are, or is there anything we can say to make sure that they are aware that they ought to put their name forward?

Yeah, so everything that we're doing is based on Kubernetes, which is, I guess, a framework for working with large distributed systems. I think that having expertise in networking and I mean, I think if somebody had experience with GPUs or things like that could all be useful. But I think expertise with Kubernetes and to some extent doing sort of SysOps work would be things that would be relevant to this infrastructure engineering role.

Okay, nice. So that was the first engineering role. What was the second?

So the second engineering role, and we're calling it our systems researcher role, is also really centrally related to this cluster because there's sort of this whole challenge of figuring out if you're running these giant models, how do you effectively fit them and set them up on the supercomputer, how do you break them apart? How do you efficiently go and make all the networking work? How do you go and get everything to run as efficiently as possible? And there's not a lot of prior art on this because these really giant models, there's not that many places that have worked with them. And so there's really this novel engineering problem of how you efficiently run these models on distributed systems.

So this is something along the lines of, I guess you're running particular kind of processes, particular kinds of algorithms, and you want to figure out how can you distribute and order those operations most efficiently when you're running them across, I guess, tons of different processes which each might have different specialties or particular kinds of operations they can do most efficiently. I don't understand how computers work, but it seems like is that in the ballpark?

Yeah, that's right. So you have a bunch of computers, and then each of the computers has a bunch of GPUs. And there's a lot of questions about how you go and lay out these large models, and you're multiplying all these large matrices. How do you lay that out across these GPUs? How should they all talk to each other? What's the most efficient way to do things? And there's both sort of high-level questions about what the most efficient way to organize things is. And you have to think about memory bandwidth and how long it'll take to go and load things between different kinds of memory and the network and stuff like this. So there's this interplay of these high-level considerations and then also very low-level considerations of just how to make things be very efficient. And the reward there from doing that well, I guess is you get more actually useful computational work out of the same amount of hardware and the same electricity bill and so on.

Yeah, that's exactly right. And so again, I think it's very easy to imagine that someone who is doing this role could go and increase our efficiency by at least 10%. And so again, that's equivalent to going and providing millions of dollars to an organization that's focused on safety.

Amazing.

Yeah. So if this is kind of a role that I guess isn't a traditional tech role or is it the kind of thing that lots of people might have worked on before, how could someone tell if they have kind of the proto-skills that might allow them to grow into a position like this?

Yeah, so I think that some of the things that would make someone effective at this role are I think if somebody has a lot of experience thinking about efficiency in low-level hardware, that would be one thing that might be a really good indicator that they could be good at this. I think also having a lot of experience thinking about distributed systems and especially thinking about efficiency in distributed systems could be another good indicator. I think that they'd have to learn some stuff about machine learning on the job. But I think for this role, it's primarily having strong engineering skills and especially strong engineering skills related to efficiency in distributed systems would be the things to look for.

Cool.

Yeah. It's slightly funny to me that it sounds like you feel like you really have to justify why these roles are high impact in terms of the amount of equivalent money donated. Seems like if what Anthropic is doing is useful in the big picture, like having good people running this enormous supercomputer system seems like it's obviously going to be important as well. You could imagine if it's running competently, then going to be wasting a whole lot of hardware and also probably just have tons of downtime that's going to slow things down. So it's not a very hard sell to me at least.

Well, I often talk to people, and I get the impression that they're seeing engineering roles as sort of these things of secondary importance. That often they're trying to figure out how they can become transition from being an engineer to being a machine learning researcher or sort of assuming that the engineering roles are less important. And so I guess really the reason that I'm saying all of this is just to emphasize how tremendously high impact these engineering roles are. I think that one of the sort of most effective people that I know is Tom Brown, who is really primarily an engineer, and he was the lead author of GPT-Three and yeah, just has been tremendously impactful in enabling research on large models.

Yeah, I guess it seems just blatantly obvious that you need someone to do this work. So I guess that the logic that's going on in people's heads is for these positions, you don't need someone who's especially passionate about AI safety as a problem in the world. Can't you just hire someone who doesn't even necessarily have to care all that much about the mission and just pay the money to run the supercomputer?

Yeah, I can see how people get that in their heads. It just seems like when you talk to organizations that are trying to do work like this, they say, no, it's actually really hard to get the best engineers, and it would really make a difference if we could find someone who was better at the job.

Yeah, that's absolutely right.

Yeah. Do you have a theory for why it is that it seems like quite often you need people who care about the importance of the mission and care about AI safety when their job doesn't seem to be the AI research science directly? It's like building all the tools that enable people to do that research.

Well, the first thing is, I think just people who are really good at these skills are rare and hard to find. Even if you weren't considering alignment, these are just hard roles to hire for. Just like getting good machine learning researchers is a hard role to hire for, and you are more likely to get somebody who's really extraordinarily good at it if part of the reason they're joining you is that they care about your mission. But I think a second reason is just that it's really healthy for an organization to have the entire organization care about the mission rather than having some kind of weird bifurcation where some portion of people care and some portion of people don't. Maybe one final thing is, at least in Anthropic, everything's very tightly integrated, and it's not like these teams are siloed or there's people working one thing and they're siloed from people working on something else. We're all working very closely together, and there's very blurry lines and lots of collaboration, and so I think it would just be a very strange culture if you had some set of people who cared about safety who were doing more researchy things and yeah, I think it would be a very strange situation.

Yeah, I guess as you're saying, with ML taking off as much as it is, I guess people who are able to do jobs like this are probably pretty sought after. There's a lot of headhunting, so it's hard to get the best people. If someone doesn't care about the mission that Anthropic is engaged in, then it might be a lot harder to retain someone long term. And ideally, you probably don't want to have annual turnover on the person who has put together and really knows how your compute stack works. I can see that leading to a lot of headaches. I think that wouldn't be terribly fun.

Cool. Okay, so that was the two engineering roles. There was a third one, right? And that was security.

Yeah. Well, earlier you were saying that maybe it was kind of self-evident why the engineering roles were so important. And if that's true, then maybe it's even more self-evident why the security role is really important. If you're going to be building these powerful systems, it's really important that you be secure. You don't want anyone to just be able to go and come and grab all of the research that you've done, especially if a lot of your work, in addition to safety, there's capabilities, relevant components.

Yeah. The whole model is that you're potentially going to be working at the frontier with stuff that you're concerned isn't quite fully baked and safe yet.

Yeah, we just don't want random hackers to be able to come and grab our models. That seems bad. It seems really bad.

I buy it.

All right, go on.

And not just because these models are potentially unsafe, or in the future, we're going to build models that could potentially be more harmful, but also because they're abusable. I think there's a lot of potential for these models to be misused by bad actors. And so I think that there are going to be bad actors who are going to be increasingly trying to go and get access to these models.

Yeah, I mean, it seems like just every tech company or every organization that's dealing with important confidential information these days really needs someone to lock down their information security and their computer security. But it's not straightforward. The kinds of threats that people face today are pretty serious, and it takes a lot of know-how to figure out how to keep information in-house.

Yeah, it's not straightforward at all, and I think it's a really easy thing to neglect. It's a really easy thing to say. Anthropic's a small organization right now. Probably nobody's going to try to do anything to us. We could leave this till later. And I think that would be a terrible mistake. I think this is something that we really want to be working on at an early stage and trying to address. Right now, is that just because once you've kind of built all of your systems, going back and figuring out how to make them safe after the fact. Once you've made all these design choices and chosen to use one piece of software over another, it's just really hard to go back and patch the mistakes that you made years or decades ago.

I think that's right. I think there's also probably something important about building a culture of taking security seriously from the start. And we're trying to do that by having some employees, especially my wonderful colleague Ben, work on security. But we're not experts, and we really need an expert to join us and be able to handle this and take the lead on it.

Totally. So is this going to be the kind of security role that people might be familiar with or similar to the sorts of work that people might do in other tech companies? Or is this slightly a weird case where people might be doing different work that a typical security person might be less familiar with?

I think that there's components that are similar, but I think we also have some challenges that are really unusual. So, for example, in addition to all of these concerns about security against sort of external threats, we're training language models to generate code. That's one of our lines of research. And we expect those models at some point to start doing bad things, and we're trying to sandbox them. And so it's pretty important for us to be thinking about how to effectively sandbox them. And so that's sort of a very, as far as I can tell, a pretty unusual flavor of a security problem. Maybe people who work with malware and need to run it in sandboxes have something a little bit similar, but it's a sort of unusual problem that comes up in machine learning and that we need to deal with.

Yeah. So this is the issue that people have recently started using language models to do programming and have been finding that these language models are remarkably capable at programming things that actually really work as well as in the long term. You might worry that you're potentially going to be playing with language models that are designed to do mischievous things because that's the kind of thing that you want to figure out, and then you don't want it, don't want that mischief done against you. So you have to figure out some way to contain the system so that it can't start damaging the computer that it's running on. Is that basically the picture?

Yeah. Or you might not be trying to do anything mischievous at all, but the model might do things anyways that aren't what you want. So perhaps at some point you're using RL to try and get these models to solve tasks and learn to solve problems and exploring safety in such models. Now you're in the situation where maybe the model tries to, it's actually quite good at programming. Maybe it tries to find some security vulnerability to go in and get itself extra reward. Or maybe in more extreme versions, it starts to generate malware. These are very speculative things, but you don't want to end up in the kind of situation where that sort of thing is a serious possibility and you haven't thought about it in advance. And so yeah, I think, again, sort of having somebody who's thought about security seriously thinking about this kind of stuff would be really valuable.

Okay, so the security role has some familiar elements and potentially some more novel elements.

Yeah. How could someone know if they're potentially a good fit to apply for that one?

I think the challenging thing about this role is that right now, we need someone to be dealing with a lot of kinds of security problems. And often when we talk to security candidates, they're focused on a particular specialty and don't sort of have the breadth of expertise to help us with the range of problems that we're facing, or in the alternatives they do. But they're really a manager at this point, and they don't have as much experience doing ground-level work. They're looking to lead a whole security team and then sort of delegate out the various responsibilities. And we're probably not big enough for that to build out a large security team. So, I think the thing that would be a really good sign would be if somebody had sort of a pretty broad range of security knowledge and was excited to go and help figure out how to make Anthropic secure.

Okay. So it's a pretty generalist role. I was just looking over all of the open positions that you've got at the moment @anthropic.com, and there's a couple of others here. One that's a little bit surprising, maybe, is the data visualization specialist.

Yeah. What are you hoping for that person to do?

Yeah, I think this is a really exciting role. If you want to work on interpretability research, there's this really interesting phenomenon where, if you look at the history of science, often new lines of research and disciplines get unlocked by having the right kind of tooling. For example, early chemistry seems to have been linked to the development of glassware that made experiments possible. And so I think there's something kind of similar for interpretability research where these models are enormous, and just being able to go and navigate through them and ask questions and look at data is difficult because they're so big. And so I think that a really powerful enabler of this kind of research is having people who are good at data visualization and can sort of be part of the inner loop of that research and figure out how to go and visualize and explore and understand all the data that

We're getting access to when we study these models. And so I think if someone has experience in data visualization and experience with a kind of web development that makes interactive data visualization possible and has some math background, this could be a really impactful way for them to support interoperability research.

All right. Yeah. Well, we've talked about four roles there. Are there any others that you want to highlight in particular before we push on?

Well, if you look at our Jobs page, you'll see both roles for ML researchers and ML engineers, and that's sort of how jobs in machine learning are often described as there being these two kinds of roles. But I think the thing that we're actually looking for underneath that is what we internally call an ML generalist, somebody who sort of isn't attached necessarily to doing research or doing engineering, but just wants to do the highest impact thing to move the research forward. So, yeah, if that resonates with people listening, that could be a really meaningful role.

Beyond that, we do have a number of other roles. I think that we're going to be hiring sort of more slowly for those roles for the rest of the year, but we'll probably be looking at them more intensely again next year. And that includes operations roles working on public policy, and a variety of research roles.

Yeah. Is there anything in particular that people should know maybe about? I suppose there's so many different roles here, it's hard to say anything in general about who's appropriate for working at these positions, but maybe is there anything about the Anthropic Culture so far that maybe people should know before they're considering applying for a job?

Yeah, well, on the sort of more technical side, I think one thing I really love about Anthropic's Culture is that we do a ton of pair programming, so researchers and engineers are just constantly pairing on different things that we're working on. People will just like, post in Slack, "I'm going to be working on this, do you want to pair with me?" And I think it's a very significant step further in the pair programming direction than any other organization I've been part of. And it's just delightful. I really love it.

I think maybe another thing that's a little unusual is we're very focused on having sort of a unified research agenda rather than just having lots of people doing their own thing. But we try really hard to set that research agenda as a group, and so we have lots of conversations sort of discussing what our research agenda should be and what our focus should be and discussing that as a group. And I think that's been really cool as well.

Yeah. So ultimately you're hoping to have an office in San Francisco. Is it possible to apply for these positions if you don't expect to be able to move to San Francisco anytime soon? Are there remote options?

Right now we're looking for people who would be eventually able to move to San Francisco, obviously after the pandemic gets resolved and after any immigration issues are worked out, but that would be our long-term aspiration for most roles; it's possible that might change in the future, but right now that would be how we would think about most roles.

All right. Well, yeah, by the time this episode comes out, the specific roles that are available might have changed a little bit, but obviously we'll stick up a link to your Vacancies page and maybe I'll stick something in the outshow about the specific roles that are on offer.

Yeah, I guess it'll also just be really exciting to see where this new project goes. It's so early, but you've kind of already made a little bit of a splash in the press. And lots of other AI labs have had these, I guess, amazing pieces of research that have gotten out and captured the imagination of people who aren't even that knowledgeable about machine learning. So hopefully Anthropic maybe is able to do the same thing.

Yeah, I mean, it's just really lovely to be working with so many wonderful colleagues and yeah. I'm just really excited about the research we're doing. I wish I could go and just want to dive into understanding what's going on inside these large models. Really just all I want.

Cool. Well, we're approaching the end at the end of this recording, so we'll let you get back to that very soon. But yeah, just before you dive back into those models, I assume you don't spend all of your time doing ML research. I guess I imagine you have some horribly, maybe slightly pointless research projects or interests on the side, because I think I find most really smart people do.

Yeah. Is there anything you've been exploring currently in the last few years that's a bit unexpected?

Yeah, so I often have some kind of small side project. For a while, I was going and doing being an amateur social scientist, using Mechanical Turk to get people to answer questions. And then I was really obsessed with trying to understand evolutionary history and understand different parts of the Tree of Life. But the thing that I've been obsessed with recently has been atmospheric dynamics. And this is kind of sad, but I guess I sort of felt like physics and biology and chemistry, those are sort of like real sciences that have these beautiful, simple theories that explain lots of things, but meteorology and atmospheric science, that doesn't have beautiful theories that explain something really nicely. But guess what? It turns out there's actually a really simple idea that explains an absurd number of things. Let me list some things that it explains. It explains why is it that hurricanes tend to hit the east side of continents but not the west side? So why is it, for instance, that Florida and Japan have lots of hurricanes, but California and Spain don't? But also, why is it that if you look at a map of the world, all of the large deserts are on two latitudinal lines? And why is it that San Francisco is wet in the winter and dry in the summer, but there's other places where it's the reverse? And finally, why is it that Jupiter has stripes? And it turns out those are all explained by that same idea. And just to highlight how crazy this is, the reason why the weather that I experience day to day in San Francisco is actually intimately related to the reason why Jupiter has stripes.

You've explained it. Okay, so the high-level idea is that there's these really large-scale atmospheric circulations called Hadley cells, where warm air rises and then moves and then falls. So warm air rises at the equator and then goes out to about 30 degrees and then falls, and then it also rises again at 60 degrees and goes back to 30 degrees and goes over to the pole. So these are called Hadley cells. And in a lot of ways, Earth's large-scale weather patterns are shaped by these lines and which one of these lines you fall behind between. And they migrate over the course of the year as the thermal equator changes. And the stripes you see on Jupiter are also Hadley cells. So it has more Hadley cells than Earth. It turns out that the number of Hadley cells that you have is a function of how fast you spin and the radius of your planet primarily, so it has more Hadley cells. And so there's this very intimate connection between large-scale weather patterns on Earth. Of course, I'm not an expert on any of this, so maybe one of your listeners is a meteorologist and can tell me that I'm all wrong and the existence of stripes that you observe on other planets. But actually, the thing that's craziest about this, to me, the thing that I find absolutely nuts is okay, so we think about phase changes like water transitioning to ice or to gas or liquids. And I guess there's this more general idea of a phase change as being like when a system goes from discontinuously, goes from one state to another. And it seems to me that if you imagine going and imagine just if you could just make Earth's radius larger and larger, holding everything else constant, at some point the number of these Hadley cells you have would change. It turns out that you have to have an odd number of per hemispheres. So you'd go from three per hemisphere to five per hemisphere. And so it seems to me like actually there must in some sense be something about phase changes in planets as a function of radius that determines large-scale weather patterns. And that's just kind of like you think of phase changes as always being about these small-scale things. But here you have this really enormous-scale thing that has a phase change. And so that's something that I found especially mind-blowing. In any case, I could ramble about this. I could literally do an entire podcast episode on this at this point, but I've been finding that an enormous amount of fun.

Yes. Okay. If you could make a planet like bigger and bigger bit by bit, and you were like, tracking the weather, you would eventually see some point where it would flip, and it would flip from having one to three or five Hadley cells quite dramatically.

Exactly. In particular. Okay, so you always expect the equator to be the wettest place, at least, I guess, if you have weather patterns that are water based, like Earth, than yeah, it turns out when air rises, as it rises, the water condenses. And so if you look at a map of precipitation on Earth, it's crazy. There's the stripe of ridiculous amounts of precipitation along the equator. It's extremely clear. And then the first Hadley cell that rises at the equator and then falls because you have three it's just 30 degrees, 60 degrees, 90 degrees. It's just divided into three chunks. And so at 30 degrees, you have a dry region, and a lot of deserts fall on 30 degrees. If you suddenly made Earth larger, or if you made Earth larger and at some point you got five Hadley cells, then I'm pretty sure this is my understanding suddenly you'd have your dry region, 90 divided by five degrees instead of 90 divided by three degrees. And so that's like kind of a yeah, I find that thought experiment really compelling and sort of kind of mind-blowing.

Yeah. All right, well, we'll stick up a link to the Wikipedia article on Hadley cells outside. That's very cool. I was out taking a walk with my colleague Neil Bauman the other day. I think he studied climate science long ago, and I was like, "Why do clouds form? I don't know." I felt like such an idiot asking this super basic question. It's like, why is there a cloud there but not right next to where the cloud is? There's also all sorts of fascinating stuff about different kinds of thunderstorms and the circumstances and wonder under which a thunderstorm can last for a long period of time. Or not.

Yeah, it turns out I don't know. I've been persuaded that there actually are really beautiful ideas in meteorology and atmospheric science, and I will endeavor to not fall into the trap of, I don't know, physics, biology, chemistry, supremacy, and thinking that those are the only sciences that have really beautiful, simple, explanatory theories.

Nice. Well, yeah, maybe we'll have an episode on climate and the weather on the 80,000 podcast at some point. We'll see if we can connect it to well, I suppose climate change is one of the world's most pressing problems.

All right, my guest today was Vita Chris Olah, thanks so much for coming on the 80,000 Hours podcast.

Chris. My pleasure. Rob.

As I mentioned in the intro, we have another episode with Chris in the works, and we hope to release it next week, or if not next week, soon after. And that one should cover topics including how Chris got where he is today without having a degree, how he managed to get his foot in the door at Google Brain and OpenAI, the journal that he founded called Distill, and how to go about explaining complex things really well, how Chris recommends writing cold emails, and a whole lot more besides that. So if you enjoyed this episode, make sure you come back for that one next week. Or the week after. And if you didn't enjoy this one, note that one has no, or at least almost no technical AI content, so you might well be able to enjoy it nonetheless.

Finally, if you're interested in using your career to work on safely guiding the development of AI like Chris is doing or working to solve any of the problems that we regularly discuss on the show, then you can apply to speak with our team one-on-one for free. For the first time in quite a while this year we've had enough advising capacity to remove the list to apply, so our team is keen to speak with plenty more of you regular podcast listeners. Our advisors can going to go over which problem it might be most effective for you to focus on, take a look over your plan and think about whether it's a good fit for you, introduce you to mentors who might be able to speed you along in your career, maybe suggest alternative roles that could really be a good fit for your skills and other things. In that general vein, you can go to 80000 Hours Speak to learn more about the service and apply if you feel like it.

The 80,000 Hours Podcast is produced by Keiran Harris. Audio mastering is by Ben Cordell. Full transcripts, including the special edition of some images and links provided by Chris for this episode are available on our website and produced, as always, by Sophia Davis Vogel. Thanks for joining. Talk to you again soon.