Michael Levin - Why Intelligence Isn't Limited To Brains.

Each one of us, and also us as the current implementation of humanity, are going to be replaced. Persistence in current form is impossible. It's impossible in biology; every species will either die out or it will change and adapt, in which case it is again not the same species. So the next question is, once you've given up the idea that you can stay exactly as you are, the next question is: what would you like to be replaced by?

Michael Levan is a distinguished professor at Tufts University. His research spans multiple fields, including developmental biology, bioengineering, and cognitive science. Levan is known for his groundbreaking research on bioelectricity, regeneration, and what he calls diverse intelligence.

What do we actually mean by Humanity? What is the important feature about being a human? Do we mean that you have to have modern Homo sapiens DNA? Do you mean that they have to have the kind of standard human organs and a human anatomy? And they better not, you know, have pieces replaced by some sort of cyborg technology? I do think that it's possible to create beings that are highly intelligent in the sense of problem-solving capacity, but that don't necessarily share these things with us. I think probably that is what we will have first. I mean, that's certainly where I see modern efforts at AI going.

Professor Levan, it's it's such an honor to have you on MLST. Thank you so much for joining us.

Thank you so much. Thank you for having me. So, um, I've just enjoyed reading your paper, "A Bridge Towards Diverse Intelligence and Humanity's Future." Can you tell us about that?

Yeah, uh, you well, uh, basically what I'm working on is considered part of the field of diverse intelligence. And this is the overall, the large project of trying to understand what it means to be an intelligent agent, in particular an embodied intelligent agent. And my claim is that we, as humans and as scientists, are really only calibrated for a very particular kind of, for for detecting only a very particular kind of intelligent agent. We're good at noticing minds that are embodied, medium size scale, medium speed, moving at three, you know, in the three-dimensional space. We're okay at noticing those things, and then we have some trouble, maybe a whale or an octopus or something like that. But, uh, I I actually think that there are many other spaces in which minds are embodied, um, and this is this is present throughout biology, and I think wider, even even wider than what we would consider biology. And I think that we need to develop um principled theories or how to detect, how to communicate with, and how to rationally relate to these other minds that are out there. And I think it's a very important project.

This is maybe a fun, a fun question here. I, uh, I get the impression from reading some of your work that you're either a fan of science fiction or you've at least, you know, you you know enough about it because, um, I mean, is that true? Are you, what is it fair to say you're a fan of science fiction?

Yes, I think that's fair. I mean, I grew up reading all of the, you know, classic science fiction. I was I'm a huge fan. I mean, honestly, I think that that uh what people call science fiction just means you have a a faster Gantt chart than than everybody else, you know, right? That's really, you know, people say, well, this is sci-fi, but but but to me, anything that isn't forbidden by the laws of physics, and I'm not sure we know that much about what's actually forbidden, um, is is possible and probably likely, assuming that, you know, we we we don't kill ourselves off and so on. So I I don't see, you know, some of some of these things that science fiction has dealt with, I see as very, very practic kind of issues and and fodder for for philosophical and scientific discussion.

Yeah, and I mean the reason why I ask is I I've, us science fiction is is really writing where you you have the entire pallet of possibility, if you will, and so you can explore essentially any any topic you want to, as you said, you know, as long as it doesn't violate the laws of physics, it's at least soft sci-fi, maybe hard sci-fi even, right? And, um, for me, it's it's been extremely useful, and I get made fun of a lot on on the show for mentioning it, but like you, I find a lot of inspiration and really just thought process that happens in science fiction, like let's explore the idea, what if, you know, what if there was an intelligence that was the size of a a cloud, as large as a solar system, and it showed up and it didn't even really recognize that there was intelligent life on Earth, you know, until after it done tremendous damage, you know, what happened? I mean, so it's just the ability to explore these ideas, right? And I think there's a lot of thought-provoking um thinking that happens in science fiction.

Yeah, absolutely. And and it also reminds us to be humble about our choice of perspective, right? So so what science fiction to us today, you know, every just about everything that you and I did today uh in our daily course of life would have been utter science fiction not that long ago, and it would have seemed completely, you know, ridiculous except to a few, you know, there there are a few Visionaries that that see some of this stuff early, but but uh so it's a moving Target, you know, these goalposts are constantly moving, and and I think that critical is to remember that our current perspective is not privileged in any way, that that things that seem sci-fi to us now may well be commonplace, and and we need to think about what that means for us.

Professor Lan, one of the reads I got from reading your paper was almost um, I guess you were railing against human chauvinism, and there were elements of what I don't know whether you would um be comfortable with me saying this, but transhumanism, and this idea of a space of possible Minds. I mean, um, Aaron Slowman, of course, wrote about that in 1984, if I remember correctly, and we spoke with Marie Shanahan about that recently, and it's fascinating. But we as humans, we tend to anthropomorphize; we think that our form of humanity is the only valid one, and we're constantly making distinctions between us and machines and um talking about things like agency and so on from our own lens. And you you spoke to this beautifully, but one theme I want to talk about is losing our Humanity. So you said, personally, if I and Humanity are supplanted by a population of highly intelligent, motivated, creative agents with a huge radius of compassion and meaningful lives who transcend our current limitations in every way, what better outcome could I hope for? What's your take on that?

Yeah, well, I want to talk about two things there. Um, first, um, you know, each one of us, and also us as the current implementation of humanity, are going to be replaced. That's there there's no there's no other alternative, right? Each each one of we're all going to die; we're going to be replaced by something. So persistence indefinitely, I mean there are some anti-aging things coming down and so on, but but but overall, persistence in current form is impossible. It's impossible in biology; every species will either die out or it will change and adapt, in which case it is again not the same species. All of us start out life as a single cell; we become an embryo, a child; we learn, we grow, we change. You know, persistence in the current form is impossible. So so the next question is, once you've given up the idea that you can stay exactly as you are, the next question is: what would you like to be replaced by? And I and I do think that this idea that we should remain uh as we are is I I I I don't know; I I I I can't wrap my mind around it because if if I think of humanity, you know, maturing as a species and we come back, let's say some some hundred years later, are we still walking around with lower back pain and a stigmatism and and and with our, you know, with whatever kind of body you happen to have been born into because some cosmic ray hit a, you know, an embryonic cell and all these limitations? It it seems, it seems impossible that this is the future that people want to paint. These limitations that that we face today uh were not given to us by any kind of rational process that had our values or our welfare in mind. This is just where Evolution happens to have dropped us off at the point where we can now participate in that creative process, and I think it's inevitable, and I think it's I I think this is what we should want. We should want the future to be better than the past, uh, and we should we should we should be thinking about what is it that we are inevitably going to be replaced by, and what are our values in terms of what kinds of things do we want to populate the universe. So that's that's the first thing. The second thing I would say is uh about humanity and losing our Humanity. So so the first thing to think about is what do we actually mean by Humanity? What is what is the important feature about being a human? Um, I I've heard there's people are working on proof of humanity certificates in the face of uh AI generated content and things like this, so that raises an interesting question. When you say someone is a human, what what what's actually important about that? And one very kind of simple way to visualize it is you're going on a journey to Mars; you're going to be there for 30 years. Who who do you want to have with you, right? What is so so you'd like a human companion; what does that actually mean? So um let's run down some options. Well, do we mean the DNA? Do we mean that you have to have modern homosapiens DNA? I don't know; I don't I I don't care about DNA that much. Um, does it does it what do you mean that they have to have the kind of standard human organs and and a human anatomy? And they better not be replaced, you know, have pieces replaced by some sort of cyborg technology? I don't know; I don't really care about that either. Um, I think as we as we think carefully about what is it that we do want to have a meaningful human human level relationship as opposed to a RBA or a, you know, a pet snake or something, then then the question is: what do you actually want? So I think we want two things overall: we want a similar uh or or at least so so when we say human, what I think we mean is at least a human level uh uh radius of compassion; that means you can actually in, you know, sort of in the linear range, you can generate the uh and respond to a certain um level of compassion for others. I think that is a a what what we really mean by humans. Um, and then the other thing I think we mean is uh being able to share uh existential the same existential concerns. So so we, you know, you want you want to have a relationship with another being that has roughly a kind of impedance match with with with respect to their cognitive lone. So the the the size of goals that they can pursue right should be should be roughly similar to yours, and they should be able to um understand your existential concerns. So they they like us should be made of Parts; they should be um uh you know what what the what you can think of as a kind of mortal computation, meaning that uh you know they're they're um perpetually at risk of Disappearing basically, right? So so none of us are are invulnerable and so on. So so these kinds of, you know, be open to these questions: what where did I come from? What does life mean to me? This is what we mean: the the ability to raise those questions and to have those concerns and to have um the degree of compassion of a standard human or or up. That is that is what I think we mean.

Just a quick point on that. So this um sounds like you're making an argument for canalization, or it's very similar to Bostrom's instrumental convergence, which is basically that for um an intelligent living entity to exist in our physical world, they would probably exhibit things like compassion and existential concerns, and of course they make the argument they would probably um exhibit power-seeking. But why why do you draw that inference?

Well, I'm not actually suggesting that they have to. I I I do think that it's possible to create beings that are uh highly intelligent in the sense of problem-solving capacity, um, but that don't necessarily share these things with us. I I think that's entirely possible. Um, uh, I think probably uh that that is what we will have first. I mean, that's certainly where I see modern efforts at AI going, which is which is that they're not really um bioinspired in the in the relevant way. Um, they are uh, you know, very good at solving certain kind of problems, but they actually don't have some of the some of the key features that are at Le at least as far as we know. I think we have to be very careful with with any kind of blanket statement here. Um, but so so I don't think those things necessarily go together. My my claim is simply that the latter is what we should care about. In other words, if you want to maintain Humanity, I don't think you should be maintaining physical embodiment. I don't think the anatomical description of a modern human matters for anything. I think what you should be really thinking about is uh is uh you know, wisdom and compassion basically, right? That's that's what you're looking for.

So you've chosen a couple of the properties of what I think if you went around just interviewed humans, you know, what does humanity mean? They they would have a list, right? And so you've chosen a few, you know, compassion and and wisdom. At the same time, those are in some ways just like our biology; they're kind of accidental, right? As to where we are currently in our in our evolutionary state and how we got here. Why should we care about those two in particular? Like, you know, maybe maybe the best form of life or future future intelligence has no compassion. Like, why should we care about these two in particular that you've selected?

Yeah, I mean, I think I think we are getting here to very fundamental values that I I, you know, it's like they're like axioms. I I don't I don't I can't give you an argument that those that that the two features that I've chosen are better than anybody else's features, but we have to choose something, and I don't think there are any arguments to be made for any of these that are sort of knocked-down um logical proofs that okay, this is this is the way that you should go. I think this is where uh we have to uh pick pick some pick some axiom and see what the implications are. I mean, so this is and and you know, and again to be clear, this is my personal view; this isn't I'm not saying that I can prove this or that everybody else should think of it this way or anything else. Um, it's entirely possible that somebody wants to emphasize something else, but I think mine are pretty, you know, pretty reasonable.

Sure. I mean, well, one thing about that which I find kind of interesting is that um a lot of things that we perceive as limitations, if you will, of of human cognition, human biology, whatever, if you kind of look more deeply and you look at the, you know, the algorithm, if you will, of evolution and the fact that what it cares about is essentially the long-term survival of Life overall, not even a particular species, but the entire life ecosystem, if you will, then they start to actually um be less limitations and/or things that that actually have value. So you bring up this concept of bow tie, right, which is like the fact that I have to take my thoughts, compress it into a language, communicate them to you, then you sort of decompress them into, you know, thinking in your head. Uh, it seems like, you know, maybe a disadvantage, the fact that we take our genetics, compress it into an egg and a sperm, and then they combine and reproduce an individual; they're all these bottlenecks everywhere, these kind of bow ties, right? Very similar to AI, you know, coder-decoder architectures, right? And most people look at those like, this sucks, you know, if only if only I could just have a neural link to transfer like data directly to you, like if I could just clone myself instead of going through this whole sexual, you know, reproduction kind of thing. Like they see limitations where actually a lot of the points that you make in your work are, no, these actually have like tremendous value when you're a life form in an environment of insufficient knowledge and limited resources; you have to survive under variation, variability. Can you kind of talk about that in two perspectives: like, one, did I get that correct? Um, and then secondly, why do you think it is that people always perceive these things as limitations? Like their frame is about their own personal limitations rather than seeing the bigger picture?

Yeah, yeah, well, uh, so so first, you know, to talk about Evolution and the goals of evolution and so on, you guys should have Richard Watson on; he's a computer scientist from the University of Southampton, and he has a he has a very interesting kind of new take on Evolution. So so that's, you know, I'll let him I'll let him handle that one. But um, I think that uh I I think you're right; I think people, you know, I think in general, and and uh this is totally like armchair psychology, right? I I don't really know, but uh but I suspect that people tend to project a lot, and in um a lot of what I see as resistance to uh some of the things that I and others have written about uh, you know, the the Continuum between humanity and other kinds of um forms of mind and so on, I I I think what a lot of people um see there is a projection of um their own limitations, you know, when people talk about what is impossible for for for for for machines, and of course they they say machines as though this was some kind of crisp category, when we know we have to deal with cyborgs and hybs and chimeras of of of all kinds, uh, you know, I I think that that there's a real um undercurrent of limitation in the in the in the sense that compassion and and those kinds of things are a zero-sum game, you know, that that if we that if we recognize too many other Minds that we then have to care about, there there won't be enough there won't be enough for us, and and we may not be able to muster um sufficient sufficient care. And so I have a feeling that a lot of this kind of resistance really does come from a deep um, you know, and as I pointed out in this in this paper and another one that's in review now, these are these are fundamental uh very, very ancient um human issues, you know, being supplanted by the Next Generation. This is not a new AI thing; this is as as old as the hills where where you know you are making high-level intelligences that are going to probably be better than you; you have no idea. I mean, you do your best or not, but but you have no idea that they're going to Value any of the things that you value. Um, you know, in fact, through their teenage years, they will tell you that they don't, and and so, you know, and so who are you going to get replaced by? This is this is a very old uh problem; this is not new with AI. So so I think all of these things are bubbling up to the surface now, um, where but but they've been with us all along, and um the thing about the bo that's uh that's like that's like Socrates uh the kids of today, right? Or the the Youth of today.

Yeah, exactly. Um, yeah, and and the thing with a bow tie. So so let me talk about that for a second. So um I I I I do think that that this this basic architecture is incredibly important because what I think it does in biology is set up the intelligence ratchet that we see in the living world, and and roughly speaking, it works like this: um, as a uh as a being that is limited in terms of a finite being under under limited resources and limited time, you don't have you you don't have the luxury of being a Laplacian demon; you can't track microstates; you have to compress your experience; you have to learn and and squeeze uh lots of um particular uh events into some kind of a a compressed representation; you're throwing away um all kinds of uh um correlations and and and you're hoping to, you know, capture a pattern, ignore all kinds of things and and capture a pattern, and and that's your that that's how you form memories. Well, um, and this is this is true behaviorally, but it is also true uh on a developmental, evolutionary um time scale, as you as you pointed out, you know, squeezing down to a genetic representation and then and then um re-expanding. So the thing about that is that um there's an algorithmic process on the left side of that of that that bow tie, which takes the what what the experiences you've had and squeezes them down into a representation, but the other side is is not purely algorithmic; it's creative, and it has to be because by throwing away all that stuff, you now don't really know what the past actually was; you just have a representation that you then have to interpret. The the necessity to interpret that in real time is what gives rise to not only behavioral intelligence, meaning Flex, you know, flexible adaptive Behavior, because um your your your allegiance is to the salience of the information, not the Fidelity; you actually have no idea what the original memory was; all you have is your memory trace, and in real time you have to you have to continuously do this kind of like storytelling about yourself in the outside world based on whatever angram you happen to have at the time, right? Because we don't have access to the past, and the same thing. So so that's cognitively I think what's happening, and developmentally what's happening is that um biology, unlike our computer technology, biology is dealing with a very unreliable medium. So we work very hard so that every layer is isolated, right? From the layers above, and when you're writing in high-level computer computer code, you you're not worried that that the metal is going to act weird and that your, you know, suddenly your your, you know, registers the numbers are going to float off and be something else; you you can trust, right? You have full trust in in the layers underneath, you know. Um, biology has the exact opposite approach because everything is unreliable; you have no idea exactly how many copies of anything you're going to have; the molecules come and go; things degrade; things misfold; it's it's and and you know, on an evolutionary time scale, you know, things are going to change guaranteed; not only the environment's going to change, but your own parts are going to change because you're going to mutate; everything is different, and each time, and for this reason, I think that um Evolution does not overtrain on the past, and this is why, and I've we we could talk about it, but I have in my papers I show many many examples of this; biology is incredibly good at solving novel problems and making some kind of coherent being in all kinds of crazy configurations that either Nature has done or or that we do in the lab, and the reason is is because even the normal embryo was not a mechanical construction to begin with; it is solving that problem every single time, you know, by I like this idea of beginner's mind; it's like you're you're given some tools, you're given some Hardware that's encoded by the genetics; you have no idea what it used to mean; you have no no no idea what kind of um environment uh, you know, and how it was meant to be interpreted, and you have no allegiance to that, and you have no ability to know exactly what that is; all you know is you're going to use that information as best you can right now to do whatever the next thing is, and that creativity, that that uh the requirement to to confabulate, really, to to tell a new story without any allegiance to what

The story used to be, um, is is is the is the intelligence ratchet that starts off in physiological and and metabolic space and proceeds through anatomical morphis space and eventually behavioral space and linguistic space and so on. Um, I I I think that that's that aspect, but having to compress and then creatively figure out what do my memories mean, I think is is is foundational to being a real agent, and I think that's what biology does. Yeah.

Reading through your work, and of course we can debate what the definition of intelligence is, but you framed it as some kind of General problem-solving ability. We've been debating on the channel for years, so we'll we'll skip that one just for the time being. But um, you know, when you spoke about the um the metamorphosis from a caterpillar to to a butterfly and and also in in your work in xenobots and anthr robots, you seem to be making the argument that evolution is bestowing the capacity for intelligence rather than solving specific problems or having a specific morphology. And you demonstrated in your work that you could repurpose low-level um, you know, cells if you like into completely different spaces and they could exhibit intelligent behavior. And to me, that suggests that you're making an argument which is kind of leaning towards primordial agency at the lowest level. I mean, we've spoken to panpsychists like Philip Goff. You're not making an ontological argument; you're just saying that the locus or the lens of agency and intelligence should be at the very lowest level, and it's, you know, it has incredible plasticity in generalization.

Yeah. I mean, I I I would like to make two arguments. One one is that I I am not sure there's any truth of the matter in the sense that um some kind of uh objective uh unique view from nowhere that where you can say this is this is what what has mind and this is what doesn't. I think that um all of this is Observer relative. Now, for for for complex agents, that Observer is the system itself, so there are this the strange Loop that kind of reifies itself at certain organisms, but basically all of this is from the perspective of an observer. And so um what you can say is from from from my perspective, I look at very basil systems, and we've now done this not only in cells and tissues, but we've done this in molecular uh Pathways. We've done it in very simple computer algorithms like sorting algorithms. We can talk about that, but but the idea is when you you look at a system like this, what set of tools are appropriate? Are they tools of mechanical rewiring? Are they the tools of cybernetics? Are they the tools of behavior signs of psychoanalysis and and and so on, right? And um I I you know, I'm I'm a panpsychist in the sense that I think that the utility of these kinds of approaches uh becomes clear very low down. In other words, you don't need anything remotely like a cell or anything that complex before you have to start thinking about from the perspective the system itself, what does it learn? What does it measure? What does it remember? What are the valences? I I think all of that stuff starts very low down, but I don't think this is a philosophical position; I think this is an empirical claim. So in other words, uh someone might say uh I don't believe any of that; I I'm looking at the system and I'm going to use good old um physical rewiring. Somebody else says, no, I think this this system has memory and preferences and goals, and they use a certain other set of of tools, and then we all get to find out who you know who had more um better research programs emerging from that, you know um so so I'm not talking about the hard problem of Consciousness here or or anything like that; that's that's a little bit of a different kettle of fish. But but in terms but in terms of uh uh you know uh cognition and intelligence, which I'm not defining intelligence that way, but for the purpose of our work, that is the the slice of intelligence that I'm interested in, which is problem-solving. I do think there are other aspects to it, but I I don't know of an experimental way to get at it, so I'm sticking with problem-solving.

Yeah, I think it might have been better if I used the term panagis. I mean, of course you're not an ontological pan, you know, you don't think that the the bits of the universe have phenomenal experience, but I think you do think that at the lowest level in the biosphere that there is the um either the facility or the capacity for agency, and if I understand you correctly, it it's this diffusion or or Collective agency which is the most important lens to understand systems. And another interesting thing about this is um I know we talked about transhumanism earlier, but I do feel that the lens of analysis determines the Viewpoint. So a lot of social scientists, for example, they look in the non-physical World; they they don't pay attention to individual human physical humans; they they look at culture and they look at language and they diminish the individual agency that we have, whereas yourself, for example, you're you're looking at the individual microorganisms in in the biosphere, and there are some intermediate views like Dawkins' view or even Yosh Shabak--we spoke with him a couple of weeks ago--he's talking about um cyber animism, which is the embodiment of of information in in into our physical being. So do do you think that lens of analysis changes the perspective?

Yeah. Yeah, I I I really think it does, and and I think that uh I yes I I think that as a as a matter of empirical Discovery we are finding that uh using agential lenses on uh on systems starts very early, you know, way be way below what most people. So so I I think um intelligence and cognition and agency are a bigger circle in terms of this Venn diagram kind of thing; it's a bigger Circle than than life than what we would call life today. When we estimate the the agency the intelligence of any system, we're basically taking an IQ test ourselves. What we're saying is this is what I am smart enough to notice about this system, and I think we're just at the beginning of getting at all good at noticing that. We we're just beginning to develop tools to to recognize these these other Minds.

Yes. So I mean, I think that you know a physicist might kind of challenge that, right, and say I I have no need for the Assumption of agency to explain like Celestial mechanics or like anything else actually, like you know nowhere in the sort of standard model of physics will you find anything that has to do with what a human being would normally associate with agency. Now you can like reduce your concept of agency all the way down to no if like a photon and electron interact and they wanted to do that or they had some, but I mean physicists don't do that. So can you give an example where in physics like we benefit from taking this perspective of agency to better explain anything that cosmologists, physicists, solid-state guys, any bread-and-butter physicist would care about?

A couple things, and I'll give you a very specific example. So one thing I think that um there there needs to be again an impedance match between the tools you're using and what you hope to see. So so if you insist on using low-agency tools, so rulers, voltmeters, that kind of stuff, you're guaranteed to see that kind of thing and and and the field of folks who look at that stuff, we call those physicists, and they see certain certain things. Now um we have to remember so so my claim is not that human-level cognition is uh is is usefully painted onto electrons; that's not what I'm saying. What I'm saying is it's a Continuum, and what we can do is we can ask a very a simple question um what would the most basic version of agency have to look like? Not the human-level version, but like stepping it all the way down because you know we were unfertilized eggs once, and so clearly somehow that gets scaled up during development and during Evolution. There's a there's a Continuum. So so what does the basement version of it look like? What do you have to have in order to uh to be on the Spectrum at all? Um and and that's and that's important because if we look at if we look for human-sized uh versions of this at at other levels, of course we're not going to find it, but we have to reduce our um expectations concomitant to going down on that on that continue. So I actually think if you do that, you find something that physicists absolutely deal with, which is least action principles. So the very basic version of having goals and having some problem-solving capacity to meet those goals is the kind of thing you see, for example, when you find out that light chooses paths that are the least action. It doesn't look, of course, it doesn't look like human agency. The question is, is there a story of scaling by which you can go from something that's simple like that to uh Concepts that we need in neuroscience and and so so one example of that scaling is free energy minimization, and there are you know that that's a that's a um a story in which, for example, Carl Friston and Chris Peels and and colleagues have used the same conceptual framework to talk about what particles do and eventually talk about what what humans and in fact ecosystems do. So so what we're looking for here is is stories of of of scaling that allow us to say, okay, this is the very basic, the most primitive version of what um through various Dynamics with life and and maybe other versions will become eventually the things that we recognize as as human-level agency.

I'll give you a very very specific example. Um, consider uh Gene regulatory networks. So what you have there is never mind the nucleus at you know all that Machinery, just just the very simple thing you got let's say um half a dozen nodes; these are chemicals that turn other nodes on and off; that's it. You don't have anything else; you don't have a cell; you don't have any and you know all all all all it is is is is half a dozen or or more um chemicals that turn other chemicals in that set on and off. Now when you look at that, and so let's say we're going to model it, and then this is this is what we've done; people model them with ordinary differential equations, and you can see exactly what's going to happen um when when when these things interact. Now now one way to look at this is to say okay, it's completely deterministic; it's fully transparent; there isn't any any magic here. So uh this is going to be uh a dumb mechanical system, and I've got uh you know a dynamical systems theory um by which I'm going to just make predictions about what this thing is going to do. Okay, so that's that's one set of tools where where where you say um I I don't see anything here that would warrant any kind of uh you know any kind of agency talk or anything like that, and that's fine, and if you do that you get certain kinds of capabilities, but you're also uh precluded from other kinds of capabilities. For example, um these Gene regulatory networks are very important in health and disease, and if you wanted to manipulate their behavior, that Viewpoint makes a specific uh prediction. What it means is that if you're going to change the behavior of this thing, what you have to do is rewire it; you have to introduce new nodes or cut the cut the connections between certain nodes and other nodes, and and in biomedicine that's gene therapy; the way you would do that is is is through is through gene therapy. So so that that philosophical Outlook has very specific implications for your roadmap for regenerative medicine, for example.

Okay, and so we looked at that and we said, well, yeah um it's transp you know it's transparent and it's and it's relatively simple and there isn't any magic, but of course that's also true if we look inside of human brains; there's no magic there that we're ever going to find either. So let's so let's do this; let's not assume we know what this thing is and let's just try things on that spectrum of agency to see what other tools are appropriate. So one other tool that's that that you might try is something that's used for brainy systems, and that's training. So so taking advantage of a learning interface, how how might you train a gene regulatory Network? Well, what you might do is you might choose uh some of the nodes as so so let's look at, for example, in the case of Pavlovian conditioning, just as an example, we actually found six different kind just to kind of spoil the whole story; we found six different kinds of learning in these in these Gene regulatory Networks. So let's let's take Pavlovian conditioning. So so you know um you've got a dog; you you you normally uh you you you ring a bell; the dog responds not at all; you show him some meat; he salivates; then eventually you pair the the Bell and the meat, and then eventually he'll salivate at the sound of the of the bell, for example. So okay uh so so that so so we have three three things here: we have the the unconditioned stimulus, that being the meat; we have the response, that being the salvation; and you have something that's initially a neutral stimulus, the Bell, that eventually becomes the condition stimulus after you've you know you've exposed the thing. So so you take your Gene regulatory Network and you pick some nodes and you say, okay, this node is going to be my response; I'm just going to watch what this node does; this node over here I'm going to find a node that's an unconditioned stimulus, meaning when I when I trigger this chemical the response gets gets upregulated, and then I'm going to find some other node, and I'm going to try I'm going to treat that as a neutral stimulus, meaning when you tickle that node nothing happens to the response. Now I'm going to do paired presentations, and I'm going to see what happens. So if you do so this is a very standard kind of behaviorist approach; it black boxes; you know, we don't have to know is it an animal? Is it a gene regulatory network? Is it a you know you don't have to know what it is; you can just kind of do it and see what happens. If you do that, what you find out is that biological networks, but not so much random networks, so that's interesting. I think Evolution actually likes this feature. So Random networks don't do this, but biological networks um have six different kinds of memory; they can do habituation; they can do um a uh sensitization; they can do associative learning; uh they can do some other things; things they can count to low numbers and things like that um none none of that is obvious or visible by by taking the the sort of chemistry and dynamical systems theory approach, and you wouldn't find it unless you tried it. Why? Because we are not good at guessing these things; most people's intuitions when they look at this thing they would not say that this thing is smart. To and and I'm not saying it's you know I'm not saying it's human level; I'm not saying it's sitting there having hopes and dreams; what I'm saying is it is on the Continuum; it is on the Continuum of agency because there are tools that you would apply to learning systems that get you to new capabilities. And so what are those new capabilities? Well, now that you know you can train the thing, you have all sorts of possibilities in terms of drug habituation and and all kinds of biomedical you know Drug Association um all kinds of cool biomedical applications which we are now pursuing in the lab that don't require you to use gene therapy because because now you understand that you have a system that has priors; it stores those priors, and that future behavior is a is a uh is an is a trainable um outcome of past experiences. So my point is is is is simply this: we have to be humble about the fact that we don't know how to just you know notice these things, and these are not you can't have philosophical feelings about it; you have to do experiments and see what that does for you.

Okay, so it it's almost like um a more sort of holistic approach to to let's say the the gene regulatory Network, a more holistic approach to trying to solve solve the issue. And the reason why I say that is because no matter how many ODEs or PDEs I wrote down with all the you know Gene regulation, it's still only because of how insanely complex you know biology is; it's probably still only a projection of what's actually happening. And so, for example, you may find that yeah, every time we introduce too much sugar and we brush your hair like it seems there should be no connection there, but somehow brushing your hair we've learned can correlate with you know um proper insulin you know functioning or something; it's entirely possible, right, just because of the insane complexity of the feedback of of your biology, you know, tactile fluences, things like that. So it sounds like it could lead to more holistic or at least um uh exploratory rather than very analytic and reduction kinds of approaches to to treatments.

Well, it's definitely not reductionist, but I it's my claim is not that things are so complicated that we should try random stuff because we'll be surprised. I mean, that's true too; no no doubt that's true too, but that isn't my my claim. I mean, you the this this learning capacity shows up in a Network that has the minimal Network that we found has four nodes, and so you really don't need to be that complex for some of some of these things to happen. But but also to be clear, the things that we found were not random things that we were you know poking and and we just happen to notice it; we took very specific protocols from Behavioral Science, and this is what we've been doing for 30 years is is stealing stuff from neuroscience and applying it other places because I think that this is not just the random vagaries of complex systems; I think these are um common threads that run through systems which don't even have to be that complex. We found things.

Okay, interesting. So it's kind of a it's it's an interesting middle ground; it's sort of like on the one hand you kind of like have some holistic kind of a woo approach where we don't even have we just try different things and we found that crystals correlate with something; other on the other hand you had the break it down, the very extremely analytical, find the gene you know to cut. This is an interesting middle ground, closer actually to that traditional you know sort of scientific approach, but but Guided by a of principles that came from like let's say a higher higher level of of emergence. So I don't know; I mean, it seems pretty interesting.

Yeah. What one one way to think about it is look um humans have been training dogs and horses for thousands of years knowing zero Neuroscience, right? Exactly. We knew absolutely nothing what was between their ears, and and the reason that we we were successful at it was because living systems control; they don't control themselves by micromanagement; they encapsulate, right? Trainability and and behavior shaping all all throughout; we're just hijacking that that normal interface. So that that learning interface that the system is using encapsulates all that crazy complexity. So I I actually think there's this kind of weird uncanny valley with complexity where um for very simple systems we can kind of micromanage them and and and we can tell what's going on and we can control them; then things get really complex and we don't have a good way of handling it, but when things get not just complex enough but but the kind of thing that life is doing um which is this multiscale competency art architecture, then even though the complexity has gone through the roof, the controllability actually gets better again because now they're offering this top-down goal-directed um behavior, and you can those are now your control knobs, not the materials, not the physical properties, not what each part is doing; you're not trying to micromanage symptoms, which is mostly what our biomedicine does; you are trying to convince--literally I'm using these words quite literally--you are trying to convince the system to uh be on the same page with you about what the set points of that homeostatic process should be; you're trying to get Buy in so that it's not going to resist whatever um medical intervention you've just done. This is this is I I think another way um to think about this is the way you engineer with agential materials. So engineering with passive matter, even with active matter or computational matter, is different than engineering with an agential material; it's you use a completely different set of tools because you're working with a material that has competencies and and um agendas of its own, and you just have to engineer differently in those cases.

Yeah, I'm I'm a big fan of of much of this. I mean, you know, with Mortal computation and biometic intelligence, you know, we're moving away from a paradigm of of abstractions and instructions, and we're moving more towards training um as you said. But you were saying some interesting things about um agential material, and your definition of agency in in the paper was basically a kind of cybernetic uh loop, and uh philosophers talk about it in terms of like self-causation and intentionality. Of course, they're not talking about micro Pro causation; they're talking about it as a lens. Many of the free energy people--I I've interviewed quite a few of them at this point--they talk about agency as an instrumental fiction or um you know basically an as-if property or or or a model. And and even the free energy principle itself, it doesn't talk about the microphysical processes which lead to the emergence of things and agents; it's just a property that those things have after they coalesce. We've got this abstraction called agency, and and then we use this abstraction to design AI systems and and to and to reason about new forms of intelligence and and so on. Is it is it a lossy abstraction?

Well, yeah, for okay, first of all, my sort of metaphysical position is that everything is an abstraction and everything is a lossy abstraction. In other words, you know, people sometimes people will say, well, that's just a metaphor, you know, when you're talking about intelligence and memory and in tissue and whatever; that's just a metaphor. Well, yeah, and and what are Pathways and what are genes and what are I mean all all of it I think all all we have is metaphor; that's I I think that and and the only question remains is is what does your metaphor let you do? So I don't believe that metaphors are exclusive; there are multiple uh different metaphors that you can have about the exact same thing. This is the kind of the nature of the polycomp Computing Paradigm um that Josh Bongard and I have been working on uh so I think it's all metaphor, and but but what we want are enabling metaphors; we want metaphors that help us reuse Concepts across fields, to develop new Concepts, to um Drive New research programs. So is is it is it a metaphor? Yep. Is it a construction? Yep. I don't know what else we have besides those things. And so now the only thing that we have to work on is uh to to to improve the ones we have now. Um you know, in terms of uh are we going to miss things? Most most certainly, and and so that's you know that's as as scientists and philosophers; that's I think what we're in the business of is is trying to open up our metaphor so that we see more interesting stuff.

Yeah, so just one point on that then because you said that you don't support um a deflationary reductionist account of biological agency, you know, which is to say we're just chemical machines, because you said that framing is utterly insufficient for progress in science and engineering, but I did get the read though earlier on that that you were making that argument, and one of the reasons for that is I think when you were talking about embodiment, for example, you said yeah, embodiment is crucial, and and and by the way, the reason why a lot of um cognitive scientists love embodiment is from this externalist tradition, you know, that they think that the intelligence is actually outside in the physical world, and when we interact with the physical world we can kind of um you know basically Curry a lot of that intelligence in in some um indirect way, but you said um you know this goes beyond robotics uh which

Gives an AI an actual body to cohabit an environment alongside us. And beyond the coming revolution in virtual reality, where many of us will spend a considerable amount of time in interesting worlds, so so so—which is it? Presumably you do think that we can have very rich intelligences completely in silico, away from the biological world.

That's yeah, um, and that's because I see embodiment slightly differently. So so, uh, typically when people say embodiment, they mean the 3D physical world, and you're moving around in the 3D physical world, and that's what they, you know, they say that organoids—let's say brain organoids in a dish—are not embodied, and you know, certain kinds of AI is not embodied, and so on.

Um, what I think is important about embodiment is the necessity of doing this perception-prediction-action loop. And that loop can happen in lots of different spaces that are not obvious to us at all. So biology—and this is not just for robotics and AI, for example—biology, so so you could have, um, and in fact we do have cells and tissues that may be going nowhere in the physical world, and nothing much is happening as far as we can tell, but what they're doing is they're solving problems in physiological space, in gene transcription space. Once you get a multicellular kind of thing going, you can solve problems in anatomical morphospace. All of this—taking measurements, judging them against your preferences, your valence, and your set points, and then figuring out what do I do next to improve my situation—all of this happens in all kinds of spaces, only one or two of which we know how to visualize at this point. And so, um, I think we have to be very careful about what we call not embodied, and also I think the choice of spaces is very much up to the observer, you know, um, as a simple example, imagine you're—you have a bacterium, and the bacterium is trying to go up a sugar gradient, right? So what we see is this loop that says take a measurement, and depending on what you see and what your memory is of the previous values, you can either go up the gradient or tumble and go somewhere else. You know, that's what we see, but actually one other thing the bacteria might do is turn on a different gene that lets it metabolize a different sugar that's at a much higher local concentration than what we were looking at. So for the bacterium, taking a step in transcriptional space is—does it represent that differently than taking a space in in in physical space? I'm not sure, right? For us as humans, those are completely different things. And when I talk about problem-solving and movement in transcriptional spaces, a lot of people like, well, that's not real, you know, that's not real space, um, but I don't know that the bacterium sees it that way. This is a—this is a, you know, a formalization that we've made, and so I think we have to be very careful. And I think that kind of embodiment is critical, but it's not the physical space or the physical body that's critical; it's the loop; it's this having to be an agent to say, what do I do next? And having a set of options and a set of pressures and constraints and and enough of a—of a—of a cognitive system, which can be very minimal, that lets you do things where the future is going to be different from the past. That's what I mean by embodiment.

I guess maybe I want to throw in one little tiny perhaps caveat in there. And when Steven Wolfram impressed me on giving a definition for an agent, I said almost the exact same thing, except I added one additional thing, which is the actions I take need to affect the environment I'm perceiving. Yeah, wise, like I don't actually have this loop: I'm perceiving something, and then I'm acting in some other space that has no influence on the space I'm perceiving. Like in your definition, does that also have to be true, or can you be embodied by perceiving one space and acting in a completely different one, and there's never any loop?

Yeah, no, no, I like the emphasis on that, um, you know, it's a kind of niche construction, right? It's—it's—it's the, um, you know, you can think about niche construction and stigmergy, you know, leaving—use—writing to the outside world as memory and so on. I like it; I think it's a continuum. So I think some spaces are much more easily bent than other spaces. And so, you know, in transcriptional space, yes, you've got the possible—all these possible dimensions of gene expression—you can bend that space by changing your own transcriptional machinery, your own translational machinery, so you can act on the space, and then what you do next is now perceived—is going to perceive somewhat of a different space. So I completely agree with you, but I—but I think there are degrees, right? Some spaces are more easily deformed than others, but yeah, I think that's an important part of agency.

Yeah, do you think there are any useful physical bounds to intelligence? I mean, I know we've just been saying that the future eludes us, and we always have people who are—they just—they can't conceive of the future, and they think what we have now is is very good. But one of the things you've said, of course, is that potentially what supersedes us could be a being with, um, you know, much much higher intelligence. What are your thoughts on that? How—how could it be potentially limited by the physical world?

I think it would be extraordinarily foolish of me to try to put limitations on—on—on intelligences that, by definition, are smarter than than I am. So I—I don't—I don't know, you know, I—I—I would not—I think we need to be very careful with limitations, period. I mean, again, just to go back, I don't think that intelligence is all there is, and there may be some kind of a filter whereby, uh, if intelligence proceeds too far ahead of things like compassion, and you know, I'm not even sure we have the vocabulary for, but but you know, some some kind of, um, you know, life-positive features, it may spontaneously self-destruct. So there may be this kind of feature of the world where just by itself, raw intelligence does hit a ceiling because because then there's some kind of destructive, uh, you know, outcomes that that prevent it from going further. But—but—but aside from that, if you—if you, you know, if you manage that, if—if the intelligence manages that aspect of—to keep up—to keep up the—keep up the growth in wisdom with the growth of intelligence, I wouldn't dare put a limit on it. I don't even know how you could put a limit on it. It's like—it's like giving an IQ test to somebody way smarter than you. I just don't know how we would do it.

Yeah, I mean, the evolution of our biosphere certainly seems like, um, almost a flash compared to the history of the universe, and people do get worried about this rapid expansion. So, of course, the AI RIS people, they say we could have an intelligence system which is 10,000 times smarter than us, whatever—whatever that means. And of course, externalists think that we're all part of a big collective intelligence anyway, and there's already some kind of homeostasis or—or some kind of limit. But you seem to be saying a little bit in your paper that we—think of ourselves as—as, um, a distinct entity, you know, we—we're—we're this—we're this human, you know, Humanity, we're humans, and—and—and I guess you're arguing that it's a category error to think of ourselves as distinct in the first place.

Um, I, you know, I don't love fixed categories at all, and so it might be an error from some perspectives; it's probably not an error from others. I—I—I—I want to emphasize, I guess, one aspect of it, which is that just because there—just because we're a part of a higher collective intelligence, which we may well be—I'm not sure—that doesn't mean that everything is great and our interests are taken care of. So, um, we actually are not very good at predicting or knowing about the goals that—that the composite systems will have, and that—that emergent, you know, goal-seeking systems—we're not—not very good at knowing what the goals will be. And if we just look at the—the, you know, one clear example where we see this happening is—is in multicellular bodies. So, you know, you could—you could obviously make the argument that a lot of new and interesting things happened when you became an emergent human and not a pile of individual cells. But, um, you know, when you go—I don't know—this is for some reason—this is the example I always—always go to—when you go rock climbing, you know, you're going to have a great day, and you're going to—you're going to, um, achieve some personal goals, you're going to achieve some social goals, you're going to have a great time, you're going to leave a whole bunch of your skin cells on that rock face, and they didn't get to ask, you know, they were not asked whether this was a good deal for them. Um, you know, people do all kinds of stuff to their bodies for various reasons. Just because you're a collective—something is—you know, is part of a collective intelligence does not necessarily mean that that collective intelligence is looking out for you at all. So maybe—maybe not. And so we need to, you know, I—I don't think we can just assume that because we are part of a greater system that there's some kind of benevolent homeostasis. Um, I think we have to be very careful with what kind of emergent systems we create. And this is the reason why this is so important. I—I think that developing this field of diverse intelligence is an existential-level task for humanity because we are constantly building large-scale emergent cognitive systems where we have no idea—we're just very bad at predicting what's going to happen, knowing or shaping their goals, um, communicating with them in any way. You know, we make social structures, financial structures, Internet of Things, robotics; we—we—we make these things in various spaces, and we still have no idea what—what we're really making because we assume that it's either going to be mechanical and controllable or random and chaotic, but we don't understand that many of these things are, in fact, goal-driven systems.

I'm glad you brought up sort of, you know, leaving the skin cells on the—on the rock face, because it's the same—same point I made earlier about people being so myopically focused on their own individual existence and opining about how all these imperfections that the human cognition, brain, body, etc., has, where they don't understand they're part of an—an evolutionary algorithm that has goals like that transcend their—their individual goals. So like a skin cell could sit there and say like, why is it that the, you know, I'm—I'm an epithelial cell, why are my fibrinogen bonds so weak? You know, how come I just so easy to get sloughed off? Why is my lifespan so short? Well, cuz—cuz you've been optimized for something that actually has nothing to do with you as an individual; it's you is a component of a larger system, right?

Yeah, and—and—and, you know, and—and there are—there are defections from that—from that system. So, for example, Tasmanian devils have a—have a weird kind of mouth cancer that propagates from animal to animal when they bite each other in—you know, when they—when they fight. And so—so cancer in general is—is—is a shifting of the—cognitive con—from—from—from, let's say, organ level down to individual cell level. So cells roll back, and they're sort of amoebas, and typically that goes very poorly for them because, you know, they—they—they have kind of a short-term vision where they're like, okay, I'm—I'm—I'm leaving this collective; I'm going out on my own. That—that doesn't typically work out very well. But for—I—I—I think the one in—if I'm—if I'm not mistaken, I think the—the—the lineage of mouth cancer in—uh, Tasmanian devils is like 11,000 years old or something like that. So—and—and there's—there's another example with—uh, clams. So clams—cancer cells and clams get released into the water; they hang out as single cells for a while, you know, no problem; eventually they'll find another clam, settle in, and you know, there they go. So typically that kind of defection from the body plane is a dead end, but not always. And—um, yeah, I mean, you know, biology has—has observers and goal-seeking agents at every level competing and cooperating with each other, and—just because you're part of a larger system doesn't mean that—that it's looking out for you.

On the—the thing you—you've done some talks about this where you've got a new way of thinking about oncology almost as a subdivision of—of the self. So it's—it's like a—a new part of yourself which has its own goals, but that lens, I think, actually could lead to new forms of treatment, right?

Yeah, yeah. I mean, I think one—one way to think about it is to ask, um, why—why do we ever have anything but cancer? In other words, we start—life started as individual—cells, when—then they started cooperating towards these grand construction projects, building, you know, livers and hearts and—and things like that. Why—why—why do they do that? So I think one thing that we've been working on is this notion of a cognitive loner, which is the size of the biggest goal that a system can pursue. And we've characterized a number of mechanisms that allow individual cells to bind together, and the network has a bigger cognitive loner than the pieces. So suddenly, when—when you're an individual cell, you care about your—your own metabolic state and—and—and things like that. When you're a part of a network, there's a little bit of a mind meld that happens with these other cells, which we characterize molecularly, and—and now you're working on three large-scale goals in the anatomical morphospace. Well, that—that process—that—that goal-scaling process—um, has failure modes, and we know how to—to induce it, and then, of course, in nature it happens from time to time. And then what happens is that cognitive loner—basically the cells disconnect from the network; that cognitive loner shrinks to now being the size of a single cell, and so now you're back to—it's—you know, it's not anything crazy or novel that's happened; you're back to an amoeba as far as concerned—the rest of the body is just external environment to you; everything you care about is around a very tiny little—little radius, both in terms of space and time. And that suggests—thinking about it that way—that—that basically these cells are not any more selfish than other cells, right? Like in game theory modeling of cancer, you would—you could say they're less cooperative; whatever. I don't think they're any more selfish than any other cell; I think their selves are smaller; that's all they're—they're still—they're as selfish as the collective; it's just their self is tiny now. And so that—that way of thinking about it leads to a very—um, specific—um, roadmap, which—which we've been following, which is let's just try to reinl their cognitive loner; let's—let's reconnect them to the electrical and physiological network of the other cells. We're not going to try to kill it; we're not necessarily going to try to look for a hardware defect some cells have—in some cells don't—you don't need any kind of genetic defect to—to—to have this kind of transcarcinogenic transformation—let's reconnect them. And—and so we've done that in animal model systems, and it works really well. And so now the question is, can we, you know, move it to human medicine? But this is—that—that's—that's another example where taking these different perspectives, which almost seem—almost seem like almost seem like philosophical perspectives, they lead directly to—uh—actionable policies in—in experimental work.

I guess it'd be good to get some of your comments on large language models and what they're good at and what they're bad at. I mean, I—I certainly think they are lacking in creativity and—and agency, but they are surprisingly good, right? They—they have somehow inferred all of this information; we have phenomenal sensations, and we understand each other with this, you know, physically embodied, informed, um, you know, setting, and then the language models act as if they have a similar kind of understanding, but—but they don't. But of course, they—they don't actually have agency. But is that a useful distinction?

I think it's a reasonable distinction, and I share your intuitions about it, but I want to say that I think we need to be very, very careful about, um, feeling certain about these things. And the reason is—is that we have—by—by exploring extremely minimal systems, like sorting algorithms, like gene regulatory networks, we have found massive surprises—not just emerging complexity; emerging complexity is easy; emergent unpredictability, it's—you know, it's cheap; it's—it's emergent cognition—basil—but emerging cognition. And so, you know, what—if—if I had to—if I had to bet money, do I think that today's language models have high agency? No, I don't. Am I certain of it? Absolutely not. Because—because my—the lesson I've learned is if I don't even know what—uh, bubble sort does, then I—I probably don't know what these things are really doing. We—we—we are very—it—I—I think it's very early in the field of diverse intelligence to be able to be certain about any of these things. Um, I, you know, ostensibly, I don't see—when I—when I look at the—at the structure of these—of—of—of today, you know, language models and so on, I—I don't see the explicit, um, kinds of processes that I—that I think are important for a—but I've also learned that just because we don't see them and we don't know how to look for them, that they're not there. So I think we should be—we should be very careful. I don't think there's anything like human minds; I think that's very unlikely. I think that, you know, for the first time, we've—so—so it used to be that if you see something that speaks, you—you're pretty sure that it's like you in—in a lot of ways. So I think we've—we've now—we've dissociated those two things, and we now have things that talk and put up a really good—you know, a really good performance in—in having a conversation. I—I don't think they're like us at all, but I think we need to be really careful about what we—you know, what—what we're sure of that they do and don't do, because our intuitions fail at—at such a low—at such a simple minimal level that extrap—you know, I've met all sorts of people who say, well, you know, language models don't have this and that; I know—I built them—like—I—I write the code; I know they don't have—and I'm like, well, you don't even know what bubble sort does. I don't think we can say what—what this thing does, so we just—I—I just think we have to be careful.

Yeah, it's weird you say they're not like us, but they—they kind of are like us, and then there's the—the, um, the realization that when it becomes indistinguishable, you know, does it really matter? But I think it's instructive, though, because in—in the—in the virtual world, there seems to be a couple of differences, right? So agency, of course, it's about, you know, having goals and intentionality, but it's also about causation and power and the ability to realize those goals, the ability to change the world to meet your goals. And what would it mean to have a collective agential AI running out there on—on the internet? You know, do you think it could—actually—does it make sense to call it agency in the same way we do in the physical biological world?

Again, I—I—I not—I don't have any evidence that today's architectures have what's necessary to do it, and—and—um—uh—so—so—but—but—but in—but in general, I think it absolutely makes sense because you could have a system that's doing the kinds of things that we're talking about in a—um—in—in a non-physical world. Now, having said that, I—I don't believe that our algorithmic—um—model—so—so having algorithms, having a Turing machine metaphor, I don't think any of that is sufficient for this. So this is not a computationalist argument, um, that says that—uh, you know, any—any kind of algorithm will automatically—can automatically do it. I—I don't think that's necessarily the case, but I neither think that you actually need the physical world for it. And by the way, these things—these things are embodied to—to an extent; we—we are their body. I—I'm not even sure that—that these kinds of things are truly disembodied in—in an important sense; we are their body; we move massive amounts of energy, you know, money, resources, whatever, based on the things these things do and don't say. And again, I—I don't have any evidence that—uh, today's language models have anything like a—a core goal-driven—agential component, but I think it's completely possible, and I think we need to be very—very humble about—feeling certain about what they—what they do or don't—don't have.

Just—a quick point on that: there's this interesting kind of agency sharing between the physical and virtual world, right? So you can think of a meme on Twitter as having agency because it's expressed by physical actors in—in the real world. So in a sense, there's a transfer, but maybe purely in the virtual world there could be no—no agency.

Well, look, you know, in an important sense, you and I are living in a virtual world; we—we don't have access to—and I'm not talking about a simulation, you know, with an alien zoo or something like that; I'm just—I'm just pointing out that, you know, you and I are conducting—our behavior under extremely limited—um, transduction from an outside reality that we don't really have any access to. Uh, you know, there's a—there's a—there's a big hole in our—you know, in our—in our visual field that we don't see; there's a million other things that are very different to how we perceive them; there are massive amounts of information that are inaccessible to us. Um, everything that you and I do is—is—is within—in a construction that we have pulled together ourselves as our—the best story of an outside world that we can tell. And, you know, yes, the physics of it are cool, and they give rise to all kinds of morphological computations, but there are properties like that in other virtual spaces, and I don't think we should be, you know, uh, too—too tied to this—this—this what we see here as some kind of privileged reality that is the only way that agency can be done.

Professor Lan, it's been an honor. Thank you so much for joining us today.

Thank you so much. Yeah, had a good time.

Thank you. Yeah. [Music]