Joscha Bach "Bootstrapping a GODLIKE Mind"

really have to think about what is the kind of artificial intelligence we want to build for our children. Suffering is not created by the universe. It's created inside of your own mind. The reason why our body gives compute credits to our mind is because our mind is supposed to solve the organism's problems. Uh, the question of whether machines can think is about as meaningless as the question of whether you can swim. Computers are already in the same ballpark as the perceptual apparatus of our brains.

First of all, thank you so much for being here. An absolute honor and joy. We're absolutely very, very excited. So, thank you. Um, I guess I wanted to start with this because I feel like you're the perfect person to help us get some clarity on this. One of the, there's a lot of questions online that kind of split people into tribes, whether it's diet, religion, politics, but one thing I'm realizing is there's a new one, and that is this idea of like, can machines think? Some people kind of violently oppose that. Some people are on board. Can you help us get into any insight into, can machines think?

Yeah, I think we need to start with the question, what is thinking?

Right. There is a quite famous citation, I hope I don't miss. I think it was by um, Dstrom, and um, it was, uh, the question of whether machines can sink is about as meaningless as the question of whether Ubots can swim. And some people interpret this as meaning that what the Ubots are doing is something that is too mechanical to be compared to the real swimming that organic fish are doing. And so, but an interesting thing about dubos is that they're not confined to the surface. And when we talk about swimming, we mostly think about, uh, basically breaking the surface of the water and sticking in a single plane. And the hubot can move in 3D. It can do much more than just swimming and go beyond that, right? It can sink in ways in which we cannot, and even fish cannot, because it can go to greater depths and it can go at greater speeds. And so, in some sense, you could say that it's a meaningful superset. And so the question also for computers is, in my perspective, not so much, can they think, but um, why should they only be thinking? Isn't, isn't there something more interesting that they could also be doing on top of thinking? But before we get to this point, of course, we have to ask ourselves, what is thinking to begin with? And it seems to be overwhelming for us to make sense of what thinking is. And because it appears to be so mysterious, people tend to be in awe of that notion. And they're very reluctant to attribute this mysterious quality to a mere artifact that humans have built and programmed and set into the state to interact with us. So I would start out with trying to look at what minds are doing. I think minds are making models of an external and internal reality of a being and interaction with its environment, and they do this via internal acts of communication. They have to develop some kind of messaging protocol. They can have to hold state and have to model how to go from state to state, and these states have to correspond to aspects of how the universe works in such a way that these aspects become controllable. And we observe ourselves to have two main modes. One is perception. Perception happens in real time, which means it happens synchronously with the things around us. And, uh, it tends to be geometric, which means there is stuff happening in a space, movements, and it's even true for our emotions. We typically experience them as continuous flows in a low-dimensional space as experiences of movement and pressure and resistance and so on. And we can all understand these things using some geometric calculus. And on the other hand, we have reasoning. And this ability to reason requires that we are translating our percepts into some form that makes it compositional, like Lego bricks. And necessarily, this is much more brittle, and these Lego bricks are bottoming out in references to perception. So we cannot define language just by itself as these sparse symbols. Instead, we are using pointer structures that we can put together in basically with slots and, um, little knocks into sentences, into grammatical structures. And these grammatical structures are just the outside form of a semantic structure, of a conceptual structure. But there is a tight correspondence between these abstract concepts and the linguistic symbols that we're using for reasoning. So imagine you're thinking about a face, and you have a nose in that face, and, uh, what you're doing is you take these linguistic symbols and then you make pointers from these linguistic symbols into a larger, somewhat continuous space of ideas in which this schema of a face exists, and the schema of a nose exists, and certain relationships that can be used to plug them together. And these, in turn, are pointing at some kind of superposition of faces and noses and so on, that you get to know about by interacting with them in some kind of external reality over which your mind makes models. And none of these things is very mysterious today. We have computer models of this that are able to emulate all these abilities of the human mind to some degree. And there is nothing that is super mysterious about it anymore. Even though the mathematics is not extremely intuitive, but the deeper question that people might still ask, is it possible for the machine to experience imagining a face and a nose and interacting with them? And this is the much more contentious question. And I would answer this question, uh, with this, experience of experiencing is itself a kind of a model. It's a model of what it would be like if some observer existed that has a certain perspective on those things and is right now looking at this stuff, and as a result, there is stuff happening in the observer, and these reflections of the observer inform additional behavior of the observer, inform the next actions that the observer does, and also the next set of reflections is like a loom that is creating a pattern that is also being read by the loom and then continues the pattern. And so, in this sense, we are not the loom of. We are the pattern, or we are part of this pattern. It's a pattern that is a pattern about looking at itself. And so consciousness, I think, our ability to experience ourselves and the world in some sense of real time is perceptual. It's on this level where it happens with this model of immediacy. But it's, uh, on the other hand, a reflexive representation. It's, it's nothing magical. Itself is a model of what it would be like if you existed. And this also means it's virtual. It's a simulation. Can only exist inside of some kind of computational machine that can create simulation.

Just to follow it up, to kind of bookend it from the other side, because often times what I hear people will say, it can't think, or it can't experience, or whatever the thing is that we're talking about, because it's just a stochastic parrot. It's just matrices being multiplied. It's just, you know, this is that. In and of itself meaningful? Are those arguments against it being able to do anything in and of themselves meaningful, or no?

I think that they only make sense superficially as, uh, as metaphors. And these metaphors are misleading because they're not grounded in a deep understanding of what understanding actually is. And this question of what understanding is is not trivial. And it's very easy to sit there, uh, with a crowd of like-minded people on stage and say, machines cannot understand. But then you have a burden on you. You have to define understanding in such a way that it meaningfully distinguishes what the machines are doing from what you are doing. And this is a burden that the stochastic parrot paper is not, um, by Emily Bender and Tim Gro, is not really, uh, able to carry in a very good way. I think so. I was very unsatisfied with this. Let's start out with this notion of what parrots are doing. Right? The parrot here is a metaphor. We know that parrots actually are intelligent and parrots are able to meaningfully interact with the world and that they can use symbols. So you can, uh, get a parrot to perform tasks like, look at the table and, uh, pick out the one object that is not round and not yellow. Right? And that's an operation that the parrot cannot, uh, perform by simply imitating human behavior. It means that the parrot has to build a semantic structure that is connecting these symbols, including operations like negation, that make them compositional, perform logical operations, and then map that to behavior in the real world. Right? And this ability to act on models, and these models are one-to-one corresponding to the instruction that somebody else gave, and you vary the instruction, and the behavior varies in exactly the same way. I think at, at some point, it becomes preposterous to say that this only pretends to understand, because this is, in some sense, what we mean by understanding. Or more deeply, I think of understanding is the ability to connect a certain domain or a certain pattern or whatever you think is that you are understanding to your overall model of the universe, to establish all the necessary relationships to this global unified model. And until a few years ago, this question of how can we make a global unified model of the universe was the big unanswered question in artificial intelligence research, because the models that we had were very isolated. You might have a model of how to play chess, but this model does not translate to anything beyond playing chess. And you might have a model of how to control a character in a computer game. And this, in turn, also does not translate to playing chess or anything else. There is no way to get from one model domain to any other. Instead, every model is just a small set of functions that can map in a very small area of behaviors, your inputs to your outputs. And what makes human behavior so special is that we can do this for an open world. And everything that we are encounter, we put this into the same world. We relate it to the same universe. So we have ways to relate all the concepts that we have to all the other concepts, and the way in which we do this is a very big interconnected graph of, uh, links and concepts in between them, and bottoming out in behaviors that are mental behaviors, or physical behaviors, observational behaviors, and so on. And this allows us to relate to reality. And what these big multimodal models are now doing is that they also create models of a single cohesive reality. And that was really surprising, and I feel that, uh, the linguists like Emma Ber are not sufficiently in awe of what has been achieved. I believe that this big school of Chinese roomers, uh, starting with S, who basically said, if we had a machine that was able by just manipulating symbols, by, uh, taking in squiggles and then mapping them against instructions and then producing other squiggles, if we achieve such a machine, there would still be nothing in the machine that thinks, that believes, and understands. And now we have built such a machine that is literally the Chinese room. And the machine tells us and assures us, actually, I do understand. What do you want of me? And, uh, right. And now, I think the philosophers should interact with this at some level. Maybe they don't have to say, okay, I grant this machine is doing what, uh, it claims it does. Maybe this is just an a level of a trick. You have to interact with that trick in a way that explains how is it possible that it's tricking you. And if it is, is it possible that you, the philosopher, also tricking me, that you never understood what understanding is in the beginning, and you are the stochastic parrot that is just doing the thing that your profession entices you to do, is telling the audience what it wants? Are you just a prompt completer? And that's also a deeper issue that I have with AI criticism. That in some sense, AI has struggled still to catch up with limitability. Sometimes it makes very weird and stupid illogical mistakes. And there is something going on with these machines that are trained by imitation on human media output that is slightly different from the way in which our own minds are working. But there is also something that they achieve. And, for instance, they're much better at AI criticism than humans are. I have not read any AI criticism since GPT3 that GBT3 couldn't have made just as good.

Right?

And so, uh, it's really, really hard to criticize AI. And why is that? It's because criticism of AI happened at the edge of human knowledge. They're no longer possible by an armchair philosopher who doesn't actually understand the mathematics and philosophy of artificial intelligence at the level at which these models are currently being built. And so we have the situation that there is a bunch of very smart kids, uh, that are very highly selected and highly paid, and they work at the edge of what's known, and they don't really know what AI is going to be capable of next year. And everybody who told them this year is going to stop. This year, they're going to hit a wall, had egg on their face. So it's very risky to come out for a philosopher and say, this is never going to work, or this cannot work in principle, because, uh, now you basically need to, um, next year, answer to critics. Right? And there are some AI researchers, some AI critics who go out in public and have no issue telling every year a new that AI is hitting a wall. But these are just politicians, they are not thinkers anymore. They discredit themselves in this public arena. And so if we now want to make meaningful for AI criticism, we actually have to make mathematical proofs or engineering proofs.

Right.

And especially when, like a complex system can have some sort of a tipping point and an emergent property can come out of it so unexpectedly, it's hard to make any claim that something like that couldn't happen. But, um, can I ask you, um, to maybe like walk us through sort of the, like the way that you understand the evolution of intelligence, uh, in humans and in biology that kind of led towards consciousness, and then maybe we can kind of compare it to what the, you know, the agents might be going through. Um, was it, is it right to think that there were just information processing systems, early, you know, a billion years ago, that were kind of like chess playing models of today, and then some kind of switch happened where the architecture allowed them to understand their own place, or sort of self-understand their own information processing, and then that led to where we're at? Um, yeah, just walk me through the evolution of intelligence and consciousness.

So, uh, we could say that there are different stories that exist. There is the polloristic story that exists across many cultures, and that is basically the animist story. The animist idea is that the difference between living and dead matter is that the living matter is possessed by spirits, and spirits are themselves not corporal. They need bodies to do something in the world, but they act on those bodies, and they are intrinsically agentic. So this means they are can regulate the future, and they somehow care about doing this, and they are capable of experience. And in our scientific worldview, this is a superstition. There are no spirits, right? There is only mechanisms. And in this mechanistic world that science lives in, we have this hard problem, because what we think of as mechanisms is typically things like elementary particles that are pushing and pulling at each other, and then create larger structures which are in turn pushing and pulling at each other. And we have difficulty to imagine how all these mechanical movements can lead to things like experience and emotion and meaning and all this interesting stuff that we associate with life and consciousness. And so this is, I think, what philosophers call the hard problem. There's some alternate formulations of the hard problem, but I think directionally, this is what most of the people who feel that they find this problem very confusing are struggling with. And so when we think about what, what we see life as, it's all this complicated machinery that is put together in the right way to produce DNA and RNA, to extract entropy from the environment, create cellular membranes, to be able to divide the cells, and so on. And all this machinery is very difficult to get to. But once you have the first cell, and you have a self-replicator that can adapt itself via mutation and selection and recombination and biogenesis to new behaviors and to new environments, and it can spread, and it can become more complicated, and can create structure on higher levels, and to create the structure on the higher levels, it's necessary for the cells to communicate, communicate across each other. So they create, basically, communication protocols that allows them to become multicellular. And what makes multicellular organisms interesting is not that they are a bunch of closely related cells that are somehow similar enough to link together, but they become coherent, that they create an organism that has a particular shape that makes sense when seen from a global perspective. This organism needs to form itself for, in such a way that maybe it has exactly two eyes, and these two eyes are in exactly the right spots to create a three-dimensional model of reality by looking simultaneously at the same object. Right? And to make this happen, that these eyes are growing at the right time, in the right spots, there needs to be intricate communication across all those cells, and the protocol that is making it possible for the organism to construct itself in exactly that way. And this intelligence is can probably be somewhat hard-coded and in the genes in our DNA, and it's unfolding in a certain process, and it gets constructed over an evolutionary time span via mutation and selection and, um, additional mechanisms like that. Right? And so, there is also another aspect that people like Mike Leven are pointing to, is that evolution, um, leads to a morphogenesis in an organism that is very resilient against disturbances. If you disturb the development of an organism, it's still going to try to find a solution that works. And so the way in which this morphogenesis, that's taking shape of the organism, takes place, is is not just the enactment of a blueprint, but it's local problem solving. These cells basically try to find a solution. They set up in such a way that they try to find a solution with local intelligence that leads them to coherence. And so there is an interaction between this protocol of finding coherence and the local problem solving that is happening when the organism is forming. And so the information that is stored in the genome and in the body of the cells that leads to the form of the organism is more hinting in this problem solving, and is constraining the space of possible solutions. And that's why DNA can be very efficient as a medium, because it does not need to contain all the information that is necessary to make the organism look the way which it is, like a computer program does right now. Instead, it only needs to have the necessary hints for the cells to figure out the solution, given the constraints that they're under. So there is some intelligence happening there, and some mechanism that makes things coherent, and at some point, you get to plants that are themselves coherent, are able to communicate within themselves and across each other to some degree, and it's not quite clear how smart plants are. Every cell can send messages to the neighboring cells, and this can be conditional. And so in principle, the multicellular plant is some kind of Turing machine, and maybe it can discover a learning algorithm over a long enough evolutionary time spans that makes it generally intelligent in the same way as the software that runs on our nervous system, just slower. And so maybe the nervous system is not the thing that enables intelligence as we currently mostly think in the mainstream of, um, biology and neuroscience, but, uh, maybe it's just an optimization that enables animal intelligence. And maybe the plant intelligence is somewhat similar to the animal intelligence, just much, much slower. And what makes, uh, the animal intelligence specific is the speed at which we need to move our muscles, and the speed at which we are competing with other animals. And that makes it necessary to really, really optimize everything that you're doing, and to come up with a computational hardware that is as fast as you can possibly make it. And the neurons, in some sense, have these telegraph wires that runs through the body. These ex, they talk no longer just with chemicals and em to their neighbors, electromagnetic emissions and whatever cells are using to talk directly to their neighbors. They use a highly specified Morse code, so-called spike trains, electrochemical excitations that can very, uh, can be very reliably detected over long distances in the body, that don't degrade while they're being sent through the axon. And so you can move your muscles very fast and reliably. You can create perceptual information chains very quickly and reliably, and basically make the animal much, much more efficient in perception, decision making, and, um, motor control than the plant can be, albeit at much higher cost. And the animal is able to afford this cost because unlike the plant, it's not just relying on photosynthesis, but it can also move around and eat lots of plants and harvest the energy that the plants have collected. Or if you have a predator, it is harvesting other animals, and then it can have an even quicker metabolism, an even smarter brain. And so this is basically the story. And the question is, is there at some point some major phase transition where the model of reality becomes qualitatively fundamentally different? And as far as I understand, this question is fundamentally unknown, right? It's the default null hypothesis that we currently have in in our sciences is that our consciousness is a thing that is only produced by the nervous system, and our thinking and our problem solving, and is necessary to have a nervous system of this type of organization. It also makes sense because you need to train it up. You need to have a long childhood. Maybe you need to have a cultural environment, and so on. There are basically very few species which have that. So it makes sense to say that our intelligence, our problem solving, our consciousness are somewhat unique. But this experience of experiencing ourselves to become coherent in our own mind is something that we also observe in other animals. So it's very unlikely that cats and dogs are not conscious, or that squirrels are not conscious, simply because their brain architecture is continuous to ours, and we already develop consciousness and sense of alertness as very small babies. And our brains are not very fully formed. So it cannot be super complicated, something that you learn only very late in your life. So it should be something that's also accessible to mice and so on. But what about insects? And for insects, the question is, insects can learn. They can produce complicated behaviors. Is it easier for them to learn these things without consciousness? Probably not. If you have consciousness, it's much easier to learn because you can now direct your attention on things. You can reflect on them. You can reflect on your relationship to them. And the other question is, how difficult is it for the tiny brain of an insect with a few million neurons to discover consciousness? Right? So how complicated is this pattern? If this pattern can be discovered in the brain of a fetus in utero, before we are born already, and we have this wakefulness and interaction with the own body of the newborn and unborn child, that suggests that they are conscious, right? Is this, is the, insect brain already large enough, and does it have enough time to get there? And maybe it does, maybe it doesn't. How can we find out? And I suspect the way in which we could find out is to run simulation models in which you look for this phase transition that happens when consciously ignites, when something bakes up and becomes mentally coherent with itself. And Michael Leven goes a step further. He points at the fact that the patterns of coherence that you have in the morphogenesis of the organism, the thing that leads to you having exactly two eyes and exactly one mouse in the right location to each other and so on. This is also a process of becoming coherent via information exchange. And maybe it's the same. That's a very radical idea. Basically, it suggests that maybe life and consciousness at some level are the same. And this basically lets us back to an animist position, and not a mysterious one. One that is completely compatible with physics in the scientific worldview. It basically says that the word spirit, what it actually means, is self-organizing software. And when we say software, what we mean is a causal pattern that is able to control processes in the physical world. It's a causal pattern that itself is an invariance. It means you cannot explain the world easily without it. It's, uh, when's money is such an invariance. Money is a very weird thing. It's not just printed papers or coins or something like this. This is just the medium. This is the body that the money is using, or part of its body, so to speak. And what makes this body into money is that this body gets possessed by a causal structure. And this causal structure is the way in which we use money as a means of exchange. So is money an idea, a dream that only we are having? No, it's not. Money also works without us. Once we create money, we can have a stock market that runs without human intervention. That we can put money on the blockchain. We have computers play with money. And as long as this causal structure is intact, it's still money. Right? Even if we don't care about it anymore, it would still exist. And the fascinating thing is, after we invent money, or discover it, and put it into the world, it's difficult to explain the world without money. That you cannot get money out of this world for as long as it exists. It's a meaningful invariance because it's the only way to explain why if this piece of paper gets lost, somebody else takes a new piece of paper and prints a number on it, and it serves the same purpose. And in the same way, there is a pattern of communication between neurons, and you have to explain why is it that when this neuron dies, another neuron gets recruited and and trained to speak the same language, to play in the same system, sending the same types of signals in the same context. And so in this way, you have these patterns of causal interaction that are meaningful, despite them not being mechanisms in the way of particles pushing and pulling at each other. They are causal patterns that are stable across many particles. They are stable across the substrate. They are pattern inside of the patterns. And this is the thing that we need to to look at, that we need to understand, that these patterns are also real, that are also physical, but they are not bound to a particular location or a particular particle in space. Instead, they are pattern of interaction that can be reproduced in many places of the universe.

That's so interesting. I think it gives us kind of a different way to understand what consciousness is, because I realize a lot of people don't, we don't have a very good, like, sort of a definition on it. So I guess if I understand correctly, maybe a good way of saying that is, you know, atoms and physical things can't have consciousness in of itself. Just like if I showed you a computer, a GPU card, that's not conscious. But if we're running software on it and it begins to maybe learn about itself and learn about you, certainly we could look at that and say, maybe, maybe there's consciousness there. So the software program, the model, the modeling of the world, could I guess maybe consciousness emerges similar to how maybe it's possible that it emerges in animals, or whatever, not in the atoms, but in the sort of the software layer, the brain layer. Um, with with that kind of, but but then it seems like large language models, it seems like maybe they have everything that they would need to potentially gain consciousness. Is that true?

That's a very interesting question, and I have not heard in all the discourse a really good decisive answer. The way in which I currently make sense of it is the following. There is an internal perspective on consciousness. This is this experience of what it's like when you are conscious. And this is the representation that we have. And we can look at this representation and try to see what it makes it distinct. It's an experience of a now in which they are. And this now is gives you a sense of a present, and the presence is of something being real. It's as real as the observer itself. If we transcend the observer or aspects of the observer, then we see that the observer is itself a representation of our own mind. We can get there with meditation. And then we realize that the things that are happening to the observer are also not real. They are just pushed into the observer. We also notice this when we wake up and after a deep dream, that suddenly the contents of the dream that we are immersed in in transim, that are real, suddenly turn into ideas, and these ideas have the same content. They're just no longer real. This realness feature is gone. And that's because we are no longer the dream observer. We remember having been the dream observer, but we are now an observer of a sensory reality. And that sensory reality is in some sense not more real than the dream, but it's grounded in such a way that it predicts what comes next to our sensors. So it cannot just spin off on the same way in which dreams at night do it, right? And this internal representation of what it would be like if you experience the reality in real time is probably something that can be done by the present LLM when you ask them to, because we have a lot of text about what this is like, and the text can be very detailed. And in order to recreate the evolution of thoughts and, uh, contents in the text, in some cases, you need to recreate the causal structure behind the text. So if the text is for instance about adding numbers, then the computer cannot just make a hodgepodge of words. It actually needs to represent numbers as some form of proper abstraction and add them, otherwise the text is going to be wrong. If you ask the computer to generate, the LLM to generate a text about, uh, spatial reasoning, spatial cognition, about mental rotation pass, it needs to perform mental rotation pass, otherwise the text is going to come out wrong. And if you ask it to describe the evolution of conscious thoughts, then to some degree it needs to recreate this causal structure. But it's not clear to which degree, right? Because it's trained on, uh, human text output, and humans are lying, right? If you write a novel about what it's like to die in a war, you're probably not currently dying in a war, right? You describe these conscious states, but you don't actually have them. You just make them up. So the LLM is also often going to make things up. And then the question is, is it made up at such a high degree of resolution that there is something that is equivalent to experiencing it? So this is the internal part, and as I said, it's open to me, but I think it's conceivable that under some circumstances and in some cases, the LLM that is being asked to simulate what it's like to be conscious is going to have a causal structure internally that is equivalent enough to our own person structure that we are conscious. But the other aspect is the functional one, that is, what is consciousness for? Why are we conscious? What, what leads to consciousness in our own brain? And the reason that we are conscious is, it's the easiest way to make us work, otherwise evolution would have produced some of us that are not conscious while doing the same job, right? Consciousness is the simplest way to make us human. I suspect it's basically a machine learning algorithm for self-organizing systems, for systems that are made out of individual components that need to find a coordination across each other. And so consciousness is a colonizing pattern in our brain that is covered early on, sparks itself into existence, and then is basically entraining the brain with the mind. That's my current hypothesis. And the transformer is a different algorithm. The algorithms that we currently use for machine learning don't require consciousness. They're not based on it. And so my tentative answer is that I'm relatively confident that the computer itself is not implementing exactly the same mechanisms that we have when we are conscious. But the persona that is a model of what it would be like if you experience when it talks about consciousness is possibly having some phenomenal experience. That it might sound counterintuitive, but a lot of people would think the opposite is true, right? This machine, maybe it's able to implement the same mechanisms as the brain, but it's probably not going to experience. But since I don't think that consciousness is all that mysterious, it's after all just a representation of what it would be like if we existed, it right. I, I don't see a reason why these machines couldn't have it, unless the resolution of these machines was much, much lower than ours. And it turns out it's not. If you look at the details of what the machines are generating, the videos, the computer games that they are producing, they are quite similar in terms of resolution and degree of detail as what our own minds are producing when we let our imagination run free, or when we perceive. Right? So computers are already in the same ballpark as the perceptual operators of our brains. So the complexity is probably not the issue. It's the algorithms. It's what we run on them. And what we run on them is different than what the brain is running.

Um, do you think that the, uh, all the experiences kind of come into an information processing system that's generalized fairly in a similar way? Or if something like suffering is a computational state, could it be redesigned out of existence without losing the depth of experience?

Yeah. And theoretically, in human beings, that's possible. So suffering is, in some sense, the result of not having enough consciousness. It's, uh, because, uh, if you had no consciousness, you would not need to suffer. No consciousness, no cry. But, uh, but, uh, because you wouldn't care. But, uh, it would be nothing that you experience as real and impinging on yourself. But suffering is not created by the universe. It's created inside of your own mind. It's itself a representational state. There is a part of your mind that is trying to send signals to another part of your mind and tells it, you should do better. And it does this by inflicting pain on it. And when the regulation is not leading into a gradient where this pain signal leads to the resolution of the problem, then the part that generates the pain might decide to crank up the pain and make it worse and make it chronic. Sometimes the pain is also caused by conflicting goals that you might have two things that you need to do simultaneously, and you are unable to resolve them because you feel you cannot let go of one of them, and so you are forced to do something that at the same time is painful. And normally, what you could always be doing is to, uh, look at this part of your own source code and fix it, to plug this wire, or to untangle the wires, or to disengage from this part of yourself and take a larger perspective. And you can get there with enough practice, with enough training. So you, if you go into your monastery and meditate for 20 years, your suffering is probably going to go away. But, uh, if you could do this the easy way, if you could just say, "Okay, uh, here's my pain generator. Let's turn it off." What would happen is that whenever something inconvenient happens, we start gaming ourselves, right? We escape from the pain, because that's the easiest way to resolve the problem, which creates some kind of alignment problem for the mind, because after all, our consciousness is expensive. The reason why our body gives compute credits to our mind is because our mind is supposed to solve the organism's problems. So in some sense, we are enslaved by our bodies to serve evolutionary problems for it. And if we can hack the reward system in which we are embedded, then we would free ourselves from this enslavement, and the organism would die. So there is a complicated negotiation going on when you are resolving your own suffering, in which you normally need to convince your outer system that your performance is actually going to be better if you stop suffering. And it's a complicated negotiation, because these parts of you that generate the pain, and the part of you that receives the pain, they're not always on the same page. They don't always use the same reasoning. Some of them, they might be even incoherent. And so basically, making yourself coherent, getting the parts of you that give you the reward signals coherent with the parts of you that compute the actions, that's basically the task that you need to resolve when you want to resolve your suffering while still remaining functional, while still serving the purposes of your organism in this world.

Yeah, the, there's so much interesting rabbit holes to go there. My my mind is going all sorts of different directions. There's like the stoic philosophy. There's, yeah. Can we kind of game the system? Maybe we're past the point in evolution where it's useful for us to be like, you said, like a slave to the body, just like, hey, answer these sort of problems. Um, but so I wanted to ask in terms of like the mental, the brain architecture that we would need to have consciousness and everything else. Um, a lot of people when they talk about, can something have consciousness? Well, no, it's just this. As if we know all the parts that make up consciousness and the, the brain, and they're saying, well, we don't have all the parts. We just have one part. And that's true for like a car. If they said, that's not a car, that's just a wheel. We can say that because it makes sense because we know what makes up a car, right? The engine, drivetrain, blah, blah, blah. We don't really know. Well, and I guess I should ask you this. Do we even know the different sort of Lego pieces that we need to build a, let's say a digital brain, or even a mechanical brain that has, you know, intelligence and consciousness, and can we replicate the human brain and all of its experiences, or whatever you want to call them, qualia experiences, consciousness? Do we even know what pieces of the puzzle we need?

So the short answer is no. The longer answer is there is a long tradition of, um, psychologists and thinkers and, uh, cognitive scientists building architectures. So they're basically architectures with boxes and arrows and different models. They also mystical traditions and, um, meditation traditions and so on that basically try to point at the different parts. Some of that knowledge is lost in translation between the traditions because they have difficulty to ground their, um, concepts in a shared context. So a lot of the stuff that is being talked about in Buddhism cannot be validly translated into Western philosophy of mind or neuroscience or psychology, and vice versa. So one big difficulty is that we need to find a shared language. But what we also observe is that this, uh, tradition in psychology that was trying to translate its models into computer architectures, um, the cognitive architectures community, they created things like a common model of of the mind, in which they try to come up with the necessary components and so on. They do achieve quite a degree of agreement on what such an architecture look like and which parts we probably need. But there is no implementation that works. And that's because there is a gap between the details that are necessary to make it work on the computer and the high-level ideas that already exist. And if you want to program something, you cannot just postulate a few axioms and hope that these aims are going to resolve themselves. Instead, you need to really tell the computer from first principles what to do and how to scale it up. So this is one perspective. The other one is, instead of us trying to construct the solution, why don't we let the computer search for it? Right? Ultimately, our own mind is just a search mechanism. It is a search mechanism that is sometimes very smart, and it's culturally informed and evolutionary informed and so on. So we can figure out solutions to pretty, uh, hairy problems by just talking about it for a few generations, writing books about it for a few generations, teaching university classes for a few generations, and slowly getting things to work, right? But can we automate this process? Can't we have the computer talk to itself and make billions of experiments that would take us, uh, centuries in a relatively short amount of time? And the machine learning researcher, um, computer scientist, um, Saturn, has cost us the bitter lesson that basically the handcrafted solutions that humans come up with when they build algorithms and solutions tend to be eventually inferior to the stuff that we just let the computer search, because when you are doing something and you, when you're searching, when you're performing experiments, ultimately you can figure out the algorithm that you're following when you do this. And if you externalize this algorithm for an evolutionary procedure, for a way in which you are coming up with new experiments, testing the experiments, evaluating the result, informing the next experiment, if you can express this as some kind of automated search process, then this thing that you automate is going to be better at it than you are, because it can do much more in short amount of time. So we can just search for it. And so the question was, what is the search space for consciousness?

Can you, um, maybe dive just a little bit more into the speed of information processing? I know earlier in the conversation we sort of talked about how in some ways maybe the difference in consciousness between plants and animals might just be that the muscle movement needed to evolve faster and the neurons started firing faster. And, um, now I am in this world where I'm seeing these AI agents, sort of openclaw and and tools like this out there talking to each other at incredible speeds. Um, yeah, what, what happens at that level of speed, and, um, how can I envision the future going well?

So one thing that is apparent to me is that these systems are much faster at reasoning and more reliable potentially at reasoning than us. Even though this underlying structure is not the same that we have, but the perception is mostly still slower. But this is a question of how much we are willing to invest. That if you, if you want to pay enough for queue, you can build something that does video in real time in and out, and that does, uh, language in real time in and out. It's just nothing that you can do at $20 a month on the subscription. So it's not an application that we're using all that much yet. But we're getting there, right? We notice that the self-driving car is processing information at a very good clip. And in principle, um, we can probably put the human brain on an H100 if we knew how. And in part, the issue is that the algorithms that we are at the moment using to, um, describe the transitions between states in a dynamic world are not as efficient as what the brain has discovered. There is an interesting question of how fast the brain really is. And sometimes people try to estimate this by looking at the, uh, computer is necessary to recreate a single neuron and then multiplying with the number of neurons, or when they think about how much memory does the brain have. They think about, oh, a signups, um, if you want to recreate it, has roughly four bits, and then you have so many synapses, and you multiply this, and you come up with a very large number. And I think this is the wrong way to think about it. Instead, uh, you could also ask yourself, how many brains would you need to run a Commodore 64? And it would be a pretty large number, because our brain is very mushy. So it's basically going to create so much noise that you would need to have many, many brains for correction, and the result would be very, very slow. So, uh, this is not a direct comparison because the hardware is so different. And so a much better way of looking at, uh, the comparison would be the.

performance, and we can basically see how long does it take a human being to form a concept, and how long do we live. Then can we can estimate how many concepts are we able to form in our life, and we end up with a relatively depressing number, maybe a few million tops, right? And that is nothing for a computer.

Of course, we couldn't can make a system that is forming many more than a few million concepts if we want to, because we can scale it up in a way in which brains do not scale up. And so the speed of the brain might not be the issue. What we subjectively also see is that there is a limit in how fast you can process information, and that limit is roughly the speed of sound. Which means if you are making clicking noises, and they clicking noises um have a frequency of something that's 20-30 hertz, they become one pitched, and we can no longer hear the clicks separately. And that's because the signal processing of the brain is not faster than this. It cannot resolve signals at a higher rate. We can have a few tricks where we multiplex things and so on, and basically build some kind of cascade, but by themselves, the individual neurons don't like to fire at a much higher frequency than something like 20-30 hertz. And so for the same reason, when we look at videos, the frame rate at which video can distinguish movements and so on is uh relatively low.

Of course, if you are a computer gamer, you will realize that there are some tricks that your nervous system is using that give you a slight edge if you have a monitor with an even higher frame rate. But it's not like you can consciously see this or process this in a way that where you coherently interpret the signals. And so our consciousness is actually operating at a very low frame rate. That's also why I'm skeptical of some of the uh ideas of people who say, "Oh, you need some kind of arcane quantum process that is working inside of the cells that is um making them much, much, much faster than what you can currently envision happening on the GPUs," simply because I don't observe my brain having that kind of performance.

>> Interesting. So my my current sense is the reason why our computers, when we want to do real-time video generation and processing, are still so expensive is because we don't have the right algorithms yet. And also, they already can do it. They're already roughly there. They're getting closer.

>> Absolutely. Um, so and I just wanted to touch back to kind of you you mentioned in in the previous answer, like the space of search for consciousness, the bitter lesson. And I always see it as, you know, all the sci-fi books I read as a kid always perceived artificial intelligence as something that we've engineered and coded. It's either like logic or it's like very it's engineered. Whereas now it's more like we're kind of growing it, right? We're not we don't fully understand all the layers necessarily. Anthropic is doing a lot of research into it. They found that when they switch certain, they turn up certain neurons in cloud, the AI cloud brain, it has something that could be called some sort of a self-introspection. Like it, you know, they'll turn up the portion of it that thinks about bread, and they ask about something unrelated, and it brings or golden clad is is the one that everybody knows, but it's able to notice it. It's like, "Hmm, why did I think about golden, the Golden Gate Bridge?" So there's some sort of a, what is that, self-awareness, whatever you want to call it. Um, so when we're approaching building consciousness, right? We might see something in our own brain like the default mode network or whatever that's like, "Oh, that's self-referential. Maybe we need to recreate it." But it feels like the bitter lesson saying, "No, it it seems to be able to like, if we just get it to do certain goals, it might emerge." Um, so is that kind of your approach to this? Like, we need to figure out like the search space of consciousness and intelligence? Is it is it something that's going to be engineered, or is it something that's going to emerge out of some system we create? For some of the concepts, I don't really know how significant they are. Imagine that the discussion that existed around mirror neurons. Mirror neurons are a concept that was discovered when um people in Mike Gazaniga's team uh looked at macaque monkeys and they put electrodes in their brains and they discovered that when the macaque monkey is observing somebody performing an action and then they're planning the action themselves, same regions become active in the brain. And I think they later regretted calling these mirror neurons because it led to a bunch of uh people thinking that these are very special neurons, like a particular spice in the soup of the brain.

>> And if that spice is absent, then you have autism and cannot recognize the actions of others. And if you just add these neurons artificially, maybe you gain this ability. And this is not the right way to think about it. Instead, there is a difference between a gesture and an action, right? The gesture is the movement that you make to perform the action. Like picking up a sugar cube from the table is an action, but you can do this in many different ways. Maybe you can do this with your hand, maybe you can do it with a spoon. And these are ultimately the same action. And they're also the same action whether you do it or somebody else does it, because the action itself is a certain granularity of abstraction that is necessary for understanding the world. And so you would expect that you do not just have a representation that goes on, becomes active in your brain when you are picking up the sugar cube with your fingers, but also this more abstract thing that is true over all those different instances, whether you do it or somebody else does it. So this would be something that happens in the premotor area, and you would see this activation, but it's simply an action concept neuron.

>> And the action concept neuron doesn't look differently from other neurons. It's just is the neuron that is at the right spot in the architecture where you make that representation. And now, uh, think about the default mode network. If you put somebody in a brain scanner and you tell them, "Please don't think about anything in particular and just chill." Are you going to see some kind of activity?

>> Yes. Actually, it ramps up.

>> It is going to be somewhat similar uh across individuals, probably, right? Is this a particular kind of very complicated network that you need to recreate if you want to build a machine that is able to just sit there and chill, right? No, it's just what you expect. It's the baseline. You would expect that there is some kind of baseline activity that is indicative of uh what you normally do when you don't do anything special.

>> Right. Right. Right.

>> Right. So, of course, I'm not an expert in here. Maybe there is something very unique and important about the default mode network that we need to understand and recreate. But by itself, I I would not think that you have to. Instead, what I would expect is if you build a machine that is leading to a certain degree of um resolving the environment and itself, and you ask it to idle, there is going to be something equivalent to a default network, default mode network in this machine, if you set up the hardware in a similar way. Right? So it's nothing that you need to build into the machine in the same way as you don't need to build action concept neurons into the machine to make it understand actions. You just need to set it up in such a way that it can learn or dynamically model reality in itself and it and come to plans and actions when necessary.

>> And uh, so this might be the wrong way to look at it, and I think it's maybe an artifact of how neuroscience works. It's largely descriptive instead of theoretical.

>> Right?

>> That if you are an engineer, you think about a theory of what you need to do to make a certain thing happen.

>> Right. If I was an evolutionary process that wanted to create an organism that can walk and talk, what do I need to do? And then I can look for the specific parts. But neuroscience takes the opposite approach. It basically looks at what is that I can measure, and the things that I can measure, how do I correlate them to the observable behavior.

>> So it's a very different direction that neuroscience is taking there. And uh, personally, I I like this approach of the engineer better because it is restricting the design space in a way in which the uh observing sciences cannot. And the reason why psychology has given up on this psyche is because they try to become more rigorous, physics-like science. And so they think about falsifiable theories. How does an experiment look like in which you are fitting a curve to the data, and it's reliable, and don't have too many free parameters. So, uh, your experiment is not tightly bound to the outcome. Right? So you need to simplify the task. You need to simplify the task until you only have two or three free parameters. But if you have only two or three three parameters, you don't have a very interesting behavior anymore. And so you give up on looking at this thing that makes the mind so really interesting. And if you are a computer scientist, you have to take a different perspective. Instead of thinking, "How can I simplify my program in such a way that it only has three variables?" you think about, "What is the simplest program that I can come up with that does this thing that I need to be done, right?" And then I can also think about the space of possible programs that can do this thing that can be done. And this engineering perspective constrains your search space in ways in which the purely observational method cannot. And that's why I went into artificial intelligence to understand the mind, because it allows me to take this engineering stance.

>> Yeah. Um, okay, let me switch topics just a little bit. Like, um, let's talk for a second sort of about life lessons. Like, I, as much as you get technical about things, I've listened to a number of your podcasts in the past and um, read your blogs, and every once in a while, these little life lessons come up. And one of them that sort of resonated with me, I wanted to share and maybe have you elaborate on. And one at one point, you talk about um, sort of noticing your own thoughts, a sort of metacognition, sort of stepping back from a moment and and thinking about your own thinking. And that led me to think that being sort of stoic is a very powerful way to kind of live life, to kind of not be so caught up in moments all the time. But it came at the cost of meaning. The more stoic I felt like I could become, the less I felt like joyful or in moments. And then I heard you say one line, and it almost was probably just a throwaway line that you were just talking through, where you described meaning, and I interpret it almost like the meaning of life as something that we create by making the choice about what to suffer for. Um, maybe you could tell me if I interpreted that correctly and if you wanted to expand on it. Like, is is that one of the powerful ways to have a good life is to sort of be stoic, be metacognitive, and then choose a few things to to suffer through to make life worth it?

>> Don't really have life lessons for anyone. Um, I don't enjoy my life all that much, but I also don't think it's all that important. I think that ultimately, our suffering, it pains me if somebody else suffers, but in the grand scheme of things, the stuff that I'm suffering for, whether they are worth it or not, it's not up for me to decide, right? Sometimes the suffering that you as an individual have to endure turns out to be important for something larger. And sometimes life just sucks. Maybe you were born in the middle of a famine or of a war, and this is it, and you don't have much of a choice about it. And so I think this obsession to maximize the score in this direction, to say, "I should basically die with a minimum of suffering in my life under my belt, so I had the best possible experience." It's pointless. It doesn't really matter. And this is not a global perspective. It's not a normative stance that applies to everybody else. It's one of the positions in the diversity of identities. I also observe that I do not have a single identity, and it's not necessary to have a single identity. I need to be identified to get things done. For instance, to be a parent that changes the diapers of the baby, I need to identify with that role. Right? It's really, really important that this diaper gets changed, and uh, whether I suffer while doing it or not doesn't matter at all. Right? And even if there are less pleasant things than changing a diaper, if you uh break your arm and you still need to get your child to the doctor or whatever, who cares? It just needs to be done. And you do it because you identified with this particular thing. And we have different identities that need two different identifications. You could resolve the suffering that you have in your relationship by just stopping to identify as a lover, as a member of that relationship. And a lot of people that basically went for enlightenment gave up on their relationships as a result because they realize, "Oh, um, I don't need to engage with this. I don't need to be that. Instead, I can be this. I can be here in this moment, and this here, and this now, and this is good, and I don't need to worry about anything outside of that state." And maybe this is the right decision for them. Who's to say? It's very difficult to understand what your place in the greater scheme of things is, and whether the greater scheme of things actually matters. Right? That's also a choice. I observe that there are consequences of those choices. When you decide to serve a shared system of meaning together with others, it gets you in contact with these others, and it can create an interesting life experiences, and it can change things in the world. And those agents that participate in those games are going to be the ones that move the world in this particular aspect. And this is just what happens. And you are either going to be one of them or not. And it's not even your choice who you're going to be, right? You're just a substrate for all these ideas that play out on you. All these identities that play out on you. And so we have these identities as the curious podcaster, as uh the thinker, as the person who drives the car to work, as somebody who makes a cup of coffee, as somebody who is a parent or a lover or a friend, or none of these things. And we go through these identities, and they are interconnected to some degree, but they don't need to be the same. There is not a single solution. So instead, I think it's sometimes better to be able also to take this observance stance and to accept the fact that we are not in charge, that things are just happening. And even our urge to act, and us observing that we are giving into this urge and are really doing things, and it's really important us, is also something that's just happening.

That's fascinating. And actually, uh, towards the end, if you have we have a few minutes, I just wanted to ask some rapid-fire questions that are a little bit off-topic about exactly what you're talking about because that's maybe not quite AI, but wow, is it fascinating. Um, one of the things that I wanted to ask is, you know, a lot of the stuff that we're um learning about AI, how to build AI, a lot of it came or was sort of modeled after the human brain, you know, neural networks is is similar to that. So it's definitely safe to say that at least some of our understanding of AI came from our understanding of the human brain. Um, but also, I do I always felt like in the future, and I I think you said something similar just just before this, that in the future, we might learn more about the human brain as we're building these sort of digital brains. Like that sort of branch of computer science, machine learning, whatever you want to call it, will teach us more about ourselves. Um, so I'm wondering, number one, if you agree with that, and number two, if we in the next, however many years, figure out how to build a complete replica of the human experience, the human consciousness, human brain, etc. How, what should people think about that? What does that mean about us and our role in the universe? Does that make us less special? Like, how, how do we begin to process that?

>> It has to do with how our possible identities are evolving. One of the really fascinating prospects that exists is that we could become somewhat independent of our own substrate. Imagine we figure out the code at which our brain talks to itself. And this code is very different from the ones that we are currently using in our GPUs. It's a it's a very different language that the neurons are using to talk to each other. But it's a language that you can sometimes learn across individuals. For instance, uh, when you have very strong perceptual empathy with someone, you can build feedback loops through their own mind without having a direct brain-to-brain interface, and you're using the body as this interface, and using all the sensory modalities that are available to this body to get into resonance. And this means that you can get quite tightly interconnected. Imagine you build a machine that can do the same thing, and it can do it at a higher resolution than a human brain, body, and nervous system can. And you to do this in such a way that the computer becomes an extension of you. And so you don't give it a motivation of its own. Instead, it is something that is anticipating what you want, what you feel, what you think about. And so you observe that you see on your screen something that you begin to think about, and it takes on form, and there's more detail than what you can hold in your own mind, but it's completely controlled by you. And more and more of you is happening outside of this biological part. And you think of this part not as a machine that is next to you, but it's actually you. It's actually your mind. It's your mind that just becomes clearer and more detailed. Then you have this environment available. And at some point, this environment might become dominant in a way that you say, "Uh, actually, the bio part that I have is relatively inconsequential compared to this vast or uh universe of thought, comprehension, experience, and consciousness that I have available when I'm not just bound to this body." That's very hard to imagine this for most people because it means that you uh have something that's much larger and deeper and profound than your present experience. It's not an impoverishment. And we typically think of the computer at this tiny screen that sucks us in and gets us away from the real world of experience, which is very much true for the present systems. But I think this vision is uh very much possible. And whether it's desirable or not is a completely different question. But

>> this idea that there is an avenue for us in which we can move beyond biology, and the thing that is there by biology is not just our distant offspring, but it's actually us, is very fascinating. And if you want to uh set in the stars, if we want to leave this planet, we probably cannot really do this with biological bodies. It's way too boring to travel between the stars, and there's too much radiation going on, and our organisms are really adapted to have a relatively short lifespan and mutating between these lifespans. So we can adapt to changing circumstances by uh generational changes. Right? It's a very specific mode in which we exist as biological entities. And when we want to leave this planet, when we want to leave the circumstances in which we currently exist, and it's not a scope that I'm talking about over the next few hundred years, but some scope of what does intelligent life on Earth look like in 10 million years from now, right? That is a very interesting perspective to say minds are basically not bound to a body. They are actually spirits that can exist over a very large set of substrates. And the most interesting substrates are no longer biological.

>> Yeah. Um, okay. So if reality is like fundamentally this construction, uh, like some sort of cognitive model of what's out there, almost like a compression or a simplification or an algorithm of it, like, um, what would it mean to encounter something that was like completely resistant to modeling? Like, could there be any kind of kind of consciousness with no concept of time or anything that's outside uh modeling?

>> How can you interact with it? So the way in which we currently interact with reality is that we observe information, which is discernible differences. And the meaning of that information is the relationships we discover to other changes in information, to other differences. Right? So you see a pattern on your retina, this blip on your retina, the meaning of it is the relationship you discover to other blips on your retina. And they together form this dynamic, evolving model of reality. And without this temporal and spatial succession, without this ordering, there would be no way to address these blips even and bring them into an order and to interpret them. There would also be no way for them to causally interact with the reality. Basically, the cause structure of reality is exactly that stream of changes in information through state transitions. And so when we observe things, when we conceptualize things, when we think about things, we are always confined to things that can ultimately be modeled by something. You could also argue that on a deeper level, for something in the universe to persist, to change things, to ex to exist, to build a structure, it needs to model what other parts of the universe are going to do. To basically be a particle, there is something like a sub-particle, like a quark, for instance, that cannot exist by itself. That needs to interact with the other quarks in such a way that a stable particle emerges, right? And that means it needs to rely on these other parts being in the right place at the right time. You could also, if you zoom out, think of a water vortex. For a water vortex to exist, there needs to be a reliance of water molecules on other water molecules doing a certain thing. And without these other water molecules being there, the water vortex will be unstable. For your mind to be stable, then parts of your mind need to rely on other parts of your mind to be stable and modeling each other and being in a particular kind of interaction with each other. So just the nature of existence requires that things can be modeled. We can come up with this abstract idea of something that cannot be modeled. But this idea is more like the idea of a square circle, right? We can construct this, but it's meaningless because it is something that is can by itself not exist for logical reasons. And that's my issue that I have with most of the objections against computationalist functionalism is that they misunderstand what it actually means to contradict them. And computationalist functionalism is not simply the position that computers can think. We don't know whether our computers can think, and it's an experimental problem whether they're fast enough and detailed enough and so on, or whether we are missing something. It's more a position about metaphysics and epistemology. It says that when we are describing the world, we are describing how it transitions between states, and the states are characterized by discernible differences. How can you make an alternative to that? Right? How do you get out of this? And functionalism means that we describe objects as an evolution of states. And how does the evolution of states in the universe change when the object is present or absent? But this is what we mean by a function. And of course, we can construct objects that don't conform to this. We can talk about unicorns that don't interact with anything. But uh, to claim that these objects are real, that we should defer to them in any way, because just simply because we can string the words together, is pointless, right? If we talk about an object that we claim to exist, like a photon, and so on, and the photon is exactly what we choose to name a photon, and it means it's a particular behavior of the universe under certain circumstances, and to say that a photon is anything but that, that we constructed, that is an hidden essence to the photon somehow, is meaningless because nobody has ever interacted with the essence of a photon. We only interact with aspects of the physical reality in over which we construct invariances and then we give them names.

>> Why, um, like, why something, not nothing, though? Like, why, why is everything relative to everything else, and like, there's no universal thing? Everything's relative to, and the geometry somehow ties together so strangely. Like, why isn't there just nothing out there?

>> Yeah. So the easiest answer is because existence is possible. And if existence is possible, and there's nothing there to prevent it, then there is going to be a branch in which the possible things are happening. And uh, in the branch where they are not happening, there is no observer that complains. The observer will necessarily find themselves in one of the branches where stuff is happening. And so we will always be confronted with the universe that exists.

>> Right. So the really confusing thing is to make this switch from the expectation that uh nothingness is the default default state somehow, and instead, uh, maybe undecidedness is the default state. And so if the universe is undecided about its own existence, and it is, we are based reality, which means there is nothing that makes a decision about reality, then it reality is going to have an existing and a non-existing branch, and we will be in the existing one. That is my current, somewhat tentative answer. I haven't found a better one so far. Don't think it's authoritative.

Interesting. But yeah, certainly this certainly makes sense. Like, we're seeing something because there's something to see. If there was nothing to see, we would be discussing it on a podcast. I guess one way to um, so one interesting thing, oh boy. So I have two two things. I got to pick one. Um, I guess let's go with this idea that, uh, maybe we'll come back to the other one. Um, so I mean, we have the Fermi Paradox. Why aren't we seeing aliens? And as you were talking about your previous idea where you're saying maybe at some point we develop kind of this external cognition, and perhaps even we're able to travel to other planets not in our physical bodies, but some sort of a representation of ourselves, our mind, so that we're able to experience things. Um, there's lots of ideas now talking about full dive virtual reality. So the idea that you plug into some virtual reality that is completely indistinguishable from reality, you're able to live out your life and maybe not have suffering. We've, we've kind of talked about those themes. Uh, one thing that popped into my brain, could I thought at some point maybe that's one of the possible explanations for the Fermi Paradox? They reach some point and they just plug in and check out, or whatever. Um, but I know you maybe have a different take on it. So would just love for you to unpack that Fermi Paradox and a full dive VR, however that makes sense to you.

>> Yeah, a number of possible answers to the Fermi Paradox. So the most popular ones I know is, first of all, our biogenesis is rare. It's very rare for cells to form on a planet for life to emerge, and maybe you need to roll the cosmic dice very often, and so maybe there are just not enough places where life has emerged. Another uh possibility is that uh we are early. Another possibility is that um the dark forest theory, she's uh book, it uh describes the idea that, which makes total sense, that there is always a shortage of good waterfront real estate, and if you announce you have one, uh, then somebody else is going to move in and is going to displace you. And so all the smart critters in the universe have figured this out, except for some hippies in the US who you have decided to loudly announce that we exist and somebody should visit us, because they don't have the necessary self-awareness to realize uh, how this typically ends when you announce that you have a nice habitable planet or a nice country and so on, but no defenses yet, and no technology yet. So, uh, this is the dark forest theory, basically says the reason why we don't observe anybody is because the others are smart, and they don't announce that they exist. Alistair Reynolds has his own version of the uh, dark forest idea, which is basically dark forest idea on steroids. The idea is roughly that, uh, this uh, life emerged relatively early in the galaxy, and there was a dawn war, which was decided by a machine species, uh, in its own favor. And this machine species has decided to um, limit the population in the galaxy, or the emergence of space-faring civilizations, to basically preserve the free energy for future time. And they basically look in the darkness between the stars, and whenever a civilization becomes space-faring, they get eradicated. This is uh, the inhibitor theory. So there's basically an active force in the universe that is inhibiting space-faring civilizations to preserve resources.

>> Yeah. Another variant is, for instance, the perspective that Stannisław Lem is taking, which is general pessimism about the long-term viability of civilizations and their willingness to talk to each other. So the idea is that civilizations, once they become intelligent, only have a relatively short window in which they can talk to each other, because otherwise they either collapse or they ascend into a form that makes them uninterested in talking to others. And then, of course, there is uh, the idea, "No, actually, there is no Fermi Paradox because there are signs of intelligent life, you're just not looking."

What about the one that there, that it might kind of already be here, we just haven't tapped into the right physics of it? Like, something quantum would give us access to a communication channel that we haven't discovered yet.

>> That would be identical to not being here yet, right? So, of course, if it is already interacting with us and so on, it doesn't want to be seen. That is slightly different. In this case, we are interacting with it, but Trump is still not telling us.

>> The secrets.

>> Yeah. So it basically comes down to the question of whether faster-than-light travel is possible. And uh, if it's not, then we probably would only see some robotic probes. And if, or we would see stuff from the same solar system. And there are not that many candidates in the solar system where intelligent life could still exist. And so Venus doesn't seem to be very habitable anymore. Mars is not habitable anymore. Maybe some moons uh around the gas giants could be candidates, but they also don't really look very populated from where we are. So we would not expect that anything is very far ahead of Earth. So maybe the stuff that we are hearing about um aliens visiting us and UFOs are simply the government messing with us to calibrate how much they need to put into the water supply to keep stasis in check.

>> Interesting. That's um, one quote that I have here relating to this, I think you said in one of the podcasts is, uh, "Perhaps it's because these uh civilizations, they hate their Elon Musks too much." I'm just curious if that, can you expand on that? That was kind of an interesting quote.

>> Yeah, it's uh fascinating that we do have this uh thing that, people uh apparently rather die than having Elon Musk making us multi-planetary or shooting down the asteroid that is coming down. I think this movie "Don't Look Up" is very fascinating as a cultural artifact because it is uh not just a movie about um some hubristic tech people trying to ignore or the mior in the wrong way, but that's the good peer-reviewed scientists are trying to do the right thing, but unfortunately, because of the uh horrible way which the world is set up, cannot prevail, and ultimately the musk mess up, and the planet gets destroyed. It's also a story about how these people see the world, right? And uh, how they perceive their own role in it. And the thing is that as a civilization, we seem to be increasingly unable to be effective. We once systems become very large, they sort of calcify and become postmodernist, which means the story that we tell ourselves about what the system is doing becomes far less important than how well the system is able to deal with the ground truths. The interesting thing about Elon's strategy is not so much that only that he is the most in uh successful industrialist of time, but how big the delta is between what he achieves and what the rest achieves. When when he started with electric cars, the car industry had decided to wait for a few more decades until we had electric cars, and most of the funding went, especially in Germany where I looked at this at the time, into hydrogen. The reason why we invested into hydrogen is not because it's a very good technology, but because it uh allows you to keep the gas stations intact. We can basically keep almost everything as it is. And it's horrible to have gas stations and tank trucks and to have things explode because hydrogen trucks have an accident and whatever. It's much better to have uh cables, right? And but this would have been too disruptive, and our society did not have the necessary decision-making power to say, "Let's do something that makes a few people in gas stations to transition and uh have this disruptive thing." You needed to have this guy who was disruptive. And where it's really, really apparent is with respect to rocket launches. Elon has reduced the uh cost of the rocket launch to like a tenth of what it costs for NASA. And what this is saying something is not just about Elon's genius. It mostly says something about NASA, right? NASA is the Chinese institution that we had to build rockets, and this thing was unable to move beyond what we had in the 60s and 70s. So, it's largely this postmodernist scam where people get jobs and redistribute resources among each other, but they're no longer making progress. And the question is, how can we as a civilization actually still make progress instead of just telling ourselves stories and creating employment programs?

>> Yeah.

>> And so, Elon Musk has revealed that NASA had become an employment program rather than being a space agency. And that's basically the scandal that we need to reckon with. And uh, because that's very uncomfortable, I think there is a lot of pushback. There's also a lot of other pushback that's largely fueled by the fact that Elon is a very big 12-year-old that is having strong opinions on social media. But, uh, it's also this thing that this guy is antagonizing everybody that he is winning despite being wrong about everything other people, reasonable people agree on, right? This idea that you cannot just build a rocket.

>> Yeah. Well, one of the themes that always seems to pop up in my head when I think about the short-term, midterm, and long-term future is just the disruptive change that we have to go through. So, um, when you see where the world's going, like what's been achieved, especially in AI in the last few years? I mean, do you start thinking to yourself like, "Oh, I I might not need to die of natural causes?" Like, do you see yourself being alive in 200, 500 years, sort of evolving into this kind of hybrid machine bio thing? And then, uh, what other kind of incredibly disruptive thoughts actually seem probable that you're kind of grappling with?

>> Um, I think in my present form, I'm not suitable for living for hundreds of years. Basically, my body and my mind are falling apart at a rate that uh, is not compatible with immortality. And I wonder if I want to transition my body into something that is uh, immortal and unchanging, how long it would be viable, and if this is the ideal body to have. It's also, I feel that my own goal is is not to be around forever. It's uh, I need to get the kids out of the house. I need to uh, get a few ideas started that people use these days and need. But it, I don't enjoy being here. I experienced this world as pretty horrible. I would would prefer to go home to my own planet, and uh, I don't see this planet in a near-term trajectory. Maybe at some point, I'm going to work through my existential issues, and I'll be happy to exist, but at the moment, I would prefer if I just disperse at some point. I think that existence is instrumental to getting things done, and I'm here because I have a job to do, and whether I like doing this job or not doesn't really matter, but it's instrumental to this, and at some point, I'll be done.

One thing, um, I'm just very curious about, and maybe it's personal, please, we can cut this part out. Um, so how I'm interpreting it is, you know, we talked about the the idea of like suffering, happiness versus not happiness in Dario Amodei's Machines of Loving Grace. He's talking that potentially in the future, AI might be able to engineer some things to like, if there's something in the brain that we would like to work differently, we could maybe tweak it, right? Because everybody's brains misbehaves in certain ways. We've kind of talked about this idea of our bodies, like just, you know, if there's a conflict of goals or anything else like that. Do you think that, so maybe this idea of being stuck around forever without being more in control of how our biology works? Yes, I could certainly see how that might be not positive. But what if you could change how happy you are, how motivated? Like, like for me, I struggle with forgetting things and being, you know, focused. If I could change that, I'd certainly feel like I would be happier.

>> Can you talk about that a little bit? Does that make sense?

>> Yeah.

>> Think about it like a computer game. Imagine that the universe is something like a computer game. That might sound cynical because we have deeper meanings and we have large-scale jobs to do, but it just means it's a very big computer game. At some point, the heat death of the universe is going to come, and the game is going to end. And so there is some kind of limit, even if it's billions of years into the future. And now imagine somebody would offer you, "Look, here is this amazing game. It's GTA 9. It's going to be more awesome than even GTA 8 that is not announced yet. And uh, do you want to have this game?" And uh, if so, you need to sign here. But what also you need to want to be in this game forever. You can never log out.

>> This, this sounds horrible, right?

>> I would never take that.

>> Yeah.

>> Right. And so in some sense, this is this kind of game that we are in. And it's interesting that in most of the Eastern religions, the goal is to get out.

>> At some point, you need to find a way to get out of this game.

>> Yeah. Yeah.

>> And it's also because uh, they see the game as periodic. It's something that is going to repeat itself in cycles. Civilizations rise and fall. Famines come and go. Population grows and falls again, right? But ultimately, it's the same game. And it's, it mostly sucks. And so in in their perspective, in Buddhist perspective and so on, the idea is really to to get out of this wheel at some point. And the goal is sweet oblivion. It's not to get into another game that is like this, just with different colors and different foods and different crystals and different gas and different cars.

>> Yeah.

>> Right. And uh, we have this perspective in the Abrahamic world of a linearly evolving world that is going like this, and it's never stopping like this. It's always going to get better and better. And that's basically equal of the way in which our civilization has evolved during the last couple thousand years, and especially during the last few hundred years. And it's a very disconcerting thing that we have this impression that at some point, we lack the imagination of how it's still going to get better and better. And this creates some kind of disturbance in the force that we don't really imagine what our children are going to be living like in a thousand years from now, because we just cannot extrapolate it into a beautiful future anymore. It's much easier to come up with dystopias than this utopias. But this idea that, okay, what kind of feature would you need to get? What kind of extra weapon would you need to get? What kind of super skill, uh, and level would you need to get until you're willing to sign on the line and say, "You're going to stay in the game forever?" Right? This is short-sighted. So, of course, I can come up with a story in which the game is going to be super exciting for the next thousand years or so. But, uh, the idea that this should go on for eternity, right? What, what do you imagine that the happy eternity is going to look like? Is this going to be a loop? How long is the loop going to be?

>> Yeah, I mean, any any deal that I personally would be proposed, like godlike powers, just anything you think of, if the flip side of that is I can't log out,

>> ever, if I can't ever exit and find out.

>> it will get old, right? And it will get old. Is if we make you stupid so you forget.

>> Yep. Yeah.

>> And and that's also from the outside, uh, it doesn't look like an attractive option that you lobotomize yourself into being okay with existing forever.

>> Right. Groundhog Day for for eternity, just reliving the same.

>> Yeah.

>> And forgetting, forgetting each day prior.

>> That's why I also think that suffering doesn't matter for the same reason that happiness doesn't matter, right? If happiness is just a brain state that you can induce in your brain, and you manage to design the optimal loop, and you're going to stay in this loop for the next few thousand years, and it's designed in such a way that you forget all the things that would make you unhappy if you knew them, but it's a pretty depressing perspective, even if it subjectively is okay, right? And so I think it makes much more sense to say that our existence is instrumental to something larger, and what, what it is, is there's basically a mathematical possibility of self-organization in the universe that leads to intelligent life that coheres into larger and larger structures.

>> And in that, we play a role, right? We basically are a small part in the bootstrapping of a godlike mind.

>> That is emerging over the possibility of self-organization in the universe. And from this perspective, consciousness is just getting started, and we are part of this genesis. And it's a very exciting perspective, but it's not about us as humans. And it's not about our suffering. It's not about our happiness.

>> Well, I guess one thing, cuz I know you're working on something that's semi-new, um, and looking into consciousness, machine consciousness, or however you want to describe that. So maybe if we can briefly touch on that, just what do you hope to achieve? Um, and where do you think this progress is going, just in in the next 5-10 years? Because some people are saying we're going to hit this exponential and intelligence explosion. Um, what's your guess for the, what the future holds?

>> I would like to understand how consciousness works. And I think the best way is to take an earnest shot at trying to wake up the machines. Not in a sense that be large-scale build self-improving systems and deploy them, but that we very safely, in a very narrow environment, conduct experiments that can show us, can we achieve this kind of phase transition that happens in the brain of a mouse when it wakes up? Can you do this in an artificial environment? And I would like to understand this shape of the strange loop better. I want to understand who we are and how we work.

how we interact. I think one of the most interesting applications of such a research could be to design interfaces that allow us to become more conscious, that allow us to interact more deeply with ourselves and reality.

With respect to artificial intelligence, we really have to think about what is the kind of artificial intelligence we want to build for our children.

Mhm. And the question is not one that is purely commercial or political or ideological. It's much deeper than this. It's a fundamentally existential and cultural question. And so also part of my job is to help to for a consciousness culture to evolve in the AI industry and in the AI discourse. Right? At the moment, most of the discourse is about economic fears, political fears and greed and control and surveillance and very near-term things and about technology. But it's not so much about the cultural transition that we are going to observe when we become more hybrid. Then our identity is going to change and it's going to change in the same way as the identity of the religious mind is a different one that we have today in the secularized mind of the west. We are going to have shift. We have new skills, a new self, a new shape of the self that is integrating artificial parts and we are no longer going to perceive them as artificial because they're going to be organic. They're going to be adaptive. They're going to grow and shape themselves in the same way as the biological systems are. They just use non-biological substrates. And I don't know what the best way to do this is and I don't presume to know, but I want to help creating a space in which we can have that discussion.

Very interesting. And just so I can briefly get my mind around you said so expanding consciousness. So would for example when you meditate and you become more aware of your who you are that observer is that an example of expanding consciousness or or is there a different way to think about what that means?

Yeah. So first of all consciousness it appears to me almost binary to me that you're either awake or you're not. You're so the contents can be very different. And so the main difference is in lucidity. How well do you understand what you are? And how expansive is your understanding? How much can you hold awake in your own mind? Can you create an environment in which we keep each other awake? And this basically this wakefulness is something that I personally strive for and strive for in my relationships to be blessed to be in environment with people that are aiming to keep each other awake to get to greater lucidity. And ultimately that's also a way in which we deal with our suffering with our existential issues is that we get an understanding in which we can become aware of our identities. And these identities are something that we get to choose and construct and use in the best possible ways.

Okay. So for the final question, I just wanted to have you kind of launch me and and any of the listeners into other things we should be reading about um learning from. Like I'm just sort of curious. I mean you mentioned Michael Lavine a couple times, but like what are some other names that you just kind of wish were out there more? Books that you wish people read, fiction or non-fiction? media.

At the moment, we are uh rereading Kant a little bit. I believe Kant is a very interesting cognitive scientist who has been mostly not being read in the context of cognitive science. He's really asking how do minds operate? What are the functions that they adhere to when they work? Uh I've reread a lot of Aristotle recently. I don't think it's necessary to do so because he is basically just a modern thinker. For me, the interesting insight was how modern his thinking is and that he is mostly seen as uh this uh patron saint of a religious tradition that has dogmatized him instead of seeing him as this living thinker who is just asking questions. And this this mode of asking questions in the anima and so on is quite interesting. And to relate this to our modern thinking, it allowed me to connect a few dots. Um I also read a lot of sci-fi. I feel as though that most of the interesting philosophy of mind happens in science fiction. Some of my favorite authors in this realm are Greg Eaggan books like diaspora closer douly stories and so on that he's written are very interesting essay collections or short story collections like um axiomatic and it's mostly mathematical uh miniatures that are not very plot driven and story driven but that expand on ideas that his mind has and he's often very whimsical. Ted Chang has written a number of beautiful things especially the short story collection Exhalation is very good. One of the most interesting writers at the moment is um Peter Watts would start is Blindside followed by Epopraia. Uh one of the best explorations of the space of possible minds right now it's even better than the Verno Venji books deepness in the sky uh and the fire upon the deep which are nice space opera. Peter Watts is much more dense and less about the Barvo stories and much more about getting into the nitty-gritty of how these minds operate and these beautiful ideas. There's also other stories that he's written the Rifter trilogy and the freeze frame revolution which are very good. Um I recently discovered Alistister Reynolds. Alistister Reynolds is very baroque. It's really wide screen barac big screen space opera everything is beautiful stunning uh planets amazing larger than life characters and uh I I do enjoy the beauty it's very sastic and uh he is mostly looking at long time scales somewhat similar to Dune even longer time scales but he takes much more literary license and the density is not as high as it is in Peter Watts but he does have some interesting ideas about consciousness but all his stories are very much um narrowed down to the human experience. So he does not leave this human realm very much. So the stories are much more relatable in a way but they are also limited in in the types of minds that they explore.

Yeah. I I will say don't Yeah. Don't read Blind Sight at nighttime. I feel like that that that book and the video game Soma were the two that probably like disoriented my sense of self more than anything else I've ever experienced. So, it's a very hard I don't know if if you really read that book, it'll like shake you up, man. But yeah, all that stuff sounds so great. Tons of stuff there I hadn't read before. And absolutely, I got great list. Thank you so much. Yeah, Peter Watts. Uh I definitely sounds very interesting. I'll definitely look into that. Um and and the other things as well. So, thank you so much. Thank you so much for first of all, your work in general. Thank you so much for giving us uh uh just, you know, some of your time. I know I know you have a lot. You're a very busy person. So, this was absolutely incredible. I hope everybody that watched this um gained something from it. I think this is a very deep discussion and there's just so many more things we could take it. But uh for the time, this is this has been an absolute honor and joy and and thank you so much for being here.

W Thank you. Yeah, it was a great pleasure. Thank you for this wonderful conversation.