This Terrifying Ideology has Infiltrated AI Labs | AI Expert Brendan McCord

People who are really high up at the AI labs, they say that we are being rebels, foolish rebels, if we don't listen to the AI. They, they truly say this. "How dare you rebel against God?" Right. "Exactly. How dare you, you silly, silly rebel."

Nick Bostrom is correct that philosophy is on a deadline. He is dead wrong about this essential, maybe central, human good. Two years from now, AGI is going to come around the pipeline. You wouldn't be doing what you're what you're doing now, right? There, there's more pressing questions to resolve.

Yes. Wrong. People right now believe that AI should tell them what to do. There are "Claude boys," literally teenagers who call themselves Claude Boys, who will wake up and they will do what Claude says and not otherwise. And this, not EA, not existential risk. This is the true challenge that's ahead of us.

One of my friends uses ChatGPT for hours every day, not just as a search engine, but as an operating system for his life. He asks it where he should eat, what he should text girls on dating apps. He gets up every day and has ChatGPT tell him what to do. My friend does this not because he's incompetent or stupid. He's one of the smartest people I know. But because ChatGPT already knows so much about him that the advice is actually getting quite good. The restaurants that it recommends, for example, are already better than the ones that he can find for himself.

My friend is not alone. Gen Z, Gen Alpha, are increasingly using AI as a holistic operating system to which they offload all of their decisions onto. And Brendan McCord argues that this kind of offloading is the real danger of AI that no one is talking about. In this interview, you'll learn why human autonomy is important, how AI threatens it, and how to harness the power of AI without forming an unhealthy dependency. My name is Jonathan B. Brendan, and I run Cosmos together to deliver educational programs, fund research, and build AI startups that enhance human flourishing. Both of us have a background in philosophy and engineering. And we believe that it's important to combine the two if we are to build actually good AI systems. If you want to join our ecosystem of philosopher builders, you can find roles we're hiring for, events we're hosting, and other ways to get involved on jonathanb.com/cosmos. Without further ado, Brendan McCord.

So, Brendan, today we're going to talk about autonomy, which is, uh, a risk, but also an opportunity that most people who are building AI, even for their entire lives, are not really focused on. Um, but first, I want to clear the ground and talk about what people are worried about. So, there are very mature and developed schools of philosophy in the Valley. The accelerationists, the effective altruists, uh, the x-risk people, and you argue that they're all missing something essential about AI development. Why is that? Uh, and give us an overview of these goals.

>> So, the two main tendencies that I found within the AI community are the existential pessimists and the accelerationists. And they roughly map to the extremes of despair and the extreme of hope. The existential pessimist is kind of three philosophies in a trench coat. It's rationalism, effective altruism, long-termism. These are all distinct, but they're the intellectual incumbent. And the prescription in light of this possibility is that we should pause development, is that we should centralize control, is that we should radically remake society on the basis of risk avoidance. So, it's a claim and then it's a series of drastic and dystopian prescriptions. You see some of the early, um, godfathers of AI, as they're called, people like Jeff Hinton, like having this almost Oppenheimer-like hubristic awe of their own creation, saying, "I wish I hadn't done it." >> "And you know, I brought this into the world." >> "And, um, you know, the three schools, as I mentioned, effective altruism, rationalism, long-termism, it's worth breaking it down a little bit. So, rationalism comes out of the 2000s. Eliezer Yudkowsky, um, uh, and Slate Star Codex were kind of the originating sources of this. And they claim to be focused on an idea of like, um, uh, perfecting human rationality. But they have a very, very narrow commitment to what rationality is. It's Bayesian updating over a value function or the value function over outcomes. It's a kind of rationality that Aristotle and Kant and, you know, modern thinkers would not recognize. So, it's a commitment to a very myopic kind of probabilistic control type of morality. And what I find interesting about this is that the school is committed to rationality. Yet, it finds itself the most fertile breeding ground for the extremes of hope and despair. And there's some kind of irony to that. You know, the next one is long-termism. And Bostrom is probably the source in long-termism. Long-termism is worried about anything that could affect the long-term outcomes for humanity. That could be an asteroid. It could be, um, you know, a pathogen. It doesn't have to be AI. AI is particularly important because its MO is that of control. Bostrom advances something called the orthogonality thesis, where you say something can become very intelligent but on a totally different axis. That does not imply that they become more moral or that it becomes more controllable. This is true of humans as well. Um, and he worries that a non-anthropomorphic final goal, meaning "create more paper clips," will cause AI to pulp us all. So, this is a concern that he has for the long-term future of humanity based on this orthogonality notion. And then the last is effective altruism. And effective altruism dominates the three when it comes to the fundraising potential. It is the most highly adopted academic theory since Marxism. M. And what effective altruism does is it rationalizes certain moral intuitions and things that are not intuitive at all. It tries to pull in as well things like, you know, other-regarding behavior should not factor in any notion of time or or, um, place. So, so what that means is I should care about my family as much as someone who lives 17 generations hence in Indonesia, or even plausibly a shrimp, equally. A util is a util whether it is yours or those other entities. That is a very radical premise. Um, it also attempts, as, as you know, these forms of utilitarianism do, to reduce moral questions to a single currency such that we can compute them and maximize them. And so, um, I reject, and I think most thinking people do reject, that morality is a thing that works like that. In other words, endemic immorality appears to be this difficulty of making pretty sharply heterogeneous trade-offs. Like, how do you compare the, uh, moral question entailed by some suffering over here with familial love, with duty, or honor in battle? These are things that don't commensurate in utilitarianisms. They do, and it's a false kind of scientistic way of viewing the questions of morality that really obviates what makes moral choice difficult. The other thing I'll say here is that, um, effective altruism tries to take one's personal projects and make them those of the universe, provide the calculational standpoint to say, rather than, you know, acting morally in a way that inherently springs from me, right? That's what we learn from Aristotle and the Nicomachean Ethics. Rather than doing that, it subordinates that impulse, that moral choice is like an individual thing, and tries to make it the standpoint of the universe, subordinates, uh, what it means to act morally. And this is why it is not surprising to me to see such profound moral failings in the EA community. In other words, what is sexual assault? What, what is the, what is the real issue there if you're doing cosmically significant work?

What are the practical consequences of following this through, of people who are actually building AI or regulating AI thinking in this way? What are the practical consequences?

>> So, two that come to mind. One is that we would look to governance solutions that are profoundly illiberal and that would lead to tyranny, um, namely the creation of a world state. So, in the paper, um, "The Vulnerable World Hypothesis," Bostrom writes about the need, therefore, you know, it's always a conclusion of these radical premises, to create a world state where basically you eliminate the coordination difficulty of having greater than one actor. So, even US and China, let alone a more multipolar, you know, world, presents this like game-theoretic challenge of racing and so forth. And so we have to get to one. We have to get to one state. And of course, political philosophy has dealt with the question of whether there should be one state many times before. There's a Kantian critique, um, that has to do with, you know, the challenges of maintaining legitimacy, uh, versus having to sort of squash dissent. And there's a Strauss-Co debate where Leo Strauss argues that it would lead to a universal perpetual homogeneous tyranny, and, um, and many other, uh, concerns. So, that's one. The other is that if we accept and hold tight those utilitarian premises, then we might think that it is, uh, acceptable to live a life with artificial intelligence that is profoundly, uh, non-humanistic. In other words, that we might, um, be perfectly willing to be downstream of AI, having it tell us what to do and guide our life like an autocomplete.

>> I see. Um, one interesting thing I want to double-click on is you use the word hubris to describe these schools, which is I think is fascinating because in many ways these schools portray themselves to be egoless, right? The effective altruist is about, it's not about my good, it's about altruism. Or the rationalist, right? It's not about my ego or my selfish interest, it's about doing what's rational. But I was reading, um, one of these, uh, important blogs of the EA community, and it said the way you should make decisions is think, "What would a benevolent, omnipotent God do?" And I think it's, it's kind of a power trip, right? To to think about it in that way. And maybe to build off of what you said about Oppenheimer, I think that to claim confession of one's guilt is at once to claim credit for the sin. It's to say, "I am powerful enough to potentially end the world." And there's also a subtle ego trip there. Is that, is that what you meant by hubris?

>> Yeah. And I think it, it's both the kind of hubris that Bacon had in saying that man would master nature, you know, at the beginning of the scientific project, and the same kind of hubris that comes through any serious philosophy project where we try to find the one true answer to all the things. Like, there is a hubris laden in that. And if you are the type of person that can subjugate the world with symbols, mathematically inclined among us, it is your moment, and what you do has cosmic significance for the future of the race.

>> Right. Now, I will say the strands of philosophy to which I am attracted tend to take a different approach. They tend to be more epistemically humble. They tend to focus on the use of reason to whittle down the claims of reason, as David Hume would say. In other words, a critical rationalism, and not a constructive rationalism that tries to sort of say, "This is the way the world is." Tries to focus a little bit more on what the world isn't.

>> Right?

>> Um, so that there is, you know, philosophy is not, uh, not solely the domain of the hubristic.

>> Right. Uh, one of my favorite lines from, uh, your favorite book from by Hayek, which we'll talk about, is something like, "Every political philosophy believes other people are ignorant, but liberals also believe that they themselves are ignorant." Right? And that's kind of the difference there. Um, many in Silicon Valley have also reacted to these dominant schools of philosophies in the way that you have. Uh, and so accelerationism was born. Tell us about that school and why you don't think that's a satisfactory response.

>> Right. Right. So, the, the other end of the extremes of hope and despair, accelerationism. And this school would have us unleash the development of AI as an end in itself. And that's a very important idea. You know, you and I, I think, probably hold technology to be a very powerful means by which we flourish as individuals. The accelerationist school confuses technology as a means with technology as the end and views humans as being a kind of instrument in this trajectory that is sweeping, transcendent, inexorable, this technological trajectory. It draws a lot not from Bayesian thinking, but from thermodynamics. And so it wants to portray humans as a kind of variable in this thermodynamic equation, in this, um, broad project to be able to harness and dissipate more energy, or to be able to, uh, climb the Kardashev scale, which means to harness not just the energy of the planet or the sun, but of the galaxy, you know, ratchet up more and more and more. These are not humanistic goals. And it's very important because one of the conclusions that the orthodox accelerationists reach is actually that we can and should hasten the "passing the baton" >> from humans. "We invite the doom to something higher." Yeah. Exactly. And I think with both schools, but certainly with accelerationism, you see a metaphysical impulse, you know, the, the, the religious impulse that humans have long had. You see it conserved but redirected towards the thermodynamic god. In other words, they're both kind of eschatological. You know, in one case, we die, and in one case, we build something higher and better and then are transcended. And here, here's the upshot: is that both of these schools are very imaginative in a sense, right? These are, you cannot fault them for being very imaginative about the possibilities here. Um, whether it's paper clips or thermodynamic god, but on the one thing that is most needful, they have a lack of imagination. And what I mean by that is on what it means to be human, what the human good is. Yeah.

>> One side views humans as a kind of aggregate of suffering or pleasure. In other words, we view the human good as the avoidance of pain. I have news for you. That is not the entirety of the human good. The other side abandons the idea of the human good entirely. In other words, it's a non-humanistic philosophy. And this is precisely where the issue is, is that we need a positive, uh, uh, uh, approach that is humanistic, that is grounded in the real goods, not reductionist to, to, you know, either Bayesian or thermodynamic effect, but really focus on the underlying human goods. If you are, as we are investing in, for example, application-level companies that are using AI, if you're worried about solving problems around autonomy, um, and we'll discuss what that means, you are implicitly making a bet and saying that there will no be there there won't be a singularity or an AGI within the next five to ten years, right?

>> You're saying that there is a human good that needs to be jealously defended. That's what you're saying. And you're saying that no one else will do it. We have to do it.

>> So, you're, you're but you're agnostic to the, the speed of the, the AI development.

>> You're agnostic to the development scenarios. You're simply saying, whatever path we go down, this is a human good that I want to protect, and I must build in order to protect it.

>> Let me push you in in one last direction as a devil's advocate for these schools, which is, Brendan, you say that, uh, you are agnostic to the AI kind of development timeline. Uh, and the core focus is figuring out how AI, regardless of development timeline, can help these human goods flourish. And that's why, for example, the main questions we ask at Cosmos are around autonomy, which we'll talk about soon, decentralization, and truth-seeking. The devil's advocate is to say, but if you think AGI is around the corner, then all of your energy should be focused on getting that right and aligning it, and it will be able to answer these questions much better than we can today. And so this is what Bostrom said when I was interviewing him, which is, "Philosophy has a deadline." And there are certain philosophical questions that are more urgent if you take that AGI is around the corner kind of kind of idea. So even though you claim you're agnostic, you're still not agnostic, right? Because if I told you, let's say I'm an oracle, I come from the future. In two years from now, AGI is going to come around the pipeline. You wouldn't be doing what you're what you're doing now, right? These questions we can save for later. There's more pressing questions to resolve.

>> Yes.

>> Ron. Yeah. So I think, you know, for example, um, when we talk, when we get into autonomy, that is a lived practice. In other words, that is something that humans do, right? And so the thought that we will build AGI and AGI will figure out autonomy is nonsensical. It's a category error. Like, it can't figure out something that must live within us. In other words, we must self-develop. We must self-direct. And so we can use AGI instrumentally in that pursuit, but it is not a thing to be figured out. It's a practice. It's something to be lived.

>> I see.

>> And so, like, I also want to, um, clarify that my agnosticism doesn't imply a withdrawal. In other words, we are building, like, we're building the future we want to see.

>> Okay. Well, this is a perfect segue. Let's talk about the risk that I don't think anyone in the Valley has really focused on in the way that we have: autonomy. What is the risk, as well as opportunity, as it relates to AI and human autonomy?

>> So, when you think about the greatest goods in your life, you probably think about things like friends and family and loved ones. You might think about the pursuit of wisdom. That's, I would say, one of your highest, highest goods. Creative endeavor, that sort of thing, right?

>> Um, it's actually eating, but wisdom is number two.

>> Yeah.

>> Eating. Exactly. That would I, I would say be one of the lowest. That is necessary for the highest. So, this is an interesting point about how the lowest things in us are needed for the highest, uh, as well. But these kinds of goods, what's common to them? Whatever you hold to be your highest goods is that they cannot be, uh, obtained on a platter. You know, AI can't give them,

>> eating, except for eating, which we'll, I'll hold to the side.

>> We'll buck in. Um, and they have to be, uh, attained as the result of some kind of self-motivated striving. In other words, you had to get out there. You had to try things. You had to enjoy it. You had to experience it. You had to get hurt by it. Being able to discover and develop one's gifts, being able to deliberate using reason and to line up our actions and be able to pursue them. So, this deliberative capacity for self-direction, okay, is I think the thing I want to call everyone's attention to. This is autonomy. And without this self-direction, we cease to live fully human lives. We may act in the world, but it isn't really our life to live. And so I say this all so far at the level of the individual. Like, the other piece of it is that it's very important as a society, particularly a democratic society, one that self-governs. And this is because we depend on individuals who can form views, who can act on those views, who can self-govern. Without that, we lose the, the greatest bulwark against despotism.

>> Right. So, draw it out practically, how, uh, this current AI wave can harm or, or accelerate us being autonomous agents.

>> Okay. So, the phrase that I would, uh, stick in your mind is "autocomplete for life." What I mean by that is we use AI systems, we obtain the incremental convenience from them where we get not just the next word in the sentence, that's what everyone's familiar with of autocomplete, but also the next decision, the next, you know, uh, uh, job recommendation, the next friend, the next relationship, the next purpose. In other words, we can sort of ladder up what AI can do for us, and it feels very harmless. It feels convenient and probably useful, but it adds up. It it causes a kind of erosion of choice. When we offload, we can see at the level of fMRI, but certainly we all recognize this in our lived experience, that we atrophy. In other words, we, um, we habitually offload in a way that causes us to then lose the skill. You see this, like the fMRI stuff is like, if you do a lot of speed reading and not a lot of deep reading, you lose some of the ability to do that. You know, or if you do a lot of calculator-based arithmetic, you lose the ability to do that. Now, I think an important point has to be made, like, why is this not just another version of that? Right? Google, Google Maps. Right? I can't drive very well without Google Maps. You're probably the same. Actually, live in New York, so you probably can't drive, period. But, so, then the next question becomes, why is AI in particular a problem for this?

>> Because this is a problem for all technology. Books and memory, right? Like driving and being able to ride a horse. Yeah. What, what's,

>> Well, actually, I think before we even talk about that, I framed it as a problem, but it's actually, it's actually a beautiful thing,

>> right? And you and I have talked about the quote from Alfred North Whitehead that "the measure of civilizational progress is the number of important operations of thought we can perform without thinking about them."

>> Right?

>> It's a brilliant, brilliant quote. There are examples about my favorite of this is Max Verstappen, the Formula 1 driver. He's a kind of a prodigy, you know, special driver. And when it's raining on the course, he can talk to his pit wall, his pit crew. And it's because he has made it autonomic. Yeah.

>> He has done it so many times that he can actually think about strategy. He's talking about the tires. Yeah. You know, he's going 220 miles. In fact, I remember one of the races, I'm a big F1 fan. He was watching the other drivers like on the jumbotron and commenting on their races.

>> I mean, it's almost like he's just, yeah, he's just like having a strategy thought while everyone else is clinging on at 5 Gs or whatever they drive at. And so, anyway, so it's a beautiful thing. It's how we build the edifice of civilization. It's how we do the higher things. It's great. So, it's a bit of a paradox, right? On the one hand, it's great. On the other hand, it's problematic. Okay, so now we set sort of the contours. So, coming back to the key question of why does AI, why is that a, a thing that's special case, special case, yeah?

>> So, one is, you have to think about what it is that you are offloading and thus potentially eroding. And in the case of calculators, it's calculation. In the case of maps, it's like positioning in, in space, yeah, navigation. That's,

>> um, in the case of writing, that's, is essentially memory, right? Primarily memory. And so these are kinds of categories of, Never before has it been possible to offload and therefore atrophy our core practical wisdom, or our core deliberation, I should say, that leads to a kind of wisdom. So, you know, now you are talking about something that is necessary to self-direct, that is necessary for moral judgment, right? It is necessary for us to decide what is good for us. And so, it's a different kind of thing that gets offloaded. The other question you have to, you have to ask, okay, now that we've pinpointed that this is a very precious thing that we should be careful to not offload, then you have to look at how pervasive is it likely to be? And AI is clearly something that can scale. It's clearly something that can be hyper-personalized. Already, 20% of human life is mediated by algorithms. Human waking life is mediated by algorithms.

>> Social media algorithms, yeah.

>> Social media algorithmic feeds, not just LLMs, but AI that, um, determines or guides or shapes what information reaches your mind, what thoughts therefore form within them. And so the scale is already very significant. But what that means is that you might not encounter different possibilities. In other words, you might not realize that there's something else out there if you have been sort of epistemically narrowed to a high degree. I think another mechanism that's important is how do you sort of pull out, how do you recover, right? In the case of calculators, you can just do kind of the inverse operation and check it. In the case of a lot of the things AI does for us, it's very hard to check and it seems authoritative. You know, it can answer questions like, "What is justice?" Like, no one knows, but you know, you give us that kind of question. And if, if AI seems authoritative and fast, the computational cost of checking it is very high, we don't check it. This is a common automation issue. We don't check it. So that fact, combined with the narrowing, destroys the possibility of error correction in the long term.

>> So, so let me summarize the conversation so far for our audience, which is that all technology, as they give you a superpower with their left hand, takes something away through dependence on the right hand. Um, this trade-off is worth it if what is taken away isn't so central. What makes AI special, your view, is it sounds like, is that because it's the most, uh, it's a technology that's most similar to humans, that it can potentially take away practical deliberation. So, it's the thing that you lose. It's, you know, practical reason versus navigation versus calculations versus memory, as well as the scope, uh, with which you lose it, because it can be embedded in everything. That's what makes this extremely dangerous, and the fact that it is hard to recover because the way you might audit that, the way you might pull it out, is through use of the very same thing that it atrophies.

>> I see. For example,

>> Right? And so obviously, I imagine people are going to be curious about the solution. What do we do about this? But before we go there, uh, I want to better understand the concern by giving you a thought experiment. Okay? And I, I call this thought experiment the, uh, the omniscient autocomplete. Let's say, whatever practical question it gives you, it'll always give you the best answer for you to do. Should I marry Sally? Should I marry Susan? Uh, and you know, you know that it's the best because historically it's been verified. Okay. So, so you run back tests, it always gives the right answer. And every little thing that you've tested, and that your friends have tested, it always seems to have have given you the right answer. So, you're, you're pretty confident empirically that it's, you know, it gives you always the right practical answer. How would you use this system?

>> I want to accept the thought experiment, but, but first, I want to understand, is it, um, omniscient through time? Like, is it, is it an oracle that, so, like, in the middle of the way through an NBA season, we don't know who wins the championship, right? Does this omniscient oracle know that? And I, I raised this because there is a class of knowledge that can only be, that's not sort of like computationally reducible. It's something that only is generated through actually playing out the thing, you know, right? And this is, for example, um, how markets function is we, you know, across many different variables, we generate knowledge. It's knowledge that is known to no one because it isn't even in existence in the world. So, let's say that it's not omniscient through time.

>> Um, but let's say,

>> Let me put this way, the wisest human ever, let's say Socrates, it would make the decision that they would make. Like, like, given imperfect information, date Sally, date Susan. We don't know if Sally or Susan has cancer, but given all the the information, it makes, like, the best possible answer.

>> Yeah.

>> The reason I asked the question is because, um, there is a mode of operation that is suitable to exploiting knowledge then known. Yeah. In other words, like, kind of what an authoritarian government does, right? There is quite another that is suited to the maximal generation of knowledge, right? And, um, for things that are not known and won't be known except by sort of playing it out,

>> right?

>> And so if you lean too hard on the former, uh, on the exploitation of the knowledge then known, you kind of deplete the stock, right? You cease to generate the new. And I would argue that the real goal from the consequentialist frame is we should want systems that can, uh, allow the anonymous individual to achieve his or her unknown ends. And if we want to do that, it is not by simply exploiting the knowledge of the unknown. It is by maximally enticing the use of knowledge.

>> Right? But, but, but the AI could, could tell you to do that. So the AI could say, you know, given what, what you know now about Sally and Susan, go with Susan, but be open to it. And then ask you questions while you're dating Susan, you know, every time you ask it a question, it will come out with the, with the best kind of practical, uh, deliberation. Yeah.

>> Yeah. So, so how would you use it? Would you have like a pair of VR goggles that it always tells you what to do? Would you never consult it? Would you consult it occasionally?

>> M mhm. So, I have a four-year-old and a six-year-old, and I kind of am raising them with the idea that this world is the world in which they are entering.

>> Right.

>> Okay. And so what I've done is on the one hand, try to utilize that oracle to develop their skills. So that, Tilos, is a self-development one. And so my daughter will do math that AI could trivially answer, but she will still do it. So AI poses questions, and she does the math, and this works pretty well. But then I time-limit that very strictly. And this is through a curriculum at Alpha School, which you know, um, in based in Austin. So, there's a time delimiting. Then for the remainder of the day, there is an experiential learning component that is completely without this oracle. In other words, she goes outside, she tries things in the world. She learns to ride a bike for five miles without stopping. She climbs a rock wall. She speaks in front of a hundred people. Okay. So, there's a non-oracle component. There can be light consultation for, like, if I want to learn how to garden, how do I do that? That's a consultation that I would rely on the oracle to tell me how to do. The, the last component is then, um, stimulating through probably human discussion, the kind of characteristics and habits of mind that are necessary to retain self-direction in a world like that. Because my biggest concern would not be correctness. Correctness is solved, but it would be the concern around enfeeblement, around not living a full human life because I no longer self-direct, because I become a sheep. And so, what do I mean by that? Well, I want to be able to cultivate a reflective metacognition that says, "What am I versus what is the pull of the algorithm?" In other words, if this is an ecosystem around me that I'm using quite regularly, I sure as heck want to know what I am, right? This is an extended part of my mind, um, that I endorse, that I transparently use, but I still need to know what my boundary is so I don't get lost in it.

>> Right. We need to know how to think in connection with machines that could do the thinking for us.

>> Right? So, let me ask you this. One way to frame your answer is to say, "I will consult your oracle, but I always need to make sure that if I follow its directions, I need to know the steps. I need it to tell me the full reasons." Is that, is that fair?

>> Uh, I think it's more than that. I want, I need to be able to exercise my deliberative capacity. In other words, knowing the reasons. It is not enough to just know them, right? I mean, this is the Mino, right? This is the, the idea that we need to have the statue of Daedalus, I think, you know, tied down. And the only way we can, uh, avoid having this thought run away is through working through it, giving an account, as Plato would say. And so, um, I, I need not just to know the reasons, but I also need to be able to work through them,

>> um, myself,

>> if that makes sense. Yeah. So, and the Tilos is really important as well, because there's a lot you can do. Like Rousseau wrote in Emile about how to tutor, how to raise an autonomous child, right? And this gives us, um, in the end, um, but one thing he does there is he configures the environment for the boy in a way that is very, I don't know, paternalistic, like controlling that environment, but there's a progressive, uh, letting go because the end goal is self-development. With an AI system, there's no end goal of self-development. You know, that oracle doesn't really care about that. That oracle would just assume you'd be, um, perpetually dependent and that you have a habit of passivity and you just do what it tells you to do, right?

>> So, it's crucial that you set that goal,

>> right?

>> of of self-development and not of, you know, unthinking dependence.

>> Yeah. So, um, I agree with much of what you said. One being that the shape of a good life is to be self-directed, right? So, so even if you're doing, you're making all the right choices, dating Sally and not Susan, if you're not feeling in the driver's seat, if you live your entire life like that, that in itself, the form of that, even beyond the content of your decisions, robs you of the good life, ipso facto, right? That, that's what you're getting at. So, I would definitely not use this Oracle in like VR glasses mode where like it would always just tell me to raise your left hand, raise your right hand, because like, and I would gladly trade off suboptimal decisions for the fact that I'm making the, the decisions.

>> Yeah.

>> But here's, here's why I want to challenge you. What if the Oracle is so advanced that we can't even understand its deliberations? And before this sounds too dystopian, I'll draw an analogy to religion, right? Um, Dante, when he goes to paradise, and he asks the eagle of justice, "Why is the poor virtuous pagan who lived before Christ, who never even had a chance to see Christ, why does he deserve to go to hell?" The eagle of justice says, "None of your damn business."

>> Yeah.

>> And in the next canto, the eagle of justice says, "I'm the eagle of justice. I don't even know. That's God." Okay. I, I just, I delight. These are almost exact words. "I delight in following God's will." The structure of faith, as I take it, is that you, uh, try to validate it as much as you can. You know, is it plausible that Jesus rose from the dead? Are, are the accounts accurate? But after you validate the oracle, the religion, the god, to your rational faculties, you, as a leap of faith, are willing to, um, take actions even if you cannot see the full reasons of those actions. So, if you now transplant that to a kind of AI oracle, I imagine you wouldn't be comfortable with that. And so, and so my question would be, do you think faith itself, and do you think faith and religion is just a deficient way of living the human life, even now?

>> That's a big question. So, the most beautiful part of the Bible for me is the end of Job, book of Job, where,

>> much the same as the eagle of justice, you have,

>> um,

>> yeah, where were you when I was building the cosmos? Right? What God says,

>> yeah, exactly.

>> And these harms have befallen Job, and it's this beautiful poetic explanation of the limits of human, um, capacity to understand God. Now, in that case, God looks to us as a, uh, from the standpoint of some form of self-development. In other words, like, um, we have a relationship with God in which we self, in which we develop. That doesn't exist in the case of the omniscient AI.

>> Um, and so it's much more likely to form a kind of passive relationship. I think it's a close, it's a close call because in the, because we do sort of like try to give ourselves up to Jesus. "My life is not my own." This is what they say.

>> Right.

>> Right. But I think the fact that, like, um, there is some reciprocal expectation,

>> not to say we're equal to God, but like, there's some reciprocity,

>> sets a slightly different frame. We also choose, in a real sense,

>> right?

>> To, uh, engage in religion and like, to, to sort of legislate upon ourselves that ultimate question. And my concern with the habituation mode of AI is we may cease to choose. Like, we may unknowingly bind ourselves to a life of dependence where we no longer are choosers.

>> Right. I see. I mean, it's not just religion, right? What I'm trying to highlight is that there's an entire sphere of human activity that I, I mean, maybe you'll consider it all to be deficient, but, but many serious philosophical traditions don't, that has as an epistemic mode, following the advice of an authority whose legitimacy, kindness, accuracy, and truth you have empirical reason to believe in, even if you can't understand this specific piece of advice.

>> Right.

>> Faith is one of them. I gave,

>> yeah.

>> Maybe the military is, is another one, right? Where you don't know the full reason that, that you are being given orders. Uh, arranged marriage might be another one.

>> Uh, parental relationships. That's a bit different because you're a kid there.

>> But, but do you see, like, what I'm trying to highlight, that there's an entire sphere of human activity where you don't trust and reason yourself on the advice itself.

>> Yeah.

>> But you evaluate the deliberator. You value the, the advice giver.

>> Well, so I think, you know, were we to be first-principal reasoners about everything, I think chaos would ensue. I think we actually have to unthinkingly accept quite a lot and not just to be religious, but to function in society. I think we do that.

>> Um, that is also the way in which I think we should relate to tradition. In other words, we should not use reason to create tradition or to, to scrap tradition anew. This is where I think Mill gets into some trouble and where Hayek's Burkean reverence comes into play in a better way. Okay. So, there's a conservative strand that I very much, um, embrace as it pertains to the epistemic value of tradition. But, um, so that's one point is like, I totally agree with that. The, the point about, um, the military, right? I was in the military. I think Kant has a good framework for thinking about the, the laws that we give ourselves. I mean, he's talking about the moral law and the categorical imperative, but more broadly, he's talking about the idea that we can,

>> restrict ourselves.

>> Yeah. We can be autonomous and not heteronomous in so far as we rationally choose to put something on ourselves. So, if I choose to join the military, and then in the military, no longer get to choose, I have to follow orders. That's completely okay. That's fine. That's autonomous. And in fact,

>> I can exit the military. I can hold the officer above me to account through court martial.

>> And so it seems to matter very much whether we choose to, um, to, to willingly sacrifice our autonomy. I'm trying to think of other things, like, you know, we choose to do jury duty. We choose to do a lot of things that, you know, sacrifice that.

>> Yeah. So, so just to be clear, not talking about AI yet, talking about, like, human interaction, pre-AI. It sounds like you're fine with evaluating the legitimacy, let's call it, of an authority, outsourcing partial deliberation to that authority, meaning, uh, following certain kinds of orders from that authority without understanding the full reasons, if you maintain the ability to evaluate it, if not fully, and the ability to exit. Right? Therefore, let's just transplant that exact structure that you talked about, the military, onto AI. Would you be comfortable with that? Again, this is why I set it up to be the omniscient autocomplete, right? The oracle that, that given every test you've thrown in it, it's given you the best practical decision. So, in this case, would you be comfortable it just telling you, "Date Sally or Susan"? And it says, "Date Sally," and you say, "Explain to me the reasons," and it explains to you some reasons, but you still can't understand the full picture. Just like the eagle of justice, would you be okay outsourcing that decision, decisions like that to such an AI? Because, like, your reason, right, is kind of perfected. Your reason is as good as it can get. But again, I'm just assuming that there's a limitation to to human reason, then that AI potentially can be higher. And so, if you're, if you're as, as good as you can get reason-wise, and you have the ability to to exit, are you okay following that order?

>> I think I would, I think on a case-by-case basis, I would be willing to do it. I would, I hesitate a little bit on marrying Sally. Um, one because I'm married to Adrien, but, um, no, but the other is there's a, um, there's a critique of utilitarianism that Bernard Williams brings. Um, it's called "One Thought Too Many." And he essentially poses this scenario in which you have, um, someone about to drown in a rescue situation. And what he says is that though the philosophers might want you to do a calculation, you know, um, what you should do is go with your moral intuition about that. You should just act, you know, in other words, you shouldn't actually run a calculation of any sort.

>> And while in this case, I'm not running a calculation, I'm just kind of deferring.

>> I think on questions of love,

>> yeah,

>> I would, I would, I think it would be one thought too many to have an external agent. But, but this is how arranged marriages in all the, like, the pre-modern societies have worked. And, okay, and maybe if, if love is the tripping factor. Let's say starting a company.

>> Yeah.

>> Like, like, should I start company A or company B? Let's, let's take love because I, I understand it's more subjective. Would you be comfortable outsourcing that decision if you were able to evaluate the oracle in all the ways that, that we described?

>> Yeah. I mean, and, and maybe it's, um, semantic, but I would outsource and, um, give consideration to that question. And, you know, if I accept the thought experiment, maybe I need to give very little consideration because it's like already answering for me, not just in general, like it's not just saying what is the most profitable, you know, opportunity here, but it's actually answering at the level of me. I think I would be comfortable using it. I mean, I would want,

>> a system of safeguards in place. And provided that was in place, right, I would take advantage. And in that way, I would be competitive with others that were that were doing the same,

>> right? And this is actually very helpful, this thought experiment teasing it out, that the safeguards are, you want to make sure your reason goes as high as it can go, right? Even if it's not to constantly, you know, it's constantly in use. It's, you're, you're as practically sharp, uh, as you can be trained by the AI. Uh, you want the ability to exit, and maybe compare systems. You want different points of view. But if those conditions are satisfied, and again, I'm not suggesting that Christian faith is like this, you're willing to make the leap of faith and and outsource certain, certain decisions.

>> Yeah. Yeah. The thing I own is the means and hierarchy towards human flourishing is one in one way, and the deliberation therein. In other words, I can use tools as an instrument to attain my goals, but I don't want my goals to be set for me. And so, if one of my goals is to, um, start a company, I can use AI to help me determine what that company should be, but I don't want it to set the goal for me. In other words, I don't want to be a blank canvas and just say, "What should my life be?" Right? However, let's say you had it as one of your goals. Uh, I want to build a company to, let's say, just make a lot of money, right? And I think you and I both agree that

A purely mercantile life is not the best life. Wouldn't it be better if the AI, I mean, didn't force you to add another end, but forced you into a journey such that you discovered the end of being mission-driven and helping others? Does that make sense?

So, it's not the AI. The AI isn't saying, "No, no, here's the real end you should go for." The AI is, but the AI is also not optimizing on your end that you told it to optimize on. Does that make sense? Wouldn't that be better?

Yeah, it does. I mean, it's kind of like an adult version of the tutor and Emil, honestly, because you're sort of setting a configuration for this child, in my case, me, to do development. Now, that presupposes that the AI cares about my development, right? It also assumes that the AI understands development in the way that I understand it, which is to say a process of self-direction that I do. And even then, I think there is this nagging question of that has been externally computed. I am now the agent of the AI. I'm the agent of an AI that appears to be highly benevolent and very focused on a kind of Humboldian vision of human flourishing. But the AI in your thought experiment is determining my end for me. It's saying that I should be developed in this way.

Yes. But crucially, it sets you up on a journey such that you take on that end yourself. Does that make sense? Yeah. So it's not like beating you over the head saying, "Money making bad, money making bad." It's, for example, going through an IPO and then giving you a business decision that makes you lose a lot of money and realize the relative worthlessness of money.

Yeah. I think a good analogy here is parenting, right? Parenting.

Well, another good analogy is the state. So, like, do you want to have a system of laws? Do you want to have a state that views its role as being developmental, like that tries to maximally endow you with autonomy? Or do you want to have a state that is more like a night watchman and preserves space in which you can experiment and try things and learn yourself? The latter tends to have far fewer risks of paternalism, right? Because it genuinely is not the role of the state in this case to inculcate this habit in me. And I think that's a foundational position I would take. In this case, we're saying, could AI fill that very benevolent, paternalistic role?

I'm not asking, could it fill it? I'm saying, let's assume it could. Would you be happy to have it? Right. Right. Because obviously, you know, neither you nor I believe that current AI can do this effectively at all. Yeah. And maybe even in our lifetimes, we won't have this. I'm asking you the philosophical question.

If you take the premise that it can do this, would you be okay with it? I.e., shaping your ends in a way that you wouldn't agree with now, but in some sense is the right answer to the human good? Because you also don't want to go completely relativistic and say there's no human good whatsoever, right? It's whatever ends I want to have now.

Yeah, I tentatively accept it in so far as the AI was setting up the maximum space for me to have this Siddhartha-like journey or whatever, this developmental journey. If AI was scaffolding that for me, then that would be highly consistent with the idea of human flourishing, and especially if I'm directing the AI to do so. In the case where AI is doing this kind of surreptitiously, I'm a little less enthused. But I think I'm trying to experiment with that as a completely unrealistic framework, but a thought experiment that could support...

You don't have a philosophical problem with it, right? You have engineering doubts or something.

I have practical problems with it, beyond just engineering. But I think the philosophical premise, I like it. I tentatively accept.

I see. So, obviously, none of our AI systems today are anywhere close to this omniscient autocomplete. What do you think our conversation and your thinking around autonomy has to offer engineers today and builders today in building systems that support and enhance autonomy?

Well, I want to say that the mere fact that AI can't do the omniscient autocomplete thing is only part of the story. And I think this is a really interesting notion that there is, on the one hand, the kind of epistemic question of, like, can it do it? We agree it can't do it. The other question is, why might we feel as though it can or we should want it to have that role in our life? And that's a psychological question. So there's the epistemic one and the psychological one.

I'll make it really tangible here with an example. So, a year or two ago, there was a 42-year-old guy named Victor Miller in Cheyenne, Wyoming, who ran for mayor. And what made his mayoral bid unique is that he ran as the meat avatar, essentially, of ChatGPT. In other words, his pitch was, "I'm going to run for mayor, but I'm going to turn around every question that I get asked, I'm going to tell ChatGPT."

Like the omniscient autocomplete that I suggested.

Yeah, exactly. He didn't win, but it's interesting for a few reasons. One is that it's potentially prophetic. Like, we may have AI playing a major role in ruling, right? That's one reason why it's interesting. The other reason it's interesting is that he thought that this was a good idea. And I don't think he was making an epistemic claim. Like, I don't think he was deeply analyzing what AI could and couldn't do. I think he believed that it was a good idea. And this is again a psychological question. Like, in this case, we want to believe that all the blood and treasure we spill on politics can be solved by a ruler that has access to truth that's authoritative, that's seemingly impartial, neutral, right? And so I think we really have to keep in mind that your scenario is far out, but it almost doesn't matter. Like, people right now believe that AI should tell them what to do. There are "Claude boys." There are literally teenagers who call themselves "Claude boys" who will wake up and they will do what Claude says and not otherwise.

Really?

Really. So they...

What do they do? Are their lives good or...?

No. But I mean, on what basis, right? I mean, it gets back to this philosophical question, right? Are their lives good in so far as they make fewer dumb teenage errors? Probably, actually. And the point is, this is a great litmus test for, do you think this is a good idea?

Yeah. Right. I have been in closed-door rooms with people who are really high up at the AI labs. They tend to be effective altruists, and they say that we are being foolish rebels if we don't listen to the AI. They truly say this. And it is because...

Well, it's religious language, right? We're like rebels, like the fallen angel, right? It's, "I'm like, how dare you rebel against God?"

Exactly. "How dare you, you silly, you silly rebel." And that is because they have a view that the things that we do should be viewed purely through the lens of a consequential lens. In other words, they don't have a thick notion of what it means to be human.

And if you don't have that, why not take the optimal path, whatever that means? Right.

Right. This is a major issue.

And I think both what you and I are suggesting is the optimal path ceases to be optimal when you only think about optimality. Like, when you only care about dating Sally or dating Susan, when you don't care about the autonomy, the agency, the self that is willing. If you give that up, your life is ipso facto going to be bad.

What is the point of optimizing a life that ceases to be your life to live?

Right. Yeah. But in the counter, and this is what I was trying to show with a thought experiment, is that AI, if it does become, and it's clearly not there yet, this omniscient practical reasoner, it potentially can help us direct our own lives better.

Yeah. Yeah. Right. So that's you're willing to grant it.

Yeah. And well, so I think the mechanism where I would be much more wholesomely embracing of this is, you know, if we built into AI the goal of fostering better self-direction, right?

Right.

And this is a little bit of this is kind of where we were getting to in the scenario, but like, how do you do that, right? So one way you could do it is you kind of reject the idea that AI should be an answer machine ever and always, right? That's what it kind of is today. We ask it a question. Doesn't matter what question we ask, math question or what is justice, just here's an answer, right? In reality, the way we think about questions is we have a question. It invites other questions. There's kind of a ball of questions around the central question. And then we have to kind of do a navigation of these things. We have to balance that search for broad understanding with a need to act in the world. So we have to reach some kind of an equilibrium that balances both desires. Okay. If you had an AI system that could guide that, that could spur, that could raise questions, that could help you make judgments that would get right at the core of what self-direction and deliberation entails.

Yeah.

And I think there's no reason why we can't do such a thing right now.

Right?

But the difference is that tends to open up the possibility for self-direction. It raises questions. It doesn't close off and just give you an answer. In other words, the problematic use case is the autocomplete, right? Like we should be very concerned when anything approximates autocomplete. When it, when AI is used instead as a provocator, instead as a razor of questions, and as a helpful tool for deliberation, then...

Yeah.

You know, that's wonderful. An example here is, I'm talking with some of my friends who are like junior consultants or investment bankers, and they're just using it to replace their own work, right? And that's the kind of problematic case. Or, you know, the kid who uses GPT to write his own philosophical essay. The way I'm using GPT right now is that I'm using it as a live tutor to kind of read the text together with me and ask different questions. And so that's the two different paths you want to put in front of us. And this is not EA, not existential risk. This is the true challenge that's ahead of us.

Although I will say, I'm not sure that what analysts do at consulting firms is that valuable.

And so they...

Confidence, right? And so, like, if they can make a PowerPoint, great. But I think ideally, what would happen is that that freed up space for them to do more, you know, things, whether that's higher-level strategic thinking or reading philosophy in their spare time. But the point is that that goes back to the benefits of offloading. There are some things that are very good to offload and automate. But we should be very careful to offload and automate the core deliberative capacities, right? And hopefully, our audience can see why the answer, even if AGI is around the corner, to just focus on alignment and building AGI, and for AGI to sort of focus on these problems, autonomy is short-sighted.

Is because by definition, we are the ones that need to be doing this. Or another way to frame it is, what alignment means in our kind of philosophical framework is to build AGI that enhances autonomy.

Yeah. Right. And so that's why, even if AGI is around the corner, we have to be building AI with this fundamental tension of autonomy and dependency in view.

Yeah. Nick Bostrom is correct that philosophy is on a deadline. He is dead wrong about the role of philosophy in thinking about the deeper conception of human good that AI needs to uphold.

So, we talked about the existing landscape of philosophical schools. We talked about what you actually think matters, which is the autonomy question that people are overlooking, and how it plays out in AI. But now I want to dive deeper into autonomy itself. Okay. So you described in your writing of autonomy as the central good. Okay. Be a bit more precise here. What do you mean by that? Is it sufficient for the good life? Is it necessary for the good life? Would you ever trade it off for other goods? Tell us about that.

Yeah. So I think it is necessary to live an autonomous life. It's something that I think develops like a muscle. So I think we try things out. We self-direct, maybe badly. Like, certainly as kids, we do that really badly. We don't even select our own projects, essentially, and then we develop and we get better and better and better. And as we do that, it becomes more and more an important contributor to our happiness. In other words, we value it more. It becomes more and more central to how we think about what pleasure even consists in. And so it's this developmental thing that sort of happens through our life. And that, for me, is why it makes it the central good, because it is the thing that unlocks our ability to know our own selves, our own gifts, to develop those gifts, and to use those gifts to live the life that we want to live.

However, it's not sufficient for a good life because one can imagine, this is another edge case, where the fully autonomous man or woman who is able to go through life like this, but just fails in all of his or her endeavors, right? If all family dies, like this is the Aristotle example, *Nicomachean Ethics*, about...

Eudaimonia.

So, this is not the only good.

I agree. Right. Yeah, I agree. It's not. It has a relationship to other human goods, but is not the only good. And I agree that it's not sufficient. I also don't think that autonomy presupposes choosing well. And so I think that is actually a consequence of having autonomy that you can choose. That you have to allow people to choose very badly. In other words, people choose a self-directed path that is very harmful to them, and you have to let them. And so, in other words, like, you know, we, it does not is not a prescription I say causally efficacious because I think it on balance tends to lead to happiness, but it certainly is not. It also comes with the weight of responsibility. I mean, there's a lot to be said about the burden that one feels when one can freely choose.

Yeah. Well, I want to push you on that point because autonomy, as you described it, is one of my like, as I live life, like one of my central goods, which is why I'm doing this here and not in a more structured setting. But I was really surprised when I entered the workforce that most people don't seem to like it. Like, when I started managing people out of college, I structured their work environment how I would like to be structured, which is, this is our goal. I'm going to explain to you the reasons why we're going after this goal. You choose how you get there. You just need to get there.

Yeah. They, most people hate that. And they won't say, "I want less autonomy." They won't frame it there. They would usually frame it that, "I want more structure in my life." So they want to be told what to do. They want... Yeah. And so, how do you reconcile that? Is it just people like you and me who value this, and it's subjective?

Yeah. This is one of the most worrisome threads that my wife will raise with me, because she'll be like, "Brendan, it may be the case that you are just like an outlier, that you care about this in a way that other people don't." Intuitively, meaning you may be like rationalizing it, but you may actually just care about it deeply. Intuitively, in the way that you might like chocolate, that doesn't mean everyone else likes chocolate.

So, we have lots of individual variation. I totally allow for that, that people would like it more or less based on their, you know, sociobiological kind of path. Um, and I think what I mean by that is that it could be like almost epiphenomenal, that somebody has certain genetics and they have certain disposition, predisposition. Okay. But I generally think that it's much more determined by the conditions in which we live and by the way in which we're habituated as a consequence. The transition that I'll mention here is that in aristocracy, one of the things that one of the benefits was that people knew their station. They knew their role. Um, I'm not advocating for that, obviously. I'm not advocating for a kind of hierarchical system like that. But that was one of the silver linings, right? In America, when Alexis de Tocqueville came, he observed a generalized anxiety. He called it "ennui," it's a word in French for like anxiety without an object, a particular object. And why he thought it existed was because there was no one to tell you what to do, you know, like you had to make your own way. And so you look to the majority to tell you what to do, right? You look to the state. Like these are the kind of pathologies of democracy, is that you sort of fill the void, and it appeases you, especially if you take religion away. By the way, religion grounds you and it minimizes this.

Well, it's the ultimate form of telling you what to do, right?

And it grounds you. And family life can ground you as well. Lots of things can ground you, but absent those things, you kind of drift and float. In America, it still was the case that people were very self-directed. You're, you know, the classic example of this is the Jeffersonian ideal, which kind of thought of autonomy as being the farming life, right? It's like you grow up on a farm. My wife's family has a ranch. They're ranchers. And it really is the case that like you are very self-directed. So, I get it. And I wonder what the consequence of being a nation of farmers, like almost all entrepreneurs essentially were entrepreneurial farmers. Yeah. I mean, like 85% of free Americans were that way, and now we're a nation of employees, right? It's given way to being, you know, the Industrial Revolution caused us to enter employment. We now are subservient to process and to people in a way that we were not. And I say this to bring all this together, is that the conditions, either the regime, like now we're in a democracy, no one tells us what to do. That's very scary for people. But also the fact that we move to an industrial system in which people tell us what to do constantly. Like, "Can you take vacation?" "No, you know, not today." Like these are kind of limits on your self-direction. And as a result, we've now come to like it, to want it less.

Right. It's Stockholm Syndrome, almost.

Of sorts, right? Yeah.

You mentioned Stockholm, and it makes me think of this case of Germany, which I fully understand is not in Germany, just so people don't think I'm an idiot, but East and West Berlin were two radically different systems, as different as it gets. You had the same family, same genetics, split across the wall, right? And the East German system, under Soviet control, was habituated to follow orders. The West German was much more like the general West and less so. And then during COVID, like much more recently, you had a very different response. You had much more obedience among the people who had been habituated by the East German system.

Yeah.

Why I say that is it appears that the habituation is long-lasting. Like, you can grow up under a system in which you're told what to do, and then maybe for the rest of your life, you are inclined to do what you're told. Whereas if you grow up in Texas, um, you may be inclined for the rest of your life to not do what you're told.

Wait, wait, hold on. But that doesn't seem to be such a strong response for the position that autonomy is the central good, whether you appreciate it or not. Right? Because what you were saying is that nurture can greatly change how people valued it or not.

Right.

But I thought you would want to argue for the position that regardless of what, like whether you valued it or not, it is valuable objectively.

Yeah. I don't want to base the idea that it is constitutive of a flourishing life on the idea that it is widely used in practice or even that it is like valued equally by all, because again, the mechanism is one of habituation, so it's not going to be valued equally by all. And I actually think having highly autonomous people is a total historical anomaly.

Oh, yeah, totally. Yeah. I mean, for most of human life, we've either been in a culture with slaves or we've been in, you know, kind of hierarchical cultures that really don't have the same presuppositions. So, I think it's anomalous and it's precious. But that's not why I think it's constitutive of a flourishing life. I simply think that like it is consistent with it is the way in which we discover our nature, the way in which we express it. It is one of the things that humans uniquely do is to use reason to guide action, to develop ourselves, and it tends to also lead to happiness.

Okay. So, so what is the reason, like what is the reason that it is constitutive, despite the fact that people can be habituated out of desiring it?

Yeah. Well, that's why it's because we have a nature that the only way to discover our purpose or our highest end is through this autonomous experimentation, is through the development that we do through self-direction. What about, again, I'm going to go back to religion here because I think this is the counterpoint here. What about someone who surrenders themselves completely to Jesus and claims that through that, that they found their true selves? Or, I mean, in the Buddhist case, they surrender themselves completely to the master, become completely obedient to the master, and through that, they think that that's the route through which their development becomes...

Well, I think in so far as they surrendered, that is a pretty powerful act of self-direction, you know, in the making the choice. I do think that is... It's difficult to square the autonomy lens with say, Islam, that views that views...

I mean, submission, it's in the name, right?

Yeah, as submission. But here's what I'll say about that. I think, you know, I think one of the most critical points one can make about the dynamism in the West is that it is a tradition that has been formed through different visions of the good life, all of which allow for some individual choice about what the good life is. And to be more specific on that...

Yeah.

One tradition, probably the earliest Western tradition, is from Homer. It's from the Mesopotamian epics. It's from the Bronze Age, and it's the idea of the heroic life, the life of adventure. Achilles is a good example of this. The Greek response to that is a life of science and contemplation and philosophy and things like that. That's a life, that is to say, a life in which Achilles is replaced by Socrates, like the contemplative life. And those are two totally different visions of the good life. A third comes in the Hebrew and biblical tradition, in Jesus, which is a life of pious devotion and centers more around the family and other things. I would say none of these are commensurable. They're all different visions of the good life, but they form a kind of tension. One, I would say the overlapping areas is some measure of individual choice as to what the good life is. Like, it's not the community's job to tell the individual. The other thing I would say is that...

Even for the Christian tradition.

Even for the Christian tradition, in so far as we get to choose that path and we navigate that path towards Jesus, there is a submission element of it. But again, like the freedom to be unfree is, I think, a satisfactory use of freedom to...

Well, because otherwise you wouldn't be able to enter into contract, right?

We bind ourselves. And a lot of the, you know, most important American figures grappled with these and synthesized them in their own way. Like Lincoln read Shakespeare and the Bible, for example. Like he's reading about these things and forming his own sort of local synthesis. And so I just say that, you know, we would do well to preserve those tensions. One of the things I worry about with AI is that the dominant schools seek to sort of come in with an answer that's like, "No, this is the one true thing. It's actually..."

Those good life things, those are quaint.

Really, what it is is maximizing utils.

Wait, but you're coming in here and you're saying autonomy is the central good.

Precisely because it is the one that leaves open the possibility of deciding the good life. In other words, it preserves the plurality. It preserves that space for the individual. I think it's a critically important thing.

Because we could just say, well, isn't it again, isn't it just like another vision of the good life? But it is very different to say that a vision that preserves one's individual ability to question the ultimate question for themselves. That is different from one that tries to reduce and replace and say, "We have found the one true answer."

Right? And just to be clear here, when you say autonomy, is it a very thin kind of simplistic "I'm the one willing it," whether that's the right thing or the wrong thing, or it's something more like Kant, where I am willing the thing that also is, like, rational and good and...?

Um, there's a subtlety there. Because so it's the former, in the sense that I believe that one must deliberate, but not that they need to necessarily deliberate well. Nor am I stating that it is a moral autonomy. Like, I'm not suggesting that what autonomy means is to give oneself the moral law, which is to say the categorical imperative. That's not what I'm saying. I'm saying that we must have a capacity for reasoned self-direction. We can do it wrong, but we must preserve that space. The reason I say it's complicated is because I do think Kant actually enters into it. In other words, I personally draw a freedom-maximizing principle from Kant. And this freedom-maximizing principle is what gives people space to be autonomous. These are two different conceptions of...

Right. I see. So it's not just the simple notion of like a whim. Like, let's say I have a whim to jump off the building right now.

Yeah.

Because no deliberation went into that. But it's also not the, you know, if I really, really deliberate, like, "It would be good to steal Brendan's wallet right now."

Yeah.

That would count.

It would count.

It would count because it's even though I didn't deliberate well, I made a fair attempt. And your response was, "If I do that enough and I and I continue deliberating, I..."

We have to make mistakes.

Yeah. Like, that's a dumb thing for you to do to want to steal my wallet. It's over there, by the way. But, um, but we have to make mistakes, you know? Like, I think part and parcel of autonomy is doing dumb things. And, you know, it's simply the wrong standard to suggest that we must always do things rightly.

Um, let's say someone lives an autonomous life. What else is missing to get to the good life or even the best life?

It's a good question. Um, one point that needs to be said is that what I've outlined so far is fairly individualistic, but I think the way in which we learn and experiment is profoundly social. And so we learn from others, especially people who are, you know, above us in terms of aspirationally, but not so far from us that we can't sort of learn. And so I don't want to suggest that we're sort of like operating in a solipsistic way or an isolated way. Um, that's important to say. The other thing way I think about this is like, how do I want to educate my kids? I want them to be autonomous, but I also want them to be autonomous and virtuous. In other words, like, I view the role of the state as being to a procedural, a formal role, to preserve freedom, to minimize coercion, that sort of thing, to provide for security. But I view the role of, you know, I view the good life as being something more than autonomous and in fact, like virtuous as well. Does that make sense?

Sometimes you need to trade off autonomy and virtue. Right? One example would be, well, okay, if I want to jump out this building right now after poor deliberation, there's a good case where you should restrain and limit my autonomy to preserve, let's say, my bodily function, in order so that I can be more autonomous in the future, but also potentially to be more virtuous or to not kill a person, right? If...

And so, do you allow for a trade-off between autonomy and other goods? And this obviously is a political question as well. One extreme idea would be, no paternalism is allowed. You can never interfere with autonomy. Like, there are other goods, but autonomy can't be traded off for other goods.

Yeah.

Yeah. I mean, in general, I don't think that the gains to welfare from paternalism outweigh the losses to autonomy. And so, I would take a very strong position on not applying the tools of the state, in particular, to paternalistically deliver welfare gains. And I would apply that to things like UBI, for example, as well as many other areas. But I think that, you know, security is a good example of a kind of a vital interest, as Mill would say, that it's not clear that autonomy has a lexical priority over security. Like, security seems preconditional to autonomy. We have to maintain security.

So, so your answer is, it doesn't have lexical priority, i.e., a trade-off is sometimes worth it.

Yeah.

However, people overly value the benefits from welfare gains and undervalue the dangers of even removing a little bit of us. That would be your answer.

Yeah. And this is a fundamental conflict because we value convenience quite a lot. I mean, this gets back to soft despotism from Tocqueville, is that we welcome the incremental convenience from a state that is far-seeing and mild, you know, that will offer this. It's happy to offer it. We're happy to welcome it.

Or from AI, right?

Or from AI, and from any sort of neutral or centralizing force. But what do we give up? Well, we give up the vigorous use of our own capacities. We become enfeebled. We become like a flock of timid, industrious animals. I mean, Tocqueville has beautiful language around this. And so this is a fundamental issue, particularly in democracy, where again, we otherwise have kind of ennui because we have no one to tell us what to do. So I think like we've staged the critical issue of our time, which is we now have built something that can deliver the incremental convenience, that can offload our deliberation. We are going to welcome it into our life. We're going to be tempted more than we've ever been tempted before, and we must find the resources within us to resist.

Yeah. By the way, this is, as you know, I just interviewed the founder of Alpha School, and where you send your kids. And I shadowed them for a week. And I was so excited by that project because, I mean, the, so for our audience, the quick pitch is that you learn basically your entire K-to-12 curriculum instead of six to eight hours a day with homework in two hours. You're like, "Okay, that's interesting." But what I found when I talked to the kids was that the greatest benefit was a fundamental change in their character, that "I am capable," "I am autonomous in this way." And which is a fundamental difference from how all the kids are being taught today, which is this kind of teacher in front of a classroom lecturing, the kind of Prussian model to create industrial slaves, industrial employees, essentially. And so, so that, like, I almost think that they're under-selling, under-pitching what they offer, offering a fundamental building of their own character and not just like being able to cram.

That's precisely right. And that's exactly the benefit that I see in my own children. When I went there to like talk about entrepreneurship to eight and nine-year-olds, and I was just so moved by the extent to which the kids had individuated. They were highly individual and very autonomous and very high agency as well. And I fear, I just to make it kind of a cautionary point, I fear a kind of divide. Right? In other words, I see the Alpha model, and I see the vision, by the way, is to get that out there, and I really hope it can scale massively. But I also understand that if one's relationship with technology is one of passivity, one of dependence, one of doomscrolling, then we become highly dependent from the beginning. And we almost create two classes of people. We almost create one individual for whom it is the best time in history to be a six-year-old, and one individual that is on the path to become an NPC.

And I, I think we must avoid that.

Yeah.

And in some sense, compared to early America, we're already there with the employee versus the gentleman farmership. So, what you're saying is that there are a lot of things that don't seem political which build the autonomous muscle or weaken the autonomous muscle in civil society. That's right. Education being one of them, workplace being another.

Okay. We've talked a lot about the intrinsic importance of autonomy, how it's constitutive to a good life. I now want to move on to the extrinsic benefits of autonomy and why it's important to defend autonomy, not for why it's going to make you live a good life, but for a flourishing civilization. Okay. So, I know you've been dying to talk about Hayek, so now's your chance.

Yeah. So, um, I'm kind of a Hayek stan, but the reason is because I think he's desperately in need of being revived for the AI age. Um, where to start? I mean, his, my favorite book of his is *The Constitution of Liberty*. And in that book, he makes a consequentialist case for liberty. So he says that liberty is useful. And the reason he does this is that when you argue for something and you start from axioms and you deduce from axioms, you just invite the challenge, "Well, I don't agree with your axioms." So, I don't care if your deductive reasoning is good, I just don't agree with your axioms. So he doesn't do that. He makes a consequentialist argument. I say that because I don't think that that is the only reason why Hayek thinks liberty is dear. But he makes a consequentialist argument, and what he concludes is that we should have a minimization of coercion. What is coercion? Coercion, for Hayek, is a kind of configuring of the decision space such that you do the bidding of another because it is the lesser of two evils. Basically, you have your decision space so configured by another that you no longer are taking action on your own plans, but you're taking actions on the plans of others. Um, the steps to get there are many, but essentially what he says is that liberty is useful because it facilitates the use of knowledge in society, and that knowledge is what allows the anonymous person to attain their unknown ends. So that's a consequentialist view. How does it do that? Well, what he says is that most knowledge is practical. It's primordial, practical. It's not the explicit, semantic knowledge that we write down. So, like the knowledge in science, what people usually think of as knowledge, he would say is the tip of the iceberg, or the droplet of the wave that's above the ocean of knowledge. What does he mean by knowledge being practical? He means the dispositions, the habits that each of us has. The way an entrepreneur thinks about an opportunity, the way a diplomat sizes up a room, the way we ride a bike. Those are all things that are sort of locked up and inside of us, and they're either inarticulate or inarticulable. And so we have this, it drives our action, but we can't share it. The way in which we can share it, the best way we can share it is through the market. We have a low-bandwidth mechanism called prices, money prices, that allows us to share our knowledge because as we try to do things, as we formulate ends, try to achieve those ends, we release bits of this knowledge. Of course, I never like excavate it and share it with you, but my action is colored by it. So it releases this knowledge. And as this happens in parallel across the entire world, we pursue our independent plans. The market allows for us to kind of equilibrate those plans. Preferences change constantly. So the equilibrium should be thought of as an asymptote, not a fixed, Cartesian equilibrium as traditional economics would say. But it allows us to do that. We share knowledge, and this is what gives us a way to benefit from knowledge that we don't possess.

Uh, part of Hayek's consequentialist argument for autonomy is, as is described in the second chapter of the book, "The Creative Powers of a Free Civilization." What do you make of the tremendous creative powers of unfree civilizations as well as unfree peoples? Let me give you a few examples. The pyramids, the Great Wall, built by basically slaves. Virgil was writing propaganda, right? Pro-empire propaganda. Dostoevsky wrote in exile, imprisonment, financial great distress. Most religious traditions had these unquestionable truths. And as you know, early modern science and philosophy flourished under severe censorship and sometimes even persecution.

So, I think if you have as your goal to exploit the existing stock of knowledge, then I think unfree societies can do that. In other words, if you want to demonstrate what command and control can do, you build the pyramids. Um, but I think to discover new production methods, for example, you need the undirected experimentation and the spontaneous order that arises through free societies. In other words, it's a totally different question to say, how do we deliver something at this moment in time based on the knowledge then known versus how do we promote, how do we bear and disseminate and generate more knowledge in the world and make progress?

But surely you're underappreciating the difficulty and innovation in building like the first pyramid, or starting to build the Great Wall, or another from Chinese antiquity, being able to divert rivers away such that it doesn't flood certain parts of areas. That's not just, these are things that they didn't know how to do before.

Right. Well, so I think you can set as a goal, like you have a known goal, you want to create a pyramid, and then we have a lot of technical obstacles that we need to get over. Or we want to go to space, and we need to do it with alloys that haven't yet been invented. To quote JFK and Rice University.

This is the kind of top-down, it's like the pinnacle of the top-down model, is that we can set an audacious goal and we can often realize it if we have the right structures in place. What I think free societies do is they secure a kind of adaptation to the future. So as future conditions change, people doing lots of experiments in parallel create variants, create solutions that just sort of bubble up. Right? That's one thing. They also are the best way to grow the stock of knowledge in general. Like, yes, you know, the creation of new alloys was probably accelerated by JFK's push, but in general, science works through a kind of republic of science, you know, to quote Michael Polanyi, where you have distributed science. No one is setting the direction. There's no House of Salem from, you know, *New Atlantis*. There's no one saying what science should do. It's just a loosely connected republic of people all trying things and experimenting.

Right. So your response to the seeming counterexample of early modern science, like Galileo, surviving and flourishing under persecution is to say, "But there is a republic that is free among the scientists, or relatively so." Right? That's what you would be forced to...

I'm also saying that there's a tension where like Galileo, Dostoevsky, they have to carve out pockets of freedom in very bad conditions in order to do their work. And, you know, the Inquisition with Galileo, or the geneticists under Lysenko, are good counterexamples where you have authoritarian control that for non-scientific reasons wants to shut them down and succeeds to some degree in those cases.

Right. So your response there is essentially a lack of counterfactual response. Like, if Galileo did what he did surviving persecution, but what if we had a parallel Europe that was free at the time? Think about how much better that would be. That's...

I think that's brilliantly put. Yeah. I think it's, you know, you look at the examples where great art was created amidst terrible conditions, but then what you don't consider is, yeah, what would the parallel universe have looked like?

Yeah. Okay. Well, well, I'm glad I trapped you there because I set up a little trap for you. Because America, which I think any reasonable person would say is the most free, in the way that you're in this Hayekian way, society, it's not totally free, but the most, a sort of nation in human history, America clearly is amazing in its entrepreneurial innovations and its creativity there. But where is America's Virgil? Where is America's Shakespeare? We've had 300 years, man. Like, like, and like, but even hard sciences, like America's is okay for like practical, applied science, but like Einstein, well, he, well, he was in America, but he's obviously he was in a different... You see what I'm trying to say? The counterfactual response would be a lot more compelling if America became this land of infinite creativity and great works being written left and right. And yet...

The only place where America seems to extend creativity is in the economic sphere.

I think it's a good point. Um, I mean, I want to say that we are not without achievements in each area, right? There is an American novel, you know, there is a Faulkner kind of thing. But I...

Right. I take your point that we have like relative greatness in other ways. And, you know, the question is like, does our, does something about our system tend to squash the other kinds of greatness that we would see?

It's a little reminiscent of, like, Tocqueville's concern, obviously, because, you know, he thinks that, um, democracy can create a kind of mediocrity, and a particularly, particularly a mediocrity of desire or of, like, aspiration.

He, he writes about, like, um, the American merchant captain, actually. I think this is a very funny and not well-known, uh, thing, which is that, uh, he sort of, like, looks at where honor still exists, a very aristocratic virtue of honor, uh, all but lost to the world. But he says it exists among these merchant captains in America who, you know, when they do something and save somebody else, they say, "I won't accept payment because it's a captain's role to not accept it." And so he's like, it exists, but I think what he says is that we need to keep the memory alive of the, like, high aspirations.

And so I would say that I probably fault not the free society, but the system of education that doesn't, like, cultivate this kind of desire, this kind of, like, highest desire, right?

>> In other words, like your example about the Prussian system of, like, industrial education, the kind of sameness that it breeds, that sort of thing, I think is one proximal cause. And then I, you know, I mean, I, I don't, I don't think I disagree that, like, capitalism, um, tends to produce the kind of person who has material desires. I'm a little bit ambivalent on this because on the one hand, I think, um, I agree probably with Friedman that, um, markets are a consequence of freedom. In other words, like, we truck, barter, and exchange. That wasn't Friedman, that was Smith. But and if we're left free to do it, markets kind of arise. Now, institutions play a big role. But if that's true, then on the one hand, markets are just kind of like a product of, of freedom. On the other hand, they clearly shape,

>> uh, the way we see the world,

>> normative ends.

>> Yeah.

>> Yeah. And, um, but all things do this, like, I mean, like the availability of clocks shapes our, the way we think about time.

>> Yeah.

>> These are unavoidable.

>> Yeah.

Well, I'm, I actually, uh, prepared a quote actually from one of my favorite passages from Tocqueville that kind of gives his answer about why he would say, why he said, uh, that America would never produce her own Pascal. And I think he's been right so far, right? Who, who's the best American philosopher so far? John Dewey. Probably not a Pascal. I quote you, Tocqueville. "If Pascal had had in mind only some great source of profit," this is what you're saying about the markets, "or had been motivated only by self-glory, I cannot think he would have been able, as he was, to gather, as he did, all the powers of his intellect for a deeper discovery of the most hidden secrets of the creator. When I observe him tearing his soul away, so to speak, from the concerns of life, to devote it entirely to this research, and severing prematurely the ties which bind his soul to his body, to die of old age before his 40th year, I stand aghast and realize that no ordinary cause can produce such extraordinary effects."

So, um, again, like, I think Hayek's, uh, consequentialist argument of creativity and freedom, and which I understand is not his only argument, would be a lot more compelling if, you know, in the American system, you had all these great artists and great, great, great creators and great writers, in addition to the great entrepreneurs that America clearly does have. But, but that seems to be the only domain that American creativity expresses itself notably in human history.

Mhm.

>> Mhm.

>> Yeah. I think this is a very tough line of inquiry. The was Pascal wealthy?

>> Uh, yeah.

>> I, I think I think he might have been a, a gentleman scholar kind of.

>> Yeah. Because I think Tocqueville also shares this kind of, like, it's a controversial point to make, but that the, the, um, avail, the multigenerational wealth that, like, um, primogeniture, like the estate law, basically the idea that whether or not a country breaks up estates or passes it to the firstborn son,

>> has a big role in the kinds of goals one can pursue because

>> says this exactly.

>> And so I think this is, I say it's controversial because, like, people don't like to talk about, you know, the literal elite and, like, you know, estate, uh, estate sort of law in that way. But I think Tocqueville is right that we, if we, you know, to the extent that we sort of break up, uh, estates, we give people, uh, a starting point that makes them very hungry to, but, but especially hungry for material things. Whereas if you're born into wealth, which I was not, to be clear, you know, if you're born into wealth, then you have a different set of ideas, like you kind of are, are kind of, um, blasé about it, and you, you're like thinking, and you either become, you know, lazy or you pursue different ends that are higher. Um, but it, I think Tocqueville thinks it's a useful experiment to have. So what you need to concede maybe is the full creativity of the market, or, or something like that. I mean, obviously the market itself is creative in the entrepreneurial sphere, but you still can preserve your point about, like, liberty itself being important for, for creativity, just different kinds of liberty are needed, something like that.

>> Yeah. Yeah.

I, I also will say, to in defense of, of, of, of entrepreneurs, that some of them have very great desires about humanity that are, that would rival the, the desires of the, the most laudable aristocrats of old. Like, in other words,

>> I don't think that honor and those questions, while I do think they're, they're, they've diminished, I do think that people get into the game for reasons like that, but it's the expression that would be kind of totally different than what Tocqueville is, is, is, is thinking of.

So, I want to move on to the last part of our interview, which is about, we talked about what right you want to achieve, mostly about autonomy as it relates to AI. We talked about why that's important, the intrinsic and the extrinsic reasons. And now I want to move on to the how, and it's in this idea that, uh, that you suggested of the philosopher builder. So, so what is a philosopher builder?

>> So the philosopher builder is a new kind of technologist. It's a technologist that thinks very deeply, contemplates very deeply about the alternate ends of technology and also has the skill to build that in the world. When you think of the philosopher builder, you should think of Benjamin Franklin, who, you know, everyone will know as one of America's founding fathers, as the face on the $100 bill. Um, what a lot of people don't know is that Franklin is an engineer of a very high caliber. He invents the lightning rod. He invents the bifocal lens. He coins positive and negative charge and electricity. He's also a philosopher. He lives by a 13 virtues idea and creates this thing called the Junto for mutual evaluation, mutual discussion. When Franklin is at his best, and when he really brings to life this idea of the philosopher builder, is when he's translating a philosophical idea into a practical innovation in the world. He's taking the idea that, for example, knowledge should live outside of the scope of authority, outside of the church and the state, and he's translating that into the world through the first lending library or the first network of independent publishers in America. That is the essence that we want to capture. The idea that you're thinking about these philosophical ideas and you're translating them into real-world innovation. So today, you know, it's never been more necessary to have that. But I think the institutions we have are failing to produce the archetype. You know, most universities, um, I would say produce pretty narrow technicians or conforming ideologues. I think that's by and large an accurate descriptor. Most tech companies produce people who are very, very good at building and thinking about the means, but who are not thinking about the ends beyond just sort of the customer use case satisfaction. And then think tanks create theorists that don't tend to build. So that's what I'm focused on. The, the inspiration that I draw is that there have been moments when institutions, uh, uh, really rose to the challenge. Institutions that come to mind are Cambridge during the industrial revolution, took a lot of mathematicians and turned them into the engineers that powered that. Um, MIT during World War II, the Rad Lab in particular, took physicists, made them into inventors, helped the war. And then Chicago, more recently, um, took economists and made them into, uh, reformers who freed markets across five continents. So we can do it, like, when a mission has that as their, or when an institution has that as their purpose and acts urgently, we can do it. That is what we're doing in Cosmos, is creating that new kind of technologist.

>> And in some ways, it's reflective of your own story, right? Because you, you started off on the building side, uh, and then you got into philosophy, uh, later on in life. So tell us that story.

>> Yeah, I mean, it even goes back a little further. Like my mom was an educator. She taught special needs kids for 36 years, and she brought us up in what people call a virtue culture. So Aristotelian, an idea that things like courage, honor mattered. And this was very effective, and it is what caused my sister, my brother, and I to all go into the military. I was a submarine officer, my brother was as well. My sister was the lead medical person when we defeated ISIS in battle with Mosul. And it's because that was a natural expression for this, um, desire for public good that that she and my father had inculcated in us. I then went, you know, went to MIT, joined the military, went to Harvard Business School. So I had this kind of classic STEM in business. And it was only after selling two AI companies and having my second of two kids, my son, who's four, that I really was hit with this like big perennial human question, like, what, what do you, what is the good life? What do you do with the rest of your career? What do you model for the little humans? That's a profound thing that happens when you have kids, is you, you realize that you're on the hook. And I didn't have answers, and I was very dissatisfied with the depth. I had cocktail party level answers. And so I started to read, and I had a mentor, uh, named Michael Strong, who gave me the gift of a lifetime, a 17-page reading list that really started with the ancients, philosophy, and went up to the Enlightenment, the American founding, contemporary debates. It changed my life. It transformed me. It made me more interesting to myself, and it totally changed the trajectory, and that's how I shifted from being an entrepreneur to being, well, I'm still an entrepreneur, but to creating, to be a philosopher. Entrepreneur.

>> Sure.

>> Philosopher.

>> Exactly.

>> Um, you said that you didn't grow up in wealth. Uh, and in some sense, what's quite crazy about your entrepreneurial journey was how quick the exits came. Right? It was a span of what, 18 months or or something like that, that there was like $400 million, uh, exits. What was, like, coming into that much wealth in that short period of time? How did that manifest for you?

>> So, I had people around me that, um, played a big role in this. So, I saw examples of lives that I didn't want to live. And these are people that I'm, in some cases, friends with. So, I don't want to be too, uh, too specific here, but I saw examples of lives that I did not want to live. And I also saw examples of lives I did. And so I had a friend of mine who is part of a philanthropic network of young people, like in our 30s and 40s, who are doing pretty serious philanthropic work. And I saw him do something that I felt was transformational at a young age. And I remember the moment when I heard this, and he had given, you know, $5 million or something to this amazing cause. Um, and I, I just talked to Adrian. I thought, you know, we could do something like that. You know, we could do something big. And the counterfactual of not doing it was felt huge. In other words, not having the benefits of not just money, but like time and effort and talent applied and not compounding that felt like a huge missed opportunity. So, I really credit it to being, uh, able to surround myself with examples of what I thought bold action looked like, and I chose, you know, very deliberately the path that I wanted to, to emulate.

>> I see. Oh, well, let's, let's go back to the philosopher builder archetype because, um, I think about it as fleshing out this third option, um, in Platonic political philosophy. So obviously, uh, Plato separates his polis into three. There's the ruling class, uh, there's the, the army, right? The military, uh, and then, then there's the producers, like the, the merchants, the builders, essentially. And Plato obviously advises the rulers to be philosophers. This is the philosopher king. Um, I think we see in the end of the Roman Republic, the philosopher general. These are people like, uh, Cicero, like, uh, like, um, Caesar, who was very learned. He was like composing, uh, tracts about anomaly and analogy in Gaul while, like, arrows were flying in his face. Um, and now you're suggesting is that it's the third class, which to Plato was in some sense the lowest class, the builders,

>> should be philosophers. Why? Why is that?

>> Well, yes. I mean, I think it draws some inspiration but differs in really import, one really important way from those, those archetypes. The inspiration is that there is a belief that you can have not a unity of contemplation and power, wisdom and power, as you see in the philosopher king, but a unity of the wisdom and the contemplation that gets you there with, uh, the ability to create, to create worlds, to build. Um, what I think it, it makes the distinction clear is actually the Greek concept of order, which divides into two distinct words. There is taxis and there is cosmos. Taxis is the order that we impose on the world from the top down. It's taxonomy, right? Or it's more generally a kind of top-down order, right? Um, and then the alternative is the bottom-up emergent order that is cosmos. And obviously, I've embraced this, uh, in so far as we're named the Cosmos Institute. But what it means in practice is that we're not looking for one individual who has a kind of blueprint that we look to to rescue us in difficult times, um, and who can implement that plan, sort of from the top. That's the philosopher king. We are looking for a much more bottom-up, distributed approach, where people may have slices of truth, slices of the solution, and are working in their corner of the world to project that, that vision forward. That's the cosmos approach and the philosopher builder approach. It's a different archetype, much more like Franklin, much more distributed, and one that I think is

>> Right.

>> You know, necessary for the, the current moment.

>> Right. So taking the insight from Plato about the importance to join worldly activity and and and philosophy and and, uh, and contemplation, but given everything you said about autonomy, transplanting that into an autonomous, decentralized way. Yes.

>> And hence you find the philosopher in the third class and not in the not in the ruling class. I, I was going to give a very different answer to, to why this is important today.

>> Please.

>> Which is that technology, like, you can argue in some sense that in Plato's time, uh, ruling and political power was the dominant pole, right? Um, whereas today, I think it's the market, uh, as well as technology that is the dominant pole of the three. Um, you see how, uh, I mean, obviously you see how technology is dominant over politics, given everything we've described so far about how technology can form political citizens to be more autonomous or less autonomous, uh, and you can also see with things like Anduril or or Palantir, how an AI, of course, has, has dual-use technology, uh, you see what I'm trying to say here, right? Like the technology, like it's important for the philosopher builder, not just because of what you said, but

>> Because technology

>> now supersedes the other two spheres.

>> Yeah. And it becomes the driving logic of the two spheres.

>> Yeah. Absolutely. No. And it's a break with the ancient idea that politics is architectonic. The regime sets the frame for what kind of technology could even be done, right? I am persuaded by that. But as you, as you sort of look at where we are in 2025,

>> It's opposite.

>> It's flipped a bit. And so, yeah, so you could talk about it being upstream, but I like your idea that it sort of is like, technology is kind of architectonic. Like it's kind of the, if you think about who the best reformers are, the most capable reformers are, it's not people like John Stuart Mill, who whom I love, but it's people like Elon Musk. And I, I don't say that to endorse

>> Everything he stands for.

>> That's not what I'm saying. I'm saying his position as a profoundly capable builder gives him enormous leverage on, you know, what we thought of as like political questions of old, or, or, but again, I just want to tease out the the mechanism because one of it is him able to use his money, um, and to be able to, for example, fund Trump. Um, but, and I think Alpha School is the better example here, like the technology of AI, when it applies to education, might be a, a much more powerful political tool for liberty than, you know, like doing anything with the government that can possibly be done today.

>> Yeah. Yeah. Because, because precisely because technology is going through all of our lives in this way.

>> Yeah. That's right. And the Alpha School example gives also color on the interdependencies because if you don't have a political order that is capable of, uh, sustaining that kind of innovative school model, um, you at the very least have a, it's hampered.

>> Right. So, so you're saying it is the dominant pole, but it's not the only pole that is constrained by the others. Um, one can respond to, to this and say that what we may say is true, but all the other waves of technology, whether it's printing press, industrial, certainly nuclear, web, PC, would also have benefited from philosopher builders. Why is there something specific about AI that that makes this more urgent?

>> I think that gets back to autonomy. Yeah. But it also, you know, more generally, I think there have been epical moments in science and tech that have made us question what it means to be human. And, you know, I think of Galileo or Copernicus and and and Darwin. I think about pushing us out of the center of the universe, putting us among the animals. Those were major, major reorienting moments in human life as a consequence of technological insight and breakthrough. AI is similar. The age of Turing brings this question of what does it mean to be human in a world in which we are no longer, you know, the most intelligent being, um, or at least plausibly, uh, uh, becoming so. And so that's one kind of broad impetus, broad strokes. But I think the question of, like, because AI substitutes for this essential, maybe central human good, then it becomes a very philosophical technology. It operates through language. It has a semantic, um, interface, you know, with, with humans, and, um, and has a mediating effect between us and the world, especially the world of words and information, right? Whereas it's just as important, maybe more so, you could easily make the argument, to get nuclear, right? Right. To get the, the game theory of nuclear, right? Um, it doesn't raise as many interesting philosophical, uh, questions and problems as AI does because of how humanlike it is. Um, and it's not manipulable through the semantic way, but rather through the numeric way.

>> Yeah.

I, I also think there's an argument to be made that AI is plausibly the end of the modern technological project. In other words, is plausibly a technology that can create other technologies or create other scientific breakthroughs. We, we are barely scratching the surface there. But if you go back to Bacon and the beginning of this whole modern scientific project, the thought of a technology that could discover other, you know, breakthroughs would have been held in a special category.

So I want to offer, uh, you a few critiques and see how you respond to this philosopher builder ideal. And the first one is a catalyst critique, and that's to say, look, uh, it's redundant. We don't need our builders to aim towards any higher normative end, uh, other than profit, because this is one of the key tenants of, of, uh, classical liberal thinking and markets thinking about capitalism, that the invisible hand of the market will turn private vice into public virtue. Uh, and so it seems like the urgency by which you, uh, uh, recommend the philosopher builder kind of undermines that, that piece.

>> Well, I think the market does turn private vice into public virtue by and large. I also think it permits private virtue. In other words, the market permits you to, um, act in a way that is, uh, yours, you know, as, as it is chosen, and you can, uh, do that in so far as you offer value to others. But what I would say here is like, you need to think about the two poles of kind of how an entrepreneur is seen to act in a, in a capitalist system. One is Friedman, who I think occupies the, the most narrow, farthest pole, 1970, what is the corporate social responsibility business? He writes this, writes this article, and it says that the, the role of an entrepreneur is to, uh, is to seek shareholder value, to deliver that. And the other end of the extreme is ESG, I would say, where you have a kind of unitary goal, a United Nations goal, for example, to do something that we, we view as being laudable, and then entrepreneurs are seen as the execution instrument for that goal. And I would say both are troubling in a sense. I'm much closer to the Friedman view, but they, they both position the entrepreneur in a low agency way, in so far as on the one hand, we're, we're, we're supposedly very limited in what we can do, we just think very financially about the shareholder value. And on the other, we're, we're seen as the agent of another who's implementing this, you know, this, this goal. And I would say there's an alternative that, and the alternative is the entrepreneur as a pioneer, the entrepreneur who sets the norms and builds that future. And there is simply nothing that restricts that. One needs to be creative. I would argue that's what entrepreneurs do today in many cases, um, but I would say one needs to be creative because the, um, customer may not, may or may not care about your underlying philosophy. And so I'll give you an example, is like, if we want to think about autonomy, there's a lot of stuff that we could do architecturally. But if we consider the case of the Bloomberg terminal, for anyone who's not in finance, is a kind of mainstay of finance, and the use of it is about improving decision quality. That's the, the point. When you use social media, often the point is to kind of check out, like that's the job to be done is like, you had a hard day and you just want to scroll. And so you need to think about that. You need to think about your business model as well. Are you paying money in a subscription, or are you being subsidized because the model is ad-driven? These are really important design questions. But in so far as opportunities do exist that resemble the Bloomberg terminal, where you're improving decision quality and getting paid for it on a subscription basis, there's an entirely consistent pro-capitalist way to build autonomy-producing tools.

>> Right. Right. Uh, so when I say the philosopher builder undermines a key tenant of capitalism,

>> I'm not taking the capitalist position

>> to be that you can't aim towards normative, uh, goals. But I'm taking it to, to say it's not necessary for the positive benefits to ensue. And yet, what I hear you saying is, if we don't have philosophers who are builders, then entrepreneurs driven by the profit mechanism can build systems that just turn people into automatons. You, you see the difference I'm trying to draw.

>> And I also, so, profit pools are incredibly important. They tell us what to go figure out about. They're, they're things that people value based on their preference, preference ordering, sort of, so to speak. But also, someone needs to care about preserving the institutions that make free markets possible. Someone needs to care about preserving the habits of mind that do that. And also markets are, while incredible, a means by which we attain human flourishing. And so someone also needs to care about that. And so those are things that are entirely consistent with free markets. It's a bit of a thicker conception of liberalism than I would say, you know, Friedman would hold. I grant that. But I think it's critical that we, we, we, we consider those other dimensions.

>> Right? So now let me push you from the exact opposite direction. Right? That was the pro-capitalist critique. This is the anti-capitalist critique. And I think OpenAI is a good example, which is, let's say there is an entrepreneur who is motivated by more than the profit motive. There are two kinds of pressures upon him or her that might deter his genuine desire to do this. Number one is when a company scales and brings on investors and other shareholders, they just might need to optimize shareholder value, regardless of what the entrepreneur, what the philosopher builder wants. And number two is competitive pressures. You might build a social media platform that genuinely helps people, that gives them Socratic dialogues, but people might just, uh, gravitate towards the addicting one, and then the profit incentive drives growth towards that.

>> Yeah, I would, I would add another, which is that for early-stage companies, which I think a lot about investors, the incremental financing, if you fund your company at a seed stage and you have a real mission behind it, but then the Series A investor doesn't care, they just reduce you to a metric because they're, they're chasing DPI, um, they are not going to be, um, uh, kind if you sort of deviate from a growth plan. And so you have to kind of, this is a big design question for, like, how you bring these companies into the world. You need to have a group of investors, capital that is

>> Aligned.

>> Aligned philosophically. And this is a big reason why we started Cosmos Holdings, which is a complimentary portion of this that focuses on venture creation and venture backing, is because you need to have aligned capital to do this. What you can't insulate from is the customer. In other words, like, right, you can provide insulation in terms of the capital and have very principled capital, but you can't insulate from the customer, which is why you need to kind of be inspired by the Bloomberg model, which is funny to say because in 2025, it's not exactly the greatest, you know, user interface or whatever. They, they beat their competitors though, because they've chosen a part of the market where it's real mutual benefit, like they're really offering benefit for their customers. You have to build in those kinds of pockets and directions.

>> What about the other pockets?

>> Well, so I think once you, once you demonstrate that, I think the costs and the design patterns will come down, and I think you will be able to kind of infect the other regions of the world, right?

>> I'm not sort of utopian in thinking, but I do think there's a path, there's a trajectory, there's a way to enter into this market, right?

>> And then improve and learn about this in such a way that we can kind of cross-apply our lessons into the more difficult areas. I also think that there's something to be said about, like, there's a great deal of entrenchment when you have one business model, like ad-driven model, that is, this entrenchment is not felt by new challengers. In other words, if you set up your business to monetize entirely differently, then you don't bind yourself in the way that Google has bound itself.

>> Like Substack, for example.

>> Substack is a great example. Yeah. Exactly. And so this is a challenge for incumbents. I think if I were an incumbent trying to drive change there, it would be hard. And that's why a lot of the top researchers that we know are people who worked at, at big tech platforms, who built something that was incredibly, I think, potent for, you know, promoting human flourishing. It went against the business model for them. The right answer is you need to either build it, you know, in academia and open source it, or you need to start a startup and find a business model that can take that into the world.

Let me give you, uh, a different line of critique. Um, and I'll begin by quoting, uh, part of an essay that we are soon to publish. "The way to acquire more stable views is almost paradoxically more inquiry." Okay, you're explaining why it's important for the builder to philosophize. In Plato's Meno, Socrates describes this with reference to the legend of the statues of Daedalus, which were said to run about if not tied down. The idea being that opinion is made valuable via inquiry, which helps to ground our knowledge and holds it more stably in place. Inquiry improves our convictions even as it replaces them. You're talking about here the importance of the builder, many of whom in Silicon Valley are just, uh, unthinking in their building, of philosophizing, that one of the benefits is that it grants them more conviction. However, many of the, of the, uh, Socratic dialogues end in appar, um, uh, sort of answer to the "what is X" question is arrived at. And even stronger, for some of them, the interlocutor seems to be made worse, more puzzled, more angry, more humiliated through the conversation. So, and of course, there's an entire school that's been founded from Socratic aporetics, right?

>> And so, why are you so certain that kind of philosophical questioning is going to lead to more certainty for builders who are even considering to take this path?

>> Yeah, it's an interesting question. So the, um, the kind of thing that we want to do is to inculcate a habit of mind. We want people who inquire, who think deeply about the alternate possibilities, who understand that philosophy, the love of wisdom and the pursuit of wisdom is a kind of quest for knowledge, but never an attainment, and who are satisfied to some degree by simply knowing more about what they don't know. In other words, we can, we can look to, I mean, a lot of the most inspirational philosophy for me is people who have un, who have tried to demarcate the limits of reason, what we can know. Yeah. Exactly. Kant, Hume, many others. Hayek. And so I think that's a perfectly acceptable place to land in philosophy that requires a, um, one to have a constitution that isn't like pathologically certainty-seeking. But if that's true, then we would rather have questioners who are constantly questioning, which, by the way, is a habit that I think is very consistent with company building. It's like, you're asking questions, you're deeply curious, you're fostering that. It's just you're asking more capacious questions, more expansive questions that, um, get at what the technology is actually going to do for what matters, which is human flourishing.

>> Right.

>> Um, what do you think is the ideal path to train these philosopher builders? Is it to take builders, such as yourself, uh, and then try to teach them philosophy when they're ready, or is it to take the best philosophers, uh, and teach them how to build?

>> Good question. So my focus is to try primarily to take the builders who have this kind of sense that they want to help humans, and just like I did, but it's untutored, just like mine was, and then be able to give them tools to, um, really try to derive a set of principles or ideas. The method I think must be a combination of education, and that education must blend the kind of textual and technical meaning. If we want to think about collective intelligence, we might read Mill on how to correct collective error, but we should also understand research at the very front, you know, very forward parts of the frontier, like what Midjourney is doing on collective intelligence. That's one part of it. The other part is practice. It's action. It's translation. So you can think about translation in two senses. One is how do you create prototypes that test ideas? And for that, what we do at Cosmos is we back, you know, individuals to build projects in 30, 60, 90 days with micro-grants, inspired by Tyler Cowen, inspired by Emergent Ventures. The other method is to do deeper research. And so for some questions that deal with, like, the heart of what is inquiry, or what it, what does it mean for a machine to, uh, promote virtue? These are big questions that I think prototypes would be valuable for. But there's a much deeper kind of question, right? And so for that, we helped to, you know, um, facilitate this by setting up an AI lab at the University of Oxford called the Human-Centered AI Lab, where you have some of the top philosophers, Philip Coralus leads this, um, he's a very unique philosopher who thinks about reason, but you combine that with people who are fresh out of OpenAI, Anthropic, DeepMind, that kind of thing, and they're building systems and doing research that I don't think could be done anywhere else. I don't think it could be done within, I don't think it would be done within academia in a traditional way. So you kind of put that alchemical set of philosopher and technologists together, and you do incredible research. Then, as I mentioned, the last part of it, which I still view as part of, like, this journey of transformation, is you kind of graduate and you build a company. That's what makes this so distinctive is that, you know, we want to take ideas and then scale them out into the world. And the way to do that is through markets, is through entrepreneurship. And so that's where Cosmos Holdings comes in.

>> Well, it's really interesting because, uh, most of the people we interview in this series, the really successful philosopher builders, all went the opposite direction. They all studied philosophy. Reed Hoffman, Peter Thiel, Colin Moran, uh, Marcus Aurelius, and then they were pulled into the real world. Yeah. I'm not saying there's a necessary tension there. I, I just think that's, it's very interesting.

>> Yeah.

>> Absolutely. Well, I mean, I think it hits people at the right moment for them. So, I don't mean to dismiss people who come at it from the other direction. We're very welcoming of that. I know for me, I took one philosophy class at MIT. I didn't like it that much.

>> Yeah.

>> And then it had to be the case that I found it in my, you know, mid-30s after selling the companies, after having the kids. It took hold. So, it's very unique the journey that each individual.

>> And, uh, same for me. It could just be be a matter of months or even a single year where for me, uh, I took Columbia's Great Books core curriculum after I'd sold the company, dropped out, and went back. Um, and that was life-changing.

>> Yeah.

>> If I were forced to take it in my freshman fall, I think I would have hated it.

>> Yeah.

>> Because I didn't realize the importance. So, yeah. Uh, let me end this interview with a final challenge, and it is a challenge from an unsuspected friend. Hayek. Okay. This is, uh, Constitution of Liberty. "Coercion, however, cannot be altogether avoided because the only way to prevent it is by the threat of coercion. Free society has met this problem by conferring the monopoly of coercion on the state and by attempting to limit this power of the state to instances where it is required to prevent coercion by private persons." So for our audience, what Hayek is saying here is that if you don't want individuals, private citizens to coerce each other, you need to have the ultimate form of coercion, or at least a singular form of coercion, which is the monopoly of violence of the state guarding these rules. Might there be an uncomfortable structure as it relates to building non-coercive AI in the sense that right now with Cosmos and philosopher builders, you're going for the non-coercive approach to build that, right? They're saying, "Let's, let's give people these fast grants. Let's build these, these, these, uh, these companies. Let's let markets do their trick." But, you know, if someone is about to build an AI and push it on the masses that is going to turn them into automatons, might we need a coercive measure to ensure that coercion does not happen, either through regulation or other means?

>> So, what Hayek is identifying there, I think you could call the paradox of government, which is to say, you've got to have a government that is sufficiently powerful to defend, to defend liberty. That invites, uh, huge difficulty if you grant a government that much power, that monopoly and violence, as you said. But it's, it's, it's essential, um, that we live with this paradox as it concerns regulation. I would call to mind three, three tests. Um, one is the test of, is that consistent? Is it consistent with the rule of law? In other words, with the idea that each, um, each law needs to be general, abstract, and perspective. That's the first. And so there are lots of ways in which regulation can, can violate that and be commands to specific groups and things like that, that violate the generality, so on and so forth. So that's a baseline test that again draws from K. The second is, is the regulation something that is made based on knowledge that we have no reason to believe that we possess? In other words, if we had made regulation at the beginning of the internet era, would we have gotten it right? I can tell you we would not have. You know, I, I can tell you that we would have been in profound ignorance about what was to come. We would have gotten it wrong. And this gets at the idea of, um, the dominant paradigm, which is ex ante, trying to regulate before some hypothesized harm occurs, versus the much more adaptive, incremental, evolutionary approach, which is ex post, through the common law. So ex post adjudication, I think, should be favored on an epistemological basis because we can't make such predictions. Thirdly, any intervention that we might have should be evaluated not just on the basis of the local cost-benefit. In other words, if we, if we intervene here, and we intervene here, and we intervene here, then this proximal harm that we can, that we can foresee, we may not have experienced it because, again, we're not taking the ex post approach, but if we can foresee it and articulate it, is it sort of worth it? And the cost-benefit analysis that really, I think, we should do in that case is don't just look at the harm that you're trying to solve proximally. Look at the overall harm to the system. And so if you, if you buy the idea that the system is just, especially because that tends to allow the anonymous individual to achieve his or her unknown ends, then you have to be committed to some idea that we care about the way in which knowledge propagates through the system and gets generated through the system. And in fact, our interventions tend to harm that crucial, crucial attribute of spontaneous order. And so that is something that needs to go on the other side of the ledger, right? And this is a good place to end it because it mirrors what you said about the trade-off function of autonomy in the good life, which is that it's not a lexical priority. It's not like no regulation whatsoever. It's that simply people underestimate the cost of doing business, so to speak, right? Of intervening. All right. Thank you, Brendan. Thank you for a fascinating conversation.

>> Thank you.

>> Thanks for watching my interview. If you like these kinds of discussions, I think you'd fit in great with the ecosystem we're building at Cosmos. We deliver educational programs, fund research, invest in AI startups, and believe that philosophy is critical to building better technology. If you want to join our ecosystem of philosopher builders, you can find roles we're hiring for, events we're hosting, and other ways to get involved on jonathanb.com/cosmos. Thank you.