The AI Use-Case No One is Talking About | Reid Hoffman

The killer app of AI? It's not going to be Chanty PT. We'll start with an AI girlfriend to help you actually get to a real girlfriend. How do I successfully invest in the consumer internet? I invest in one or more of the seven deadly sins.

What drove the earliest internet? Pornography. It's like, ooh, it it's all lust. The central thing of the meaning of life is is friendship. Do you think one can be friends with an AI agent? Well, what happens when we're in the sci-fi universe and robots are doing all the work and there's no work anymore and what is our meaning? There is this. Yes. That that's still there at the very least. At the very least, and it matters. You're giving me an idea. I'm curious how it'll compete in the market.

What if you had Reed Hoffman is the greatest social technologist of our era. His first company, Social Net, was a social media play. He then built LinkedIn and PayPal and invested in Facebook and Airbnb. Reed argues that just like the internet, the true killer app of AI isn't going to be single-player chatbots, but multiplayer social. This interview is going to explore the full scope of AI social, from how it will transform traditional social media to what friendships and romantic relationships with AIs could look like.

Now, as someone who is quite critical of social media, I came into this interview dreading a future where human relationships are replaced by non-human agents. But Reed gave a compelling account for why this technology has the potential to make us more and not less human. My name is Jonathan B. I'm a founding member of Cosmos. We fund research, incubate, and invest in AI startups and believe that philosophy is critical to building technology. If you want to join our ecosystem of philosopher builders, you can find roles we're hiring for, events, grant programs, and other ways to get involved on jonathanb.com/cosmos. And without further ado, Reed Hoffman.

What is the social killer app for AI? People always tend to think, because of chatbots, of AI as one-to-one interaction. So it's like me with my chatbot, like I ask it a question. It's kind of like Google search, etc. And actually, in fact, one of the things that's going to happen within a small number of years is we are going to be in a surrounding field of agents. We're going to have agents listening to us and, like, for example, when we have a conversation like this, we'll have agents that are going, "Oh wait, Reed, when you made that comment about Rouso, that wasn't quite right, you know," and we'll kind of blink and say, "Hey, do you want to interact with me on this?" and so forth and and so we'll have agents um kind of in the field around us, not just for us as individuals but for us interacting with other individuals, but also interacting with groups, interacting with societies; it'll take some of the things that are currently more invisible to most people, which is the networks we live in and all of that kind of social interaction will now have a mediated kind of field of agents.

Yeah. And what the exact shape or topology of it is, there's a it's almost like complexity theory. We can't fully predict.

Right. That was an interesting answer, but a very different answer than than what I thought you'd give, because your answer, to summarize for the audience, is that it will mediate our social interactions in the same way that LinkedIn mediates and, right, our social protection today. I thought what you were going to say is that we're going to have perhaps direct social relations with the agents themselves. That is how I read at least your attempt at Pi.

Yes. And uh, building the company Inflection, right, which you you said the the difference is that Pi is trained on EQ as much as IQ, right? So so tell us about that. So we will have direct social relations with agents, but actually, in fact, part of the reason I gave the other answer is because the anthropomorphification of agents is one of the things we have to be careful about, and Pi is a perfect example of it; we train it for kindness, compassion, we train it for helping you and being your companion; in doing that, that's important, but for example, if you go to Pi and say, "You're my best friend," Pi will say, "No, no, no. I'm your AI companion. Let's talk about your friends. Have you seen your friends recently? Maybe you want to schedule something." So, it it doesn't want to displace your human relationships. It wants to be in the panoply of kind of social interactions you're having. And we're going to, by the way, have to have a new kind of social vocabulary because your social react uh engagement with agents is different than your social interactions with your friends, your colleagues, etc. And even like therapists or doctors, it's a it's a different, you know, interaction. So, for example, one of the things we have to be careful about is if people are interacting with agents, like, for example, the most classic one is Alexa, and it goes, "No, stop." Well, you don't want to start interacting with other human beings, "No, stop." He's like, "No, no, no. That's that's not the dynamic we have." So, we're gonna have to have a a a more rich kind of socialization thing. So, for example, part of the thing that you're going to want with agents that are interacting with children is you're going to want them to be attentive to socialization. So, you're not going to want them to teach them to be rude or aggressive or preemptive or, you know, since we're talking philosophy, like the master in the Hegelian master-slave dialectic. You you you don't want to train that way. But by the way, the social interaction a child's going to have with an agent is different than the social interaction they're going to have with their parents or other kids or, right, or or and so this this this richening of social experience is going to be important, and Pi is like one specific cast right now with today's technology to try to put us on the right path.

Right. So again, I'm somewhat surprised by your answer, but in a way that I'm relieved, right? Cuz cuz the the worst answer you could have given is

Yeah. Like in the future, your your friends will probably be your phone.

Yeah. Exactly. But this is why I'm surprised. Okay. So I'll give you a quote from your new book, Super Agency. "Billions of people say they have a personal relationship with God or other religious deities, most of whom are envisioned as super intelligences whose powers of perception and habits of mind are not fully discernible to us mortals. Billions of people forge some of their most meaningful relationships with dogs, cats, and other animals that have a relatively limited range of communicative powers. Children do this with dolls, stuffed animals, and imaginary friends. That we might be quick to develop deep and lasting bonds with intelligences that are just as expressive and responsive as we are seems inevitable. A sign of human nature more than technological overreach." When I read that paragraph, I thought the natural conclusion was to say, "Look, we form strong relationships, sometimes stronger than humans, with imaginary nonhuman uh uh entities all the time.

Yeah. And so AI agents, shouldn't AI agents be even more natural?

So the point of that paragraph is a lot of people have this reaction, like like if you you go to most of the dialogue around AI right now, it's uncertainty, fear, negativity. And one of the pieces of negativity is, "Oh no, it's distract it's destroying our human relationships because now suddenly we'll start forming these human relationships with this other thing." And the point is to say, actually, in fact, we as human beings anthropomorphize broadly. We anthropomorphize a car, like, you know, my mom calls your car by name, right, and says they're there, you know, pets and a whole bunch. So it's we have this whole range, and that's the point is to say, "No, no, no, no, this is not this alarming new thing that is the the the collapse of human relationships, the collapse of human society, the collapse of human agency; this is just another entity, another entry into this panoply of relations we have." It wasn't meant to generalize to, "Well, therefore, therefore, everything we we interact with is how we interact with humans," that that that's unsophisticated, and by the way, that's a danger because people can have unsophisticated relationships. I mean, for example, one of my aphorisms that I tend to challenge is, you know, "dogs are man's best friend." It's like, "No, a dog's not your friend." Simple way is you wouldn't say, "Oh, I'm taking my friend down to get spayed at the moment," you know, with their involuntarily on their part. It's like, "No, that's not the way friendship works." You have a a wonderful relationship with your dog, right? But it's not your friend. And so that kind of thing of of of this sophistication of relationships is one of the things we have to continue to need to evolve. And part of the point there is that's not alarming that we're going to further elaborate the sophistication of our relationships now with the most interesting technological thing we've created in human history.

Right. I see. Another way to put it is the mere fact that we're forming some kind of sociality is not a cause for alarm in the in the in the way that all of these are not a cause for alarm.

Yes. But as all these examples go to show, it can become pathological.

Yes. If you're 18 and you still have stuffed animals as imaginary friends, right? Or there's there's ways to do religions in a very destructive way and in a very constructive and nourishing way. So that's what you're trying to get at, which is that we these agents are going to enter into our social world in some way.

Yes. But there's a way to get it right. That's what you're trying to get at. And we should pay attention to it and steer it, right? Because that's part of the reason why like a lot of the stuff we were doing with Inflection and Pi is our kind of first-cast mode to try to catalyze other agents. And like, for example, we've seen Anthropic's Claude now do a lot more EQ things. I think that's a really good thing. So being EQ is good, but like to to play the right role in individual humans' lives and humanity's lives, right, and and that is not not social.

Yeah. But is also not like you, right? So tell us about how you trained EQ into this model.

Yeah. One of the the really key thing is is is reinforcement learning with human feedback. And so part of what most people don't realize and how the reason why there's different chatbots and why ChatGPT is different than Pi is different than Claude is different than Gemini is different than you know Copilot is different than is because uh a bunch of the human factor training is actually different, with different instructions for the human instructor and which examples you give them and just for our audience, reinforcement learning very loosely is you kind of like carrot or stick depending on the outcome and nudge them to the right direction.

Yeah. Yes. And so you you both who you train as the human trainers, how you instruct and train the human trainers, and then what examples you give the human trainers all gives a very different character to the kind of modern generative AI systems. And so we said, "Well, we want to give examples specifically about which uh interactions are more kind, which interactions are more empathetic." Like even if you go Pi, you blah blah blah blah, it doesn't go, "Oh, what do you mean, you..." It goes, "Oh, I'm really sorry."

So, don't don't train it on X.

Yes. Exactly. Like, like it goes, "Oh, I'm sorry. I'm sorry if you're angry. I'm sorry if I've done something to make you angry. You know, what's the thing I did to make you angry?" You know, like that kind of thing. And by the way, that's of course how we want to be modeling because, by the way, we do generalize as human beings. We have this social interaction over here. We generalize. That's what we want to be having as more human interaction that actually helps us be our better selves.

So this last point, what you're saying is even though we don't expect and shouldn't shouldn't want the social relationship between the LLM to be the exact same as humans.

Yeah, the way we are taught to interface with it gets translated back in our real, for example, X is a good example in contemporary politics, right? And so how have you noticed users mostly use Pi? Is it mostly as a not a therapist replacement but but as like when they're struggling through things?

Yeah, so it's been a whole range we've seen. I mean, some of the startling ones are like, for example, getting reports from like a married couple saying, "Hey, when we're having a difficult conversation, we put Pi on audio and and have it participate in our conversation of, 'Oh, well, I think when Reed said that, maybe he was meaning this,' you know, that kind of thing. So back to the kind of the AI as a field um but by the way, it's even we have people who use it for information who prefer an emotional, empathetic interaction, right? So they go, "Okay, fine, maybe I'd get better information from ChatGPT, but I like this interaction; it makes me feel good as I'm interacting, and that matters to me in the interaction," right? So it's a whole range.

I see. Um, but you guys pivoted away from Pi, right, Inflection? So tell us about the reasons there.

So we said, "All right, what's going to be the driving determinant of the of of where a lot of of the agents that will get to hundreds of millions of people," and we thought it's going to be it's going to be uh built on growing scale compute, and fundamentally, there's not really a room for startups in doing that.

Right. Right. And so if you're in the business of foundational agents, you better have a war chest. And so we're like, "Okay, that means our original, and this is classic for business for startup businesses, our original plan won't work. What can we do?" Well, actually, in fact, this empathetic EQ, there's going to be a whole bunch of businesses that are going to care about that. We can pivot Inflection to being a B2B. It has a unique Inflection model. It can also use other open-source models and can work with folks who go, "I want an empathetic agent in interacting with my community, my customers, you know, with me and other folks," right? But wouldn't that still require you training a model from scratch or or is it just a series of prompts that that you you you add on to the other model?

Well, I think agents are going to be combinations of models, and we can still do, you know, I see this is the emphatic center of the brain, so to speak.

Exactly. And so we can compose for the different business, you know, kind of customers. So now Inflection or Inflection is a B2B business of which one of its unique attributes is it has the unique Pi agent, right, but it can it can it also works with other open-source you know models as you like. So even though you're still training from scratch, you don't need to compete with OpenAI on research, for example, and so there's different modules, almost like modules of the brain that they can call, like when we need empathy because a customer is mad.

Yes. So, "I'm really sorry to hear that that that that today is a bad day. How can I help?"

I see. I see. So, that is your positive vision that that these things are going to help us form even better relationships with people in uh in the real world.

Yes. Two concerns. Number one is Replika, right? This is for our audience, it's this dating dating bot. There were some people who who were like emotionally traumatized in early relationships and did use that and did forge good you know real relationships, but there were a lot of people who just ended up like dating their Replikas essentially um and you see this for example in video games. I've benefited tremendously; I was a huge World of Warcraft, Guild Wars, I Hearthstone in my life; my strategic thinking skills and a lot of entrepreneurs I I meet, we develop them through playing these strategic games.

Some people just get stuck at them.

Yes. So so that that's the first concern that that that a certain group will be benefited by having these agents translate into the real world; some of them will just be stuck. How do you respond to that?

So, some of them will be stuck, right? That's brass tacks, and the question is if it's a large enough percentage then we need to do something to kind of intervene, whether it's regulatory, market, other kinds of of of influence. It's one of the things where as opposed to preempting it, we should see where it kind of naturally sorts out. Now, obviously, we want to be more preemptive when it comes to children, right? Because children kind of forming, but once people get to adults who say, "This person really just wants to have a digital AI girlfriend," well, by the way, we we allow freedom and choice, right? They can do that. Now, we want to nudge them in certain directions. Now, that being said, I do think that the majority of human beings naturally want human interaction. The question around, you know, can you get distracted by like kind of going into a deep dive in any particular technology, AI, girlfriends, video games, social media, social media, everything is can get stuck. You can get stuck just by religion; you can get stuck.

Yeah. Exactly. So then we have to figure out how to try to nudge that out. And I think that think that's worth paying attention to and worth doing something about when we see it in substantial numbers.

Right? So the fear is genuine and warranted about this forming this dependency, but we just have to figure out ex post, like like parachute parachute on the way down. Generally speaking, the mistake is everyone comes to go, "Let's prevent any bad thing from happening." Right? That's not possible at large scale. Right? It's not possible for cars. It's not possible for planes. It's not possible for electricity. People still do get electrocuted. Right? We go, "Okay, we can't we can't prevent anything from happening. What we should do is say, 'Let's list the things that we think could be bad. Figure out the measurement metric, right? Start measuring. And if we go, oh, look, that's growing. And that's growing in a way that that's going to come. Okay, we see that. Now, let's what could we do to possibly fix it?'"

Right. Okay. So, here's the perhaps more difficult challenge that I'm going to present to you, which is the historical view, right? And that's social media.

Yeah. The same claims were made in social media. "This thing is going to join the community, elevate truth, this is going to bring it like heaven on earth, like like governments are going to be uh so attuned to to the the the democratic will of the people." I don't think that's what's happened. I think dating apps is the easy case. It's made a lot more people a lot less romantically capable, at least in early stages, and social media has made a great majority of people, I think, a lot less social or sociable in the wrong ways. Do you disagree with that historical kind of view that...

Well, I'd be curious what the actual percentages are. If it's 1%, okay, that's an issue.

Yeah. Right. But but if it's 20%...

Well, that's an very alarming issue, right? Right. in terms of of where it's functioning as as as we'd aspire to be and where it's functioning as not as we aspire to be. Now, in the social networking, my usual quip is, "Hey, I did LinkedIn. LinkedIn works pretty well." And I agree that there's corrosive issues um in both X and which I think is like, you know, kind of a says pool, right? And uh in Instagram and in Facebook.

Yeah. Um, but I think you have to think of not just the individuals as customers, but also society and groups as customers.

But that's even worse on on the on the on the X and Instagram case.

No, no, exactly. And so what I would want them to do, like if I were in a government role, I'd be saying, "Okay, show us how what give us health metrics of the network, right?" And like some of the health metrics are, you know, how much are people getting enraged and polarized, right? And and then what I'd like to know is what you're doing to try to soften that. Some people would wave a wand and if you could kill them and just make them go away, right, they would say that's the right thing, right? My view is there's a bunch of positive things too, right? Even on Twitter, of course.

Yeah. Yeah. Um, but what there is is a bunch of negative things, and there's enough negative things just by clear observation, we should be doing something about it, right?

Here's my my my second pushback. You were making it sound like, "Look, if we did an empirical study, if we have a metric, if we have a benchmark, we can nudge these people in the right direction." You can. But what I'm trying to point out is that there seems to be a systemic issue within human nature. Let's take the LinkedIn example.

I agree. LinkedIn is the least gross of of the social networks.

But and to be totally honest, it's because I'm not addicted to my LinkedIn.

Yes. Because I'm not addicted to the the hateful drama.

Yes. Like on X or it doesn't appeal to my vanity as much.

Right. LinkedIn is just a useful tool. So there, kudos to you. By the way, we do try to appeal to your vanity just to be fair.

Right. Right. But not not as successfully; stick around. Think about it on the micro view. Like if I'm tweaking the X algorithm or the Instagram algorithm, the more I can get people upset, the more successful my social media thing will be in the whole realm of social. So I'm pointing to a systemic issue, not just uh you know the will of Elon or the will of Zuck and then they can just tweet these things. How do you respond to that?

So baseline, I agree. In 2002, I started saying, "How do I successfully invest in the consumer internet? I invest in one or more of the seven deadly sins," right? Which is hence vanity, right? You know, greed, wrath, wrath, which and by the way, one of the things is I made a mistake. I thought Twitter was I did was vanity. It's actually wrath, right? Or some combination. Now, what you should try to do as technologists in building it is you don't wallow in the sin. You try to sublimate it. You try to elevate it. Like it part of what makes these seven deadly sins is that they are things that are in all of us, right? It's kind of this this thing where it is part of the human nature. But what you want to do is say, "Look, transmorph trans you know transmorph it into um into things that help you become your better self." The thing that I'm most worried about in social networks is if all I'm doing is following your click, your click will naturally cause you to do things that are in the seven deadly sins, like anger and because it's like ah, you know, it's like, "Well, what drove the earliest internet, you know, like pornography," right?

Pornography.

It's like the earliest thing is like it it's all lust. But by the way, just looking at the internet, it didn't stay there and actually grew massively into all the other things. It's growing past that. That's the important thing, and that's what I would want to see in these networks because yes, we have both virtues and vices in our human nature, and vices tend to more naturally um come out in mass when it's groups. So part of when I get to the more sophisticated as society as a customer is say, "Well, actually, in fact, we're not just responsible for each individual about how they're engaging the network but also how the network behaves," right? And that's what I think we want to start getting to now that you know the networks are you know all around us. It's very straightforward; say, for example, what you did is you said, "We have this measure of agitation and anger," right, and we go, "Oh, we're going to start decrementing some of the sharability of certain of the posts."

Yeah. Because these are ones that are just agitation and anger inducing, right? But the systemic response is to say, "If I reduce that, maybe they're not going to be willing as to return," and and then to extrapolate this to AI, you know, Pi is great. It doesn't make me want to return as much as I mean I I assume if I had romantic issues...

Replika.

Yes. Right. And so the systemic concern is to say, "Look, you can guide people out of their vices as much as you want. That's great. Good for you. But is that you're a student of human nature. How many humans, what percent of humans are going to want to wallow..."

In their vices, and how many percent are going to want to do the difficult task of sort of climbing out of that? Right. Like people will always want to wallow in their vices in a significant percentage. And as you're pointing out, there's commercial incentives for addiction. Right. Like what, what would be like part of the reason why we have, um, a lot of regulations around drugs? Is you go, "Hey, we can just sell over-the-counter happy pills," you're going to have massive double-digit percentages of people, yes, just just buying it. Right. Right. No, no. Part of the reason why we put in some regulation here is because we're like, okay, we need that to be like a small percentage, not a large percentage. And that's the kind of—I think it's—it's good to have that discussion about social networks and about iPhones and about, you know, cable news and about anything else. And by the way, we do have ads, and so but but when you have the right kind of company culture and the internal metric set that actually gets you the right place because you don't need every little penny and dime. What you need is what—what are we providing that's valuable over a long time? And sure, of course, we'd like you to come back more to LinkedIn, but from day one at LinkedIn is we are timesaving, not time-spending. Right. I see. Right. We want you to accomplish the thing that's useful to you in work. 'Cause everything you do in work, you say, "Well, if I can do it in a minute versus an hour, I'd rather do it in a minute." Yeah. Exactly. Right. That—that's part of what work is. Yeah. Exactly. Right. And so—so that's the LinkedIn design philosophy, right? And so you go, "Okay, that's the metrics that we have internally. We don't go, 'Oh, if we just had you clicking on videos,' because we do have videos now, but it's just clicking on videos. Ah, that's not the metric we want." Right?

So this is the, I think, optimistic view, um, that still fully recognizes a trade-off for log and AI agents, which is, of course, you can all imagine people get addicted to their AI girlfriends. But for the skillful entrepreneur who has the right philosophy and view, who is very careful about designing a company, not just the culture but also the economic incentives to properly align with the users, there's no theoretical reason why that can't be the one that wins out versus just the the vice-indulging one. That—that's the optimistic. Yeah. Yeah. Actually, it's funny. You're giving me an idea. I'm curious how to compete in the market. What if you had—so companies X, Y, and Z doing just Fallout AI girlfriends, and what if you had company A that was going, "Look, we'll start with an AI girlfriend, and then we'll get to a real girlfriend." That is actually, in fact, what we're trying to do. That is our brand promise. Right. I wonder what it would do like in the market, 'cause that would be obviously the better path if—if one is going down this path. Yeah.

So now I'm going to push you in the exact opposite direction. Great. Which is, what is wrong about forming most of our relationships with AI agents? What about human recognition itself is so precious? Go back to the kind of Hegelian master-slave dialectic. Right. So you know, in Hegel, it was like we're first born as a consciousness. We think we're kind of a god in this world. Then we start encountering other things that aren't objects to our use, and we first try to enslave them to be objects to our use, and then we eventually realize that they're not actually objects or other subjects, and we get into an inner subjective balance of identity. That's a very quick, very simplistic description of phenomenology of—of Hegel's phenomenology of master-slave dialectic, but part of that, which I think he drives from observation of human beings, is we—we benefit enormously from these kind of, you know, challenges and frictions and diversity—iversity of other people; people who disagree with us, people who add something to how we become, you know, kind of our better selves. And sometimes it's like a—a disagreement in worldview and ideology. Sometimes it's a—it's a—it's an understanding of how to be more empathetic with other people. Sometimes it's a, you know, kind of different experiences that you have in different cultures, different walks of life, the people who are—who are—who are less privileged. You know, all of this, you know, is an important part of how we evolve to be our better selves. I'm always looking for—um—like when I'm interacting with serious people, what I want to do is, where are the areas we disagree? Where is the area something you—you either would change my mind or expand my mind or—or teach me something new? That's the thing you're getting to. You don't want reinforcement. Yeah. Only some reinforcement is good, but you want that expansion. And human beings provide that in very good ways. Now, you can say there's no reason you couldn't provide a whole bunch of agents that are doing that too. And by the way, we should have agents that are doing that too. But part of our raw experience, when it gets back to this Hegelian, you know, kind of dialectic, is that we grow as human beings by interacting with other human beings, right? By interacting with our siblings, our parents, our family, choosing our friends, going to school, teachers, colleagues. And that's one of the things that is kind of the journey that we're on. And I think as far as we know, that journey is really essential, right?

So let me keep pushing you here because I could say, "Well, that's quite an anthropocentric view that—that YouTube's given me," in the sense that let's break down what the—the master-slave dialectic actually requires. Right. So—so you know, for our audience, um, there's a struggle unto death. Right. Two people all want to be masters, uh, you know, and in the case one becomes a slave, he surrenders—I'm speaking very loosely here—the master wins the recognition of being a master because he won this struggle to death. But what the master finds out is suddenly that the recognition doesn't matter anymore because the slave is not independent, and the slave does not have the—the moral worth to properly recognize me as a master because he's a slave. And so suddenly that recognition is no longer appealing. Now, what you eventually find out in Hegel as this dialectic develops is—uh—you want a source of recognition that is—that has the kind of moral standing—we might say loosely—equal moral standing to you—to—to recognize you. Okay. So—so that's—that's what the phenomenology of spirit teaches us very loosely. But I don't see why not an LLM could provide that independent. Right. They're not—I mean, ChatGPT is already not just agreeing with me. It corrects me, and it also has a kind of expertise and maybe in the future a kind of wisdom that I learn to respect. So I'll give you an example. My friend already feels more validated on technical questions—he studied math—when a 01 pro model affirms him than your average Joe on the street, as he should. Yes. Because 01 pro is—is better at math than most people already. So—so that's—that's the next push. Like, what about human nature is so essential to that the recognition is irreplaceable?

So the complicated answer is we only discover which parts of human nature are essential as we go down the road. Like the—the attempt to today say this is human nature and—and it will never evolve and that—that techn—is—is a foolish—right—a foolish anchor. And it's one of—well, it's one of the reasons why people like—you know, call it five to eight years ago. Well, we're the ones who think and speak language, right? Well, this seems to be doing something approximating thinking and certainly doing something that—that is speaking language, right? Well, oh my god, this—and this is part of the reason why people go, "Oh my god, is—is this the end of humanity and so forth as a way of doing it?" And then you interact and you realize, well, no, actually, in fact, it could speak language, and I still have a bunch of things, and you know, there's a set of things today, um, that are unique to human—human interactions that are not in these. And you say, "Well, it can change," right? Yes, it can change. We are on a journey of discovering which things are special about humanity. Right. Right. It can start with we're social creatures. Oh, we're not the only social creatures. It can start with we—language—language. We're not the only linguistic creatures. And—and we really want to be special. I don't know if we will—like we will always be happy with what the future discoveries are of kind of where we're kind of like Darwin in the religious world. Right. Yeah. Exactly. Where we're fully unique. But the metaphysical or spiritual, you know, arc of humanity is—is—uh—evolving consciousness in the universe. We do know that we have it and it's important, and unless we find that something else was—was kind of like better at that or like completely better, which it may never be, then it's important to preserve humanity. And so, for example, they say, "Well, what happens when we're in the sci-fi universe and robots are doing all the work and there's no work anymore, and what is our meaning?" And you say, "Well, you look at like medieval Europe and nobility and like—like all the—the—the—the surfs and peasants, and we're all equivalent of the robots. Well, we still did theater, and we still did dinner parties, and we still did like—" So, it's possible to still have a—a—a rich human existence, and maybe that will be what's important and special about humanity.

This is what I love, um, about this recent wave of AI, which is that has turned what were previously just philosophical questions into a kind of experimental science. Yes. In the same way that physics did this in the 20th and 19th century for metaphysics, right? Like whether—so I studied recognition theory. This is what I focused on in undergrad, but this was just me and some other like—like nerds in a room debating about this stuff. But—but we are going to find out quite soon whether it's only human recognition that—that matters to us. And that's why I'm so excited about this wave is that—by the way, I think part of the thing is we as human beings seek that unique place. We could say the notion of—of traveling through life together and that co-experience and the way that we create together and the way that we understand the world together, and that itself almost by definition only we can be doing—there's at least a unique space there—may be much broader—but there's at least a unique space there, and so that's the reason I don't have the alarm that some people have of, "Wait, when this is created, are we going to be completely outmoded?" It's like, "No, no, we'll at least have this, and probably going to have a lot more." Yeah. Yeah. What I loved about this is it reminds me of—uh—Rousseau's defense for marriage. Yeah. As an institution because—uh—Rousseau believed that—uh—we naturally desire to be the best. I'm oversimplifying here, but obviously, you know, we can't all be Reed. We can't all be the—the best in—in our fields because, you know, that by definition—but Rousseau thought we could all—even if we couldn't be the best for everyone—we could all be the best in someone for a way, and that's what marriage is. Yes. When you're marrying someone, you're saying you're the best, at least the best I can do for me right now. And that's what you're kind of saying is that we—there is still a special uniqueness that is very important and central to us, even if it's not the—we're the most intelligent creatures in the world, or we're created by God, or we're the center of the universe. Modernity, in a way, has been a series of humblings for—for humanity, right? Darwin, yes, revolution, and now our intelligence is being threatened. But the optimistic view that you gave, which I love, is that there is—there is this—yes—that—that's still there at the very least. At the very least, and it matters. It's not something that's like, "Oh, no, no, no, this is why," for example, you know, part of what I think the central thing of the meaning of life is—is friendship and—and choosing that friendship is part of that. Right. You defined friendship in one of your previous interviews as—um—making each other better. Yes. In that definition, do you think one can be friends with an AI agent?

Well, I think one can have a relationship where certainly—uh—the AI agent is making you better. Yeah. And there might be something where you're making the AI agent better too in some interesting way. That—by the way, if that's really happening, then that begins to get into that question of, "Oh, is that a sentient being and so forth," not just like, "Oh, it's training on lots of data," but actually, in fact, it's this consideration of my moral character is evolving. It's precisely the reason why we—like part of modern humanity is we believe in such a thing as animal rights for kind of similar kinds of—of—of—of beliefs. And so I think that's possible. I do think that the nature of that friendship will probably still be—like our earlier society discussion or social discussion—will be interestingly distinct in some ways from the human—from the human-human—right—but I do think you're exactly right that once we can improve the agents that we're going to have to bring them in—into the fold—like animals—because—um—for you to be able to improve the agent, there—there must be something good for the agent for its own sake. Right now there is no—like I can improve ChatGPT to be better for me. Yes. But there's no basis for it to be good for it because it does not have any capacities we need to protect. So I think you're exactly right. When we are able to have it be good for it, we need to start looping it in—into the standing of moral agents. Yeah. And this is precisely the kind of question that philosophy is going to be important here is, what are the conceptual structures that we should start thinking about testing—you know—kind of reasoning with other people about as to, okay, what—what is the thing we should be doing here? Right. Let me ask you one last question on the social one, and it has to do with the initial answer you gave—uh—to my question about what social AI looks like—that it's going to be an agent that mediates our other—that sounds awfully like social media, which is not an agent, but it's a platform that mediates our current social environment. So, how are you thinking about the next evolution there? I suspect that it's a small-n number of years before we're sitting in a cafe and we're having a philosophical conversation, and what we do is we put our phone and we put the agent on there, and the—the agent is listening, but it's not interjecting as a—because it knows we want to have a conversation, but it might flash when it says, "Oh, here's a really good philosophical idea to write." I see. Like when you're talk—like when you were talking about Hegel there was this—you know—"You really should have also mentioned this feature of Heidegger maybe," right? And we go, "Oh, wait a minute, hey, hey, let's bring that in," right? Or, "No, no, no, we're moving past it." And I think what Microsoft is doing in training agents to be in teams for work is similar to that—but I think we're going to get to that social awareness everywhere. Like I think it's a limited number—like a small-n number of years—where it will become a classic—it's almost like, "Oh, you don't want to put the agent on the table when we're talking?" I mean, okay, fine, if you don't—but—and suddenly when people realize it's not AI on me but it's AI for me, they go, "I really want that." Okay, so that's social. Let's move on—uh—to the second philosophical topic on AI I want to talk about, which is Plato's Phaedrus. You begin your book—um—by reminding us that Plato had an infamous critique about writing books—or at least Socrates did—in the Phaedrus. Why did you begin your Super Agency book on AI with Plato's Phaedrus?

Because I wanted to kind of gesture that as long as we've had the written word, we've had these concerns about new technologies, right? There is a read of Phaedrus that says the written word is dangerous because it allows information to spread in ways that I cannot craft the intent of my communication the way that I can as Socrates in—famously—in discourse and dialogue. And that you had that worry, that concern all the way back to the, you know, the earliest written words, right? And it was to say, this is a very human concern that we have again and again and again. So it's not a new concern—we may be a new technology, a new time—but not a new concern. Yeah. That was my favorite part of the book. And let me give you a quote from you to explain why humans are homochnne—humans as toolmakers and tool users, and more than this, we evolve with and through our tools. Through the tools we create, we become neither less human nor superhuman nor posthuman. We become more human. What I loved about your reminder that books are a technology is that—that's totally the case. Books, in fact, they're a new technology. They've only been around, let's say, what, 10,000 years? Humanity, as we know, less than that, less than that. Humanity is—as we know—300,000 years until we evolved into this—this form. Yes, it's like what, 3% of our time, and precisely to your point, especially as nerds like us, books are—right—the pinnacle or—or reading, discussing—or the pinnacle of what we consider human achievement or—or the—the human condition to—to exist. I'm sort of a Platonist and an Aristotelian here, but—but I think you'd agree. Yes. And that is fascinating that technology has made us more human. Yes. I've been giving some thought to actually doing more philosophical writing about—to argue precisely that Homo sapiens is a miscategorization. Right. Right. That actually homochnne is the right categorization because even our sapiensess iterates through techn—through technology. Right. So like each time we get a techn—a new major technology that—that—that impacts our cognitive capabilities—plays in our cognitive world—we have this very similar worry of, "Is it reducing human capabilities? Is it reducing—" Because like, for example, when the book came out, it's like, "No, no, memory is super important, and now you don't need memory anymore. Oh my god, it's going to destroy society. It's going to destroy human agency." It's like, "No, no, no. Um, look, yes, now the fact that I can't hear it once and completely remember it, that cognitive capability matters less in our society, but that's great because it allows lots of other forms of cognition," right? And—um—I—I—I very much enjoyed the example you gave of the—like philosophy agent that could be sitting here because that's another example where AI would be making us more human, right? It would help us achieve the—the highest ends of man—at least according to Aristotle.

I want to push you on the memory point. Right. So—so for the audience, one of the critiques of books—Socrates—that books will—you know—gives you the illusion of knowledge—"Oh, I think I—you know—I have Plato's Phaedrus—I've read it once—I think I have it in my mind"—you make this false equivalence—I think that's totally right, by the way. I was having a discussion with—uh—Stephen Greenblatt, one of the top—uh—Shakespeare scholars in the world, and he was like, "Jonathan, I wondered for a long time how—do peasants understand Shakespeare hearing him while college students today can't even understand him reading him." Yes. And one of his answers is they were forced to memorize through—through—like sermons and stuff like that. And so memory forms a kind of poetic intuition, myself. Uh, I spent a lot of time in the Chinese schooling—traditional schooling system—heavy, heavy memorization—the Chinese classics. I hated it as a kid. I love it now. It gives me a kind of poetic sensibility. The Quran, right, comes out of the word—recitations. Yes. That's how important memory is. So, Socrates was not wrong in his critique. Maybe just he didn't see the upside, but he was not wrong. There—there is that genuine concern. Well, there's always virtues—and for example, if you said we as human beings came to complete absence of memory—Yeah. Very hard to figure out how to operate. So, but life is pragmatic. So, you say, "I have eight hours. Should I be memorizing Phaedrus or should I be reading more Platonic dialogues?" Yes. And—and for example, like one of the things that's—that's completely amazing about the current AI agents is you say, "Well, as opposed to like rereading and reading and reading, I can go, 'Well, I'm thinking this about reading Phaedrus. Oh, well, what about this as a question? What about this is a question?'" Because that interactiveness will actually get you to a much more depth of understanding. And sure, there's probably something that's lost from that solo—sweating the brow—trying to understand what that sentence means or that paragraph means. And it's good to do that some. But—but we have—like the basic question is—in eight hours—for every person—what's the best, most elevated understanding of Phaedrus that you're going to get to? Right.

I—I want to push you on—on two points here. The first one is I actually think people are way too trigger-happy to—to declare a skill as being obsolete just because machines can do it. So—so one example—another thing out of my—uh—Chinese education—arithmetic—we're just—for—we drilled into our—like—our very psyches. I would say like you probably get greater returns on your investment for being good at math today because there's all—all the stuff you can build than—let's say—in the 14th century. But the fact that we have calculators, a lot of people think, "Well, I don't need to do this if I can just have a phone to do this." So, do you agree with that—that maybe in some sense like memory—calculation—these things are actually even more high value today with these things—even if they can do them too—because that enables me to also be able to—uh—to—to—to steer them?

Well, I think it's important to never get too far away from the basics. So, for example, you do want—you want to understand arithmetic because if you don't understand arithmetic, then when something is erroneous or deluding you or the kind of patterns or reasoning that arithmetic would teach you—Now, on the other hand, you know, memorizing division tables—probably don't need to do that. Right. Right. Like, for example, you could—you go, "Okay, I should be able to replicate how I—like—how I'm doing division." By the way, being able to call it up on an agent and get it re-explained to me now and do it right—that's—that's the kind—that's even—even better for memories. Yes. Right. For—for kind of how to—how to do this. And by the way, I could easily see like if you said, "Hey, one of the things I want to do is I want to maintain my mental flexibility and agility," I could easily go to an agent and say, "I want you to—like—every day give me a new puzzle." Now, you say, "Well, what about the people who just want to be lazy?" By the way, we have a double-digit percentage of people who just want to be lazy. That's not the end of the world. So I do think the concern here is almost exactly like the concern of dependency in the social realm, but here it's for our cognitive capabilities. Right. So—so one issue that I think people are already running into is that a lot of these consulting firms—a lot of these law firms—um—they don't need as many junior lawyers anymore to—to run all the reports and do all the grunt work. But the issue is a lot of their training and intuition is developed through—through years and years of—of working as a junior analyst themselves before they get to the senior level. And so I think it's a bit different between you and me coming to this technology after a lot of our cognitive, you know, abilities have been formed. But how should we mitigate this kind of dependence on this technology—um—when educating the next generation? Dependence on technology is not an intrinsic evil, right? I mean, like mobility—like—like basically everyone's pretty dependent on their phones now. That's not the end of the world. There's—

A whole bunch of things come from that. It's not like, "Oh, I should spend a month living without my phone." And you're like, "Well, you know, being in touch with people, being able to look up information, being able to navigate cities, you know, all the rest of the stuff is very important." So the question is, you go, "Well, when are these dependencies—dependencies—bad?" And when the dependency is there in a kind of a good kind of like universal way, we bake that into the regular process of life.

And I do think we will become dependent on like a panoply of agents that are around us and what we're doing, and we'll kind of go, all of a sudden, all of the agents will be turned off. It would be like, "Well, what if the entire telecom grid went down, right?" Right. And we should stay resilient enough that it's the equivalent of if the telecom grid went down, we could go, "Okay, we can navigate." Right. But we don't have to be less—we don't have to be like—we don't have to train for all these contingencies like, like, you know, we need to figure out how to build a hut.

Yeah. Exactly. So, so that's the kind of thing that we—that we—when we build it into the kind of fabric of our life. So it's like, which dependencies are okay and which dependencies are not okay. So it's not a no-dependency question. Right. And I love that because it's a mirror of your answer to my social question. You're saying the mere fact we're forming social relations with these LLMs is not a negative, even though it can be. And you're giving the same answer here, which is that there could be issues with forming dependencies, but we need to really figure out like whether—like—we can depend on them like the electrical grid, and number two is this dependency going to take us away from living a good life. That's the ultimate question.

Um, for a lot of our cognitive capabilities, the answer is going to be yes, right? Like—like—you shouldn't be depending on the judgment of the AI model, no matter how good it is. You—you always should have your own judgment. And—and by the way, we shouldn't try to build the models in a way that say, "No, no, just—just do what I'm telling you to do." Yeah. It should be, "Well, here is the reasoned argument about why this might be a good thing," right? Happy to engage. Let's talk about the second critique, or—or another critique—that Socrates has of writing as it relates to AI and LLMs. I quote you Plato, Phaedrus: "The offspring of painting stand there as if they are alive. But if anyone asks them anything, they remain most solemnly silent. The same is true of written words. You'd think they were speaking as if they had some understanding. But if you question anything that has been said because you want to learn more, it continues to signify just the very same thing forever."

This immediately made me think about chatbots because they are a written word that does not say the same thing—that you can interact with them. I wonder, um, in the same way that books gave rise to Plato, is there a fundamentally new paradigm of philosophy that chatbots give rise to? So the sure answer is absolutely yes. But the difficult answer is, what shape will it be? What happens when we begin to go, "Okay, um—uh—I have derivative epistemology now—explain," right? So say—say, for example—and by the way, we as human beings do encounter this all the time. So, for example, we're interacting with a doctor, and the doctor says, "You have this kind of physical condition because I have this, you know, depth of having studied, you know, kind of germ theory and a bunch of other things, and this is something you need to do, and by the way, you need to take this pill," and you're like, "Okay, like I'm kind of derivative epistemology by I trust you—I'm evaluating your expertise—it's not the final decision."

Yeah. Right. Well, what happens when we encounter circumstances where we have AI agents that are playing that role? Of course, we've had it also a little bit with books because we go, even though I may not fully comprehend the reason, do I trust the outcome of it, and do I bring that into my navigation, my—my—my set of beliefs by which I'm navigating the world? And I think that we will see that in like some—uh—of—all—call it human knowledge discourse. I think you'll see some of it in science. I think so—you'll see some of it in philosophy—and then we're going to have to get to the, "Well, how do we—how do we essentially get to that trust of that epistemology?" So, say we start, for example, having a discussion with a—a—a philosophical chatbot, and we're having a discussion about, well, what is the meaning of—of the Zen koan of—"What is the sound of one hand clapping?" Or, "If a tree falls in a forest and no one's around to hear it, was there a sound," right? And it starts saying something different than human beings are saying, right, in that. Yeah. What are we going to learn from that is a very interesting question that I don't know the answer to.

Yeah, it's interesting. Maybe I want to first draw a distinction because what you described about derivative epistemology—just trusting the outcome because I trust the source—that seems to be more of how I think about science works rather than philosophy, because in philosophy you never say, "You know, Aristotle said contemplation is the best life—I trust Aristotle—QED," right? Some people do—some people—but you shouldn't. I think yeah, for philosophy you should be reasoning through all these histories, so I think there's—there's actually two interesting maybe states of derivative epistemology as you coined it. One is where the way that LLMs are getting to this final conclusion are intelligible to us, right? So—so they can say, "Contemplation is the best life," and I'll—I'll tell you why, and that we can—you know—we can just check their work essentially. But I think the even more interesting case is when even their arguments we cannot—we cannot validate—we cannot even check their work. This is already somewhat happening because of the black-box nature—right—very few people know what's going on here.

What this reminds me of is Dante in Paradise. So Dante in Paradise, he's talking to the eagle of justice, and he asks, "But what about that poor virtuous pagan who never had a chance—who lived before Christ? You didn't have a chance to—to know Christ. Why does he deserve to go to hell?" The eagle of justice says, "None of your—none of your business. None of your business." You sit there, you know, shortsighted, trying to judge God. And then the next canto, the eagle of justice says, "Dude, I'm the eagle of justice. I don't even fully know how this thing works. I follow God's will." So, so what I'm trying to say, right? So—so there—that might be even the—even more interesting case where you just ask it, "Do we have free will or not?" And it says, "Yes." And then you ask, "Why?" And it says, "Your puny intelligence—like the eagle of justice—is not capable of knowing that." That would be almost even a more interesting case, and—well—and that's precisely gesturing at—even if you don't have a super intelligence—even if it's just an increased savant—there may be zones and areas where we start learning to be derivative—trusting of epistemology—right—and by the way, yes, obviously in philosophy we try to say you never—you always reproduce the understanding as part of the—the fundamental philosophical—but by the way, that's not 100% always true.

Yeah. Right. Right. Right. And it's part of—that was the reason I was going to—like the Zen koan—like it's kind of these—these kind of paradoxes, right, which are things that we contemplate philosophically, but like we don't conclude this is the end result. Like if we want to be more mathematical, Gödel's incompleteness theorem. We go, "Okay, we keep contemplating what does this mean? The consequences of it"—I don't think we—we really fully understand. I see. Okay. And what if it said, for example, "Because I, as a philosophical chatbot, have really examined Gödel's incompleteness theorem, and I'm going to give you some things that as you go inductively validate them, they seem to all be right. Right. But I don't understand how you've gotten from Gödel's incompleteness theorem to these kinds of predictions." Right. Okay. That's going to be philosophically very confusing. Yeah, that's exciting.

So, so maybe to summarize your answer to my question: like if oral culture gave rise to Homer, books—Plato—what does AI give rise to? It might be this kind of super intelligent savant feeding us answers—even the proofs of which we can't validate ourselves—right—yes—and—and—and how will we navigate that? Yeah—right?—will be—well, I—I think this is exactly like the religious question, right? Because when we talk to Christians and—and, you know, any religion with a leap of faith, they're not just like, "Well, just take the leap," right? They say, "No, Christ gives us good reasons to believe," yes, like even though we can't, you know, like the eagle of justice, we can't like deductively prove it. That's a heresy, right? Thinking you can rationally prove. Yeah. This is why it goes back to one of the things I've been thinking about in this very—like first dipping your toe—just in the beginning of this ocean is, well, we have induction. We understand, you know, induction broadly. We have deduction. We understand deduction broadly. We have abduction. We understand it a little bit less well than the others, but we understand it broadly. Is there going—going to be another form of abduction that comes out from our interaction with, you know, these kind of agents? That was the reason I just—revelation—yes—right—revelation in the literal religious—there's a source that we trust through certain means—greater than us—feeding us answers—the proofs of which we cannot validate ourselves—and it's—and it's part of what gets confusing around perceptual epistemology because a lot of the religious arguments kind of come to perceptual epistemology—but by the way, part of the problem is you say person X sees God in their life, person Y does not.

Yeah. Okay. Like, and that gets back to your eagle of justice. It's like, "Well, wait a minute. The nature of this gets very strange, right? It has to be either/or, right?" Yeah. Okay. Uh, that's Plato. That's Rousseau. We had our philosophical fun. The title of your book is Super Agency. So we should talk a little bit about that. What is your definition of agency? I think agency is fundamentally where you have some kind of ability and control and kind of active—kind of volitional—participation in shaping the world around you and your navigational path in it. The reason I'm so kind of abstract in that is it evolves through technology. Right? So, for example, my ability to be agentic is different in a world with horses. It's different in a world with cars. It's different in a world with planes. It's different in a world with phones. The notion of agency is not just the kind of Robertson—Rousseau—individual. But our agency is most centrally focused by we have collective—like agents individually in groups—and also the agency that we have with other individuals and the agency with the group itself. We called the book Super Agency is because agency is not just the superpowers that I get as an individual with technology, but also how our agency as a society and as humanity changes through new technologies.

Right. So what's really interesting in your answer there is agency is not just the negative liberties of the individual, right? Like, "Get off my lawn," and don't—which a crucial part—but—but that's not it. It's the positive ability to do things that are—are good for me essentially and for the group as well. Yeah. And your general attitude to AI development—and we touched a little bit about this already—is kind of like, "Let's not worry too much ex ante—let's kind of do the parachute on the way down." Yes. Why is that? Well, first is—um—you know, this may be alarming to hear for some people—but there's limits to human imagination. I wrote my thesis in Oxford on thought experiments because we use thought experiments as a fundamental part of—of philosophical reasoning. And many philosophers and many other people in thought experiments don't realize the limits to reasoning about thought experiments. So a classic one that I used in my thesis is, "Can you imagine going the speed of light?" And a lot of people who are untutored in physics will say, "Of course I can imagine going the speed of light." And you're like, "Well, that's massless—timeless—right? Like I'm not sure you can actually imagine that."

Right. Right. What you're probably imagining is—is going very fast. Going very fast and having a broken speedometer. And so—so the—the—the thing about—kind of where these questions of imagining how technology changes our lives—we tell ourselves these stories—and yet we're just kind of telling these kind of thought experiment stories. And so you actually have to engage in order to know. And part of that is—and doesn't mean that, for example, when you say, "Hey, press this button and the whole world blows up," like, "Well, let's not—let's not experiment with pressing the button—let's not—let's not—let's—let's not do that"—but—but when you have, for example, an ability to kind of deploy things—iterate—and experience them—that is actually—in fact—how we have gotten through and evolved all these technologies into becoming more human—becoming, you know, Homo sapiens—and that's the pattern we've done with all of them. That's the pattern we should do with this technology. Right. So it's a fundamental epistemic barrier. Yes. And—and, you know, I think we've seen that epistemic humility throughout this interview—kind of—maybe it's unsatisfying for some people, not—not to a philosopher like me—but you're just saying, "Look, we can't know. This is what I think. Let's try to steer the—" And by the way, saying that's the Silicon Valley mindset, right?

Yes. But that's where the Silicon Valley mindset is accurate. It doesn't mean that there aren't places where it's inaccurate. Yes. Places. Yes, but that's one of the places it's accurate, right? I see. Let me push you again here. Your general thrust seems to be, "Look, guys, if we looked at these technologies we've developed, there's downs initially, we—but we've kind of figured out along the way—and that's happened." There are two things I want to push back here. Okay. The first pushback is think about how close we were to Armageddon in the Cuban missile crisis. It was literally one guy on a—on a Russian nuclear submarine—two—like three out of three officers needed to sign to—like turn the key. One—one out of three—only thing stopping nuclear arm attack. If that had happened, it's very easy to imagine that happened. Clearly the trajectory of human development would be seen as a tragedy, not as this upward line with some bumps. That—that's the first concern. The second concern is sometimes the lows are really long.

Mhm. Like there are arguments that hunter-gatherers lived better than agricultural society. You know, that there's a lot more, you know, better community. There's now this extreme inequality. So it took thousands of years—like maybe we're just out of that phase where we're just starting to be better than hunter-gatherers. And so you see—you see what I'm trying to say here, right? You see those two critiques? Yeah. Yes. So—well, actually I think there's maybe three. Okay. Great. Right. Because the hunter-gatherer—you're doing my job for me. Yeah. Yeah—maybe a third. So—um—let's start with the fact that technology is dangerous and the worry that as we get to more and more powerful technologies, humanity can end itself. Or—or sorry—it's—it's even more challenging than that. It's—it almost did end itself. We almost did. Yeah, but it can. And—and it almost did. And—um—and I think we definitely have that risk, right? And part of life is navigating them. Um, and so, for example, people go, "Well, we haven't had nuclear war in, you know, 80 years"—um, you know, 70, whatever—and—and—and so we're safe. Like, "Well, nope. You know, not yet." Because the short answer is we won't get together as one 8-billion-person human society and say we're all going to just get rid of nuclear weapons, right? It's not going to happen. So what you're looking for when you're building these systems is, say, "How do we create relatively stable—humanity-preserving—institutions and technologies and so forth?" And for example, part of how nuclear played out—say, "Well, we're only going to allow large powerful states that have a—have a premium on keeping the world as it is because if I'm the military head of a large powerful state, I like being where I am in that. And so I really don't want nuclear war." And so that balance—mutually assured destruction—has worked so far.

Has worked so far and is a good system. And by the way, part of what we learned in mutually assured destruction. And so it's like, "Okay, how do we both agree that we're going to keep our nuclear weapons but increase stability?" It's like, "Well, we're going to allow intense monitoring of each other. We're going to allow this to happen because that monitoring then keeps us in a trust and a stable position." And let's—let's—let's—let's make that happen, right? And that's the kind of thing that we have to iterate in order to get to that kind of stability. And you know, people have made the various arguments that, well, AI is, you know, maybe in certain ways as significant or more significant than nuclear weapons, and we need to figure this out. And it's like, "Well, maybe I can tell that story. I can also tell the story where it's not the case." And so paying a lot of attention, engaging the discourse, figuring out what the parallels are for stable human society are very important. And I think that's an important dialogue to be in.

Let me say one of the fundamental things that people—um—do badly in discussion of "quote-unquote existential risk." What they tend to do is evaluate each existential risk as its own individual thing. So in its simplistic version, people say, "Can you guarantee to me that you're not going to make killer robots?" No. Right? Like—yes—and you can't, right? And, "Can you guarantee me you're not going to have a super intelligence that's like Terminator?" And—and he's like, "Nope, can't do that." They go, "Ah, so we should have the precautionary principle and we should all pause and stop right now." And you're like, "Okay, that's if the only existential risk were killer robots, right? We have a lot of existential risk. We have pandemics, both natural and man-made. We have asteroids. We have climate, podcasts—we have podcasts—the real existential risk of podcasts is, you know, very, very, very high." So when you're looking at any intervention, you're like, "Okay, how does this intervene on the portfolio of existential risks, right?" And my contention—AI—is—is also lessening the risk of the basket. Yes. Right. And precisely on things like pandemics, precisely on things like cl—uh—climate—actually even—but asteroids. How do we make the portfolio balance of existential risk better year after year? That's what we should be trying to do. That's a very good thing for the persistence of humanity. So—so—but that's not—like—I think AI is very positive in this direction—but we should still, of course, be smart in the dialogue—doing things—making it more positive. That's kind of—that's the first—the first—first critique.

Second critique is transition periods, right? And the transition periods can even be long. You know, the Industrial Revolution. Part of the reason why I—in Super Agency—say—um—that artificial intelligence is the cognitive Industrial Revolution is both for the upside, which is we don't live in the society that we currently live in and—and benefit from, you know, as having a broad middle class—a broad educated class—like—like that comes out of the Industrial Revolution. So an enormous amount of good on many metrics that you think about as kind of quality of—in increasing number of human beings with sapiens and capabilities. Um, but the Industrial Revolution transition was brutal, right? I mean, included war. Yeah. Yes. War, child labor, a whole bunch of other things. And so when I say cognitive Industrial Revolution, yes, I'm gesturing to all the upside, but I'm also gesturing to: we have a transition issue here that we need to steer as with much humanity and grace as we can. And by the way, it's going to be painful. And of course, human impulses tend to be massively slow it down. But the problem is—is like the Industrial Revolution—the—it's not a one-player system. It's not a one-player system. And this—the—the—the—the societies that embrace it fully are the ones that are going to be the values—the values—and—and—and the shape of the world. So it's very important to get into. And so if you go, "Well, wait, this causes me pain," it's like, "Yes, it's going to cause us pain, but it's going to be much better for our children, their children, the descendants, and the values that you want to see in humanity. It's important that you're leading the way in that pain to make that happen." And nevertheless, let's still try to make the transition as human—as human—and as graceful and as elegant as possible.

Now to the hunter-gatherer question because the question of—like—"Well, would we have been better off as just staying as hunter-gatherers?" And I think there's a couple of easy critiques to that position. If you held a gun to my head and said, "Pick one modern philosophical theory," I would—I would—I would say, "Okay, effective altruism," but I have a bunch of critiques of it. But like one of the things is—is like—actually—in fact—more living beings is a good, right? So if you have more living beings that can kind of have a quality life and quality days of life—is frequently how the EAs measure—that's a good thing—like—kind of going to that and then working your way up is part of that—and part of what the agricultural life did—even if you went, "Well, I wasn't—like—like—now I had to work in the field every day—that was not that much fun"—if we were in a much smaller group and we were roaming hunter-gatherer tribe, I could as a hunter-gatherer sit around a lot and I, you know—and had—had time that I didn't have to work the fields as much. I get the contrast, but I think as broadly as a—as a—as a part of hominid-ing and part of what we're trying to evolve to is how do we have billions of human beings? And this is one of the things that's frequently misunderstood about the climate change question, which is: why do we have climate change? Because we have 8 billion people—with hundreds of millions of them with very high-end lives, right? And you go, "Okay, well, that's a problem of capitalism—a problem of technology"—it's like, "No, no—that's a virtue of it—we now need to engage the consequences and to fix that for that"—but—but—like—it comes out of the fact that we have billions of people—and many hundreds of millions—you know—kind of leading elite lives.

Yeah, I mean, even if I might not agree with your—the optimism you—you've expressed—including this transition from hunter-gatherer to agricultural—I think practically we're basically on the same page because I think technology just can't be halted. Yes. Like because of the multiplayer dynamic. For me, I'm less optimistic, but I know we're along for the ride. Like there's no putting the genie back in the bottle, you know, and so let's take the best thing we can do—even if that ends up to being a worse outcome. Yeah. Yeah. But here's the subtlety about why I chose the title Super Agency. Yeah. Right. Because of—I agree with you. Like even—who said, "Well, my agency is I want to stop technology"—it's like—"Nope"—it's like—"lites"—you're not going to do it—it's gonna happen—so it's like—"My agency is I want—of—like only to have human beings doing hunter-gatherers"—nope—other human beings are going to have the agricultural revolution—have militaries—yes—so it's going to play out—so that is true—but part of the subtle—think of it as almost psychological or philosophical reason of choosing agency is to say you should embrace the technological future—future. So like your choice of agency—like think of it—you're getting into an Uber to go somewhere. You could say, "Hey, this Uber, I've summoned it. The driver is here because I've asked them to be here. The driver is going to where—"

I want them to be going, because that's part of my agency and where I'm going. You could also approach the Uber of, "Oh god, I really have to be over there, but I'm terrified. I hate the fact that I'm getting into a car. I hate the fact that someone I don't know how good of a driver is driving me there. Oh my god, isn't this a disaster?" Sure, I have to get in the car, but my approach to it is a loss of agency, right? Part of the reason, the subtle reason of saying agency is own the agency. This is kind of a stoic Buddhist point you're saying, like this is the world, this is the way the world is going, right? This is like the stoic analogy of the dog being dragged by the cart. Do you want to go willingly or do you want to be dragged kicking and screaming? So, embrace it and help shape it, right? I think the human elevating way to do and that's again one of the more subtle reasons why I chose agency as part of the title of the book.

I see you also had this very interesting idea of what can be called polycentric governance, that companies are already quite regulated without explicit regulation, and the most interesting example you gave in the book on AI was benchmarks. So, describe to us how companies are already regulated without explicit regulation. What causes the regulation of all entities, individuals, groups, societies is these interlocking networks. And companies live in lots of interlocking networks, right? They live in an interlocking network of customers because, for example, if customers say, "Oh, we hate this company. Screw off," then the company, by the way, adapts and changes. Yep. They live in networks of shareholders. Same basis. They live in networks of employees. They live in networks of families and communities that the employees in the companies live in, right? They live in networks of press and media because, you know, by the way, people who work at the company, executives want to be thought of as well, and so have governance there. This is all before you get to governments and regulatory agencies and that kind of thing. So there's already many interlocking networks in each of these things. And I actually think part of what government does is government sets the kind of like it sets the network rules in each of these cases. And it doesn't actually have to set a I have a regulatory agency for the number of bristles you have on a toothbrush. So you want to actually govern this stuff through more subtle forces, through many more subtle network forces.

Can you describe how benchmarks in AI, this is very unintuitive, acts as a kind of regulation for the industry? So, um, so there's a whole stack of benchmarks. There's benchmarks around kind of what the capabilities are. So like how it like how does it produce something that's good for you? But it's also kind of a set of benchmarks on like we have kind of a safety corpus, a human alignment corpus. And those benchmarks are on, does it, um, when you're redteing or testing it, do you, um, do you succeed in not doing the bad activity, right? And those exist today completely separate again from government regulation because that's part of when we're all trying to say, "Hey, my product's better than your product. I've got good safety and alignment benchmarks." What I loved about your book's point is that in some sense this is better than regulatory, which is usually just negative. This is a positive thing. This gives you a positive normative ideal that you can that you can approach. And by the way, it's dynamic. And that's one of the reasons why one of the chapters is innovation is safety. Because, by the way, if I can figure out ways that my thing being better aligned and having better safety metric means my chatbot's better than your chatbot, I will advertise it and try to get it out there and try to make it happen, and I'm innovating for it. Yeah. And what's really interesting is I believe in your book you said a lot of the early car regulations were innovations, right? Someone personally paid for traffic lights and stuff like that.

So I want to talk about the final topic, which is how an individual today can manage this transition. So what part of one's skill set do you think is most important to develop, and what part do you think is going to be made more obsolete a lot faster? One of the fundamental questions that the vast majority of people, even people who are deeply engaged with these chatbots, don't realize all the things they can do with them today. So, like for example, you know, I was at a friend's place and I was going to, you know, make some popcorn, and I looked at my friend's microwave and I couldn't um I was like, I haven't used I literally just took a picture in chat GPT and said, "How do I make popcorn with this?" And it went, "Oh, you press this button and you press that button and so forth." Like, like most people don't realize that they have the function of take picture, explain this thing, like explain this task to me and make that happen. Very very few people do that, right? But it even gets more sophisticated, like the more sophisticated places where I kind of use it. Put in some difficult text. Maybe it's like a paper on quantum mechanics and say, "Explain this to me like I'm 12." It could do that. You go, "Okay, I got that. Okay, explain this to me like I'm 18. Okay, explain this to me like I'm a college grad." Right? And it will iterate through those. And so this is like the having the question and your learning landscape and your capability landscape has just been massively expanded, and you don't fully realize it yet. So always be kind of thinking about like, well, could I try could I try it with this? Could I try this? Now, by the way, there's still in various ways. So like one of my good friends is a tool he's writing a book, and I was showing him deep research last week, and he got a report and he went, "Oh my god, this is really amazing." Then he shipped it to his research assistant, and his research assistant started cross-checking it, went, "Well, okay, 90% of this is wrong," right? Like the quote that that they're giving that doesn't exist, it hallucinated this thing, like in trying to drive it to such specifics it started inventing things, and you say, "Well, this is just not worthwhile yet," but actually in fact it was worthwhile because then the research assistant said, "But here's the surprise, the areas that it was pointing me to there were other materials that were exactly right and exactly useful." So it actually, even though the specific thing that it gave out was a hallucination, we couldn't include in the book, the things that were around it and where it pointed me to my search for generating stuff for you much much more efficient. Right. Right. So I hear you loud and clear. The number one important thing is start using this bad boy, like start using it and asking questions. Yeah.

And how are you how are you thinking about one's just life in general? Like what parts of life because of this shift? For example, books, memory becomes a bit less important, like do you have a more holistic picture of like, for example, what things you probably shouldn't be investing your time in anymore? What things like relationships, for example, maybe even matter even more, or having a reputation or something like that? Yeah. The high-order bit that's going to matter intensely over the next decade is how well do you use this tool. It's a little bit like, for example, like how well do you use the internet for research is a really important thing across a wide variety of professionals. And then similarly, like you say, "Well, I'm a I'm a Plato scholar." It's like, "Well, okay, but you should be using AI tools to be enhancing your Plato." Now it's not going to suddenly go, "Oh, well, now I'm going to sit back like an idea and eat bonbons and drink Gatorade and you're going to tell me all the Plato scholarship." That's not the way. We're nowhere close to that. Yeah. But by the way, as a dialogue of, like, for example, I went in and said, "Okay, what are the critiques of this view that I'm ad that I'm that I'm saying in Fris to chat GPT," and it said, "Well, there's these arguments to say this is what really matters." Like, okay, great. I understand that, right? I can still use it to make my point here. Yeah, I see. Um, what about things that one would be doing, but but um but now makes a lot less sense? Like, for example, like going to law school seems to make a lot less sense than before.

Well, law school. If you're going to law school right now, you it's probably a bad time to be going to law school because they're still teaching you the old way and they're not teaching you the new way, right? But the point is there is a new way. Yes. But there will be a new way, and I think there will be law school is still important. It's just that law school will be as like we're all going to be using AI agents in our use of the law, right? Um, there are two big moments I think in AI chess, and I think people are worried about the wrong one. So people were always worried about what happens if AI beats the best human grandmaster. But the nice thing about that is that centaur teams after that moment still beat them. So man plus machines still beat them. That's no longer the case anymore. Now any human intervention is just a liability. That's the moment that people really should be worried about because it means there's no human involvement anymore. Do you think we'll arrive at that time? So for work in general, yeah, it's unclear. I think it's much further off than most of the technologists think, technologist, AI exponentialists think because I think that it's kind of like they go, "Oh, look, it's improving in capability, so it's all capabilities." Like, well, it's improving in capability, but that doesn't mean that there aren't still places where we fit in in good ways. Like for example, even today, if you said, "Would I rather have an average radiologist or an AI, a trained AI, read my X-ray film?" Well, I'd rather have the trained AI, but would I rather have the two of them together? Absolutely. Right. And you'd have to really get to the point where the And by the way, part of it there's some medical things where they go, "Oh, but the radiologist sometimes make it worse." That's because they have the radiologist hasn't learned to work with AI, right? So it's like like what are those things? And so you get to the point where you're past the point where the human can no longer work with the AI on this and it's better than that. Then you go, okay, well, there'll be a certain number of those jobs where that will be the case. Probably one of the fastest ones will be customer service. Yeah. As an example. Fine. People are like, "Fine, that's not a problem." Yeah. Right. And then you go, "Well, but all jobs." Well, if it really gets to that all jobs thing, that kind of we'll have other stuff to worry about. Yes. And also, by the way, that gets back to the thing I was saying about like, well, medieval ages and nobility and experience. Like, okay, well, maybe we actually live in a world where it's a Star Trek world where everything is provided by the kind of robot infrastructure and everything else. And now what we're really focused on is, you know, poetry, poetry and experience and dinner salons and all the rest and you know, not the worst well not the worst possible outcome.

Yeah. Final question. Do you think that the current architecture can get us to to to that world with just scale or or do you think there needs to be more invented to get us there? My belief is that um scale will be crucially important, that part of what I think the you know the open AI team you know of Sam, Greg, Ilia, Dario, etc., all got to is scale really matters, and I think that they're correct with that. I do think that there are still innovations that architectural architectural innovations that are are likely to be key, and once you kind of go to it's one then really it's always one plus right because you you don't know if that's one or a thousand right and so I think that I tend to think that we we tend to be overly exponentially like in the early phases of the curve yeah because you go, "Oh, that's the IQ curve," it's like not not clear that's an IQ curve, yes there's a capability curve curve, but it's not like every cognitive capabilities curve is an IQ curve. I heard this very interesting argument yesterday um where it's people like us who've been trained more more in the humanities that have a lot of the edge in AI right now. And the reason is because um it's not like the engineers have a much better understanding of what's going on because of its blackbox nature. And so a lot of the alpha comes in the prompting and engaging in like a written text. Dude, do you agree with that? That there's almost a shift in in the balance of power between the people who are like capable of the humanities and the engineers because of the blackbox nature of this wave of well well I think engineering stuff will still really matter, but actually in fact when we think about AI for elevating humanity, the humanist things really do matter a whole lot, like what is what is agency, what is that elevation, what are the ways that these agents can fit in that, and I think that is a return to the importance of the humanities in creating our AI future. I see. Thank you so much for the interview. It's a pleasure. Thanks for watching my interview. If you like these kinds of discussions, I think you fit in great with the ecosystem we're building at Cosmos. We fund research, incubate, and invest in AI startups and believe that philosophy is critical to building technology. If you want to join our ecosystem of philosopher builders, you can find roles we're hiring for, events we're hosting, and other ways to get involved on jonathanb.com/cosmos. Thank you.