Dominic Cummings: is AI already in control?

[Music] [Applause] So, thank you all for coming. Uh, I'm soft-spoken, but but but the good news is Dom will be doing most of the talking, and he has a a voice that will project. So, um, just a few things at the start. So, as Miles mentioned, um, I thought it was a time where it would be instructive to have a sequence of conversations about the intersection of AI with various domains of of Human Experience.

Um, when I myself think historically, U you know, I tend to think of naturally as a scientist of advances in science, advances in technology; the ordering is often unclear. Uh, but you know, but then so that's the sort of deepest driver, but then you get to economics, you know, maybe that takes, you know, 50 years; then there's that tends to shape culture, and then but on the other, you know, but then if you get down to an even shorter time scale, there is the classic subject of politics about power and its exercise and and how that plays out. So I thought it would be interesting to have a discussion on where things might go on a political time scale, which you can think of as 5 years, 10 years, something on that range.

The other reason this is interesting is that even though no election to date, including the recently concluded American election, has had the advance of AI as a major theme, in fact, if all you do is read what some of the heads of major AI labs are saying in their public writings and interviews, you have to wonder why this is the case. So, to take just one example, Dario Amodei, who, by the way, was a graduate student in physics at at Princeton, but I was director of Graduate Studies, um, may end up being the most significant uh PhD in physics from Princeton uh, you know, for a while, maybe forever. Um, he wrote an essay saying that he expects that human-level performance across the board is likely to be overtaken by stuff going on in their Labs as early as 2 years from now, right? Maybe, and that it will be followed by the proliferation of such superhuman or at least postum int, you know, extra human intelligence systems, to be followed by perhaps a 100 years of science coming by in about 5 to 10 years, right? So that's if you think of this happening in two years, you really have to wonder where we are.

For the purposes of today's discussion, Dom and I agreed that we would take a much more modest Baseline; we would somehow imagine that something like this will happen over 10 years. I have a graph here that you can't see, which has a straight line of sort of, you know, competence level versus time and sort of a genius level thing at at the at the top for humans, and this idea that in 10 years from now you'll achieve a Crossing. 10 years is still remarkably short on human scales, right? The generation is 25, political Cycles are 5 years, you know, so this is the sort of mismatch between, you know, chemistry-based life forms, which is what we are, we are and, you know, sort of electronics-based, you know, the electromagnetic interaction versus chemistry, right? The these These are different time scales.

Now, with that comes this notion that unique human skill hours, so I don't want to say people will become obsolete, but particular skills will become obsolete, some fraction of the time. I teach physics; honestly, even today, if you ask GPT-4 fairly sophisticated questions in physics, you know, you can get answers which are not bad most of the time. Uh, the reasoning models, again publicly available, will do stuff that entry-level graduate student might do in terms of, take the following differential equation, you know, tell me if this is true, and you know, you can; it just tells you I spent 11 minutes, seconds thinking about this, and so on. So all of this stuff is, you know, is is sort of visible. So you could ask, well, what's the, you know, let's talk about remaining unique human skill hours, and you can imagine another curve, maybe let's say it's here and there now, and over the same 10-year period, you know, that's going to decline. So view that as a backdrop to the conversation that we're about to have; this is apparently about to start happening, and there's just no way that this process proceeds without it intersecting with the political real.

Now, I had asked two people; uh, Josh Simons unfortunately had to cancel because politics involves, especially he's an MP, there are some farmers who are unhappy with the Starmer government in London; he had to stay and um either deal with them or or so on. So he was uh very unhappy he couldn't make it. He, by the way, uh did a PhD from Harvard's Kennedy School of government; he wrote a book on AI, so he was an obvious candidate. We will ask him back. Dominic read, I think around 2000, Bill Joy's famous essay uh called, I think, the future doesn't need us, or yeah, that why the future doesn't need us, I think something like that. And so since then, even though he's not a scientist by training, he's been absolutely convinced that technology will shape the future, uh and if you read his copious writings, which I can't say that I've read all of them because they are copious, but I I do recommend, he's been thinking about technology, the generation of Technology, what lessons it has, but also just, you know, the future for a long time. So really, uh he's I know he's known for many things in his actual practical political skills, which could be the subject of a completely orthogonal conversation, but today he's here as someone who's thought deeply about the Arc of history and how it intersects with politics.

So that's the background. The way we will proceed is we have some things we've sort of decided we will talk about; we'll talk about, and we will leave time at the end for all of you to join in, uh and of course there's a reception at which point, you know, he's available for uh General discussion. So so with that, um maybe I can so start with one high-level question, which is that we could ask the sort of question, if all of this is going to happen, what might be a desirable trajectory for societies and humans, and what is a likely trajectory? So pick whichever one you want, and and you know, start with, you know, what do you think these what challenges do you see coming, you know, uh coming up as as as these what are going to be the signal events, you know, how will people really notice that this is going on?

Um, well, I think like for sure it's it's inevitably going to be just um unbelievably chaotic. I think there's no way around that. If you look at um how these how technologies have interacted before with human events, if you look at things like the railway and the telegra telegraph coming in in the 19th century um or later on with mass media, these big technological shifts always lead to huge changes in power; they're unbelievably hard to to predict how they're going to work; it's very hard to know what things are going to be centralizing and what things are going to be decentralizing. Um, you know, just to give one obvious example, at the moment it seems to me that I was reading recently some accounts of French peasants who were drafted into the war um with Napoleon, and they were kind of thrown across the continent to Moscow, came back and asked all of these questions, and you can read them now, their answers; they basically had no idea what on Earth the whole war was about; the whole thing was completely baffling to them; there was no real consensus reality in kind of 1800 Europe because of the media and information structure and Technology structure.

If you go forward to to a period like 1940, you have a time when um people like Stalin or the head of the BBC or the head of one of the big American um radio and then in the 50s TV networks could kind of determine consensus reality for hundreds of millions of people; Stalin could make certain kinds of decisions; he could take people out, photos; he could dictate that the Soviet encyclopedia will say blah blah blah. So for a brief period of time, you have this period where a very small number of people and a very small number of very centralized institutions exert this great power, and there's all kinds of consensus reality amongst Elites and amongst the public. What's happening now, it seems to me, is that the world is already Poss arguably now, but if not heading very quickly towards a situation where it's going to look more like 1800 than 1950, and you can even see this just over the last sort of 10 years ago. If you go back and look at um people like Obama and Elon Musk and editorials in the Times and all these sorts of things, you go back 10, 15 years, there's a clear sense of kind of like consensus reality among Elites, but if you look at it now, that's all just completely fragmented in the in the last few years. You have people like Elon and a whole bunch of that that kind of network in Silicon Valley who regard the mainstream media as essentially like either mad or actively malign, but in any way obviously not something to be trusted, not something which defines reality for them, and you have those same old institutions which were extremely friendly towards people like that even 5, 10 years ago saying that these people are essentially fascists, deranged, crazy, pushed even to the sort of extraordinary extent to which it's become completely accepted conventional wisdom amongst political journalists in Britain to give their views on Elon as as a tech company manager.

So over the last three years, they all tweet away happily saying that Elon doesn't understand tech companies; he's completely useless; meanwhile, Elon has catching skyscrapers falling from space with chopsticks. So you have this very, very odd situation happening. Um, I think the same thing has just happened with the election, right? Again, you've had this; they've had the same experience that we now see repeatedly: political Elites just being hering towards opinions; those opinions um smashing into reality; them being completely confused by what's happening; and then they double down. So we saw that with Brexit in '16; we saw it with Trump in '16; we saw it with when me and my team were in Number 10 in 2019; we've seen it again with Trump now; seen it on Ukraine; seen it on on on Covid; you just have this repeated process whereby the political and mainstream media Elites who think we're the the sensible ones, the public are idiots, fooled by disinformation and conned by the sort of, you know, these tech companies whatnot, and and a growing counter-Elite emerging saying, actually, we've it's the old political systems that are the problem; these things are like irredeemably corrupt, irredeemably useless; we actually need to build new things.

So if you look at these Trends, I think um almost inevitably I think it's going to is going to continue to be chaotic; the political world is absolutely determined in not to change in all sorts of ways um uh certainly in Britain; Washington's more complicated um but in in Britain um we've had the biggest pandemic in a century, and we've had the biggest war, land war in Europe since Hitler, and both events have have led, and Brexit of course before that, and all three of those things led to an extreme determination of the governing systems inside Westminster to say we will not compromise with the electorate; we will not change tack; and we certainly will not start changing how institutions work to absorb how these Technologies are changing; in fact, the ex the exact opposite. So one very simple story to to to illustrate it: in 2020 um I brought in various people from the worlds of physics, data science, AI and whatnot and built a team in Number 10 which brought in Elite Talent to the PM's office; we started doing this in January, actually before Co got going, but obviously Co gave it a huge turbo boost; that had a very dramatic effect; it led to all sorts of things working immeasurably better; you actually had uh extremely able technical Talent sitting 30 meters away from the PM and available to give the PM advice on day one of the new government going in; and K St arriving in Number 10, one of the first things in pure classic srey fashion happened was he was obviously given lots of documents to sign; one of the documents was to sign off the destruction of this team and this unit. So the data science and AI team that was built to support the Prime Minister, the concept which has been copied all over the world, is as we speak being dismantled by the cabinet office; the people have all been told that they're not welcome; they're all being pushed off, etc., etc., etc. So that gives the sort of um uh I think a picture of the kind of pathological nature of these old political institutions; now, even as this revolution is happening, they are they see Power as a sort of zero-sum thing, and they don't want to open up any of these political institutions to to these tools and to the people and to the talent because fundamentally it's threatening the core of the cabinet office; and treasury hate of this is information is power, and if you have a situation in which the Prime Minister has access to the best information and tools that we can't even use and don't even know how they work, well, that completely upends the power relationships, right? And the the way the British state has worked for a long time is that the Prime Minister office is not actually in control of the government; the cabinet office and the treasury are in control of the government; hence their destruction of this team because they want to make sure that the old the old ways continue.

So let me maybe try and theorize a little bit about what you said in the context of this. So is it fair to say that some of the political chaos which is not solely in Britain by any means—America is very polarized, um Israel actually, you know, until the war and so on—is that not itself sort of a result of this? Because the information revolution really dates to the '90s, right? We're now getting to AI, but we're already 30 years into an enormous expansion of information flows. Is it the case that the technology is preparing the way for itself because it's making it impossible for normal political processes to work? Obviously, what you're suggesting is there are better ways of doing it, but so yeah, so I mean maybe if you could speak a bit to...

Yeah, I think so; clearly there's a lot there's a lot of these things are happening which are not um not at all limited to a particular country, a particular regime, and Co is a very good example of that in all sorts of ways; the the um Western bureaucracies failed in very similar ways basically everywhere um but also uh subsequently they also behave very similarly almost everywhere. So so two very simple obvious examples: two of the things that were most successful in Britain were a we created the vaccine task force to try to figure out how to do vaccines much faster than previously was done, and secondly, we created uh A system that brought together uh Wastewater monitoring and AI people. Mhm. Both of them were closed down in 2021 rather than reinforced; what obviously should have happened is Co task force was on; what obviously should have happened is um Wastewater sewage monitoring was was expanded; instead, they were they were both closed down because they were both seen as so um threatening and um uh uh to White Hall so much so that after an outcry, the C office tried to rebuild the the Wastewater AI system, but they said to the people that built the first one, you're not allowed to bid for the process to rebuild it with the same people that built the first one because that means that your bid would be really fast and really cheap; so that would be unfair on Del and all of the our normal suppliers; so you're you're the only people banned. So we the people who actually built the system which was then copied from South America to Asia, they were explicitly banned from rebuilding the thing that was vandalized in the first place. Sounds could of completely crazy, but very similar things happened in other parts of the of the world. So for so Operation Warp Speed was similarly closed down in America, and that and that's my point that these pathologies, I think, are that there's a set of as you say technological things which are disrupting all these regimes and the economies and information; the old media, old media is tanking at the same time; subscribers and everything are collapsing um but also it's creating these path but also the pathological institutions are failing in very similar ways, so it's not just a British thing; that's the the core patterns you can see everywhere.

Okay, so maybe we can summarize this part by saying that you are highly pessimistic that existing States will be able to cope with this.

I think so. If you just, you know, if you just look at how they cope with, say, gain of function, okay? Now think of how they dealt with regulating gain of function before the pandemic; was already completely insane and right and disastrous. You'd have thought that after Covid that would be a bit of a wake-up call to get a grip of gain of function; no, no, no, no, no, no, no, no, of course not. So we've continued to have completely mad attitudes towards this kind of research; we've we've continued to have the CDC and FDA in America vandalize a lot of actual inspection of these things; we've seen massive cover-ups of um in bureaucracies Across the Western World in terms of how they handled all of this and what they funded. So again, you see this this kind of now gain of function relative to AI is like trivially simple, right? It's like it's it's a matter for like expert people to sit down and look at and try and weigh up what the pros and cons of it all are and then how to regulate it and then how that actually should be done. Um, it's uh, you know, it's relatively just extraordinarily simple relative to the extraordinary issues with AI where there's just a whole bunch of complete unknowns that we can't even sensibly weigh up, and the greatest experts on the planet who are actually building these things completely disagree about. So if the old pathological political institutions can't cope with relatively very simple things like gain of function and just have even even have honest debates about their incompetence about it is beyond them, then it seems practically impossible to imagine them coping with this.

All right, so from that though, um let me segue to something else, which is to say both Dario's essay and there's another one by the way by Leopold Aschenbrenner, who uh it's called Situational Awareness, which you might also look at. So Li is a very, you know, the people you want to read at this point are all shockingly young, as is often the case with revolutions. So he was uh I think a superstar student at Columbia; somehow managed to get uh an A+ on every course, and even though there's great inflation at American universities, it's still not easy to get an A+ on every course, and he was I think leading the alignment team at OpenAI, and then he left and uh and he's outside. So both of them I have suggested this notion that as this AI explosion takes place and generates an enormous amount of latent power that the thing to do is for the United States, leading the Western World, to somehow raise ahead of China in the context of the current International situation, uh obtain, I think, you know, maybe a two-year lead, which given super intelligent systems, you know, could mean the same as having a 20-year lead or a 50-year lead, and then negotiate from a position of strength, you know, with China. So I guess my question for you is uh is this at all plausible that never mind, you know, to to zero the approximation of the question, well, you can address both: is this is this a desirable course, and is it at all possible that the states that you function that you were just describing, the inability of governments to do basic tasks, that they're capable of bringing off anything like this?

So I think on the the the second question first, um I think anybody who goes and reads some of the classic books um such as Now It Can Be Told by General Groves, who who was Oppenheimer's boss um and ran the ran the Manhattan Project, um if you go back and read some of these classic books about how the American state functioned decades ago, how it ran Apollo, the ICBM project, Opa Park, Xerox and whatnot, um that that America doesn't exist; and part of the reason that America doesn't exist is that the PE the sort of people, the General Groves type people, are now are people like Elon; they people like Patrick Collison; they're building SpaceX; they're building um Stripe, etc., etc., and they these people have um it's part of a broader question, but I think one of the key things that's happened um and one of the reasons why we're in the state we're in is this huge Talent collapse that's happened in Western politics um uh which is really really very dramatic I think historically because in the past there was always a a big section of the intellectual Elite and the kind of capable Elite that could build things was involved with political power. I just been reading about the 1790s and 1800s and Whitehall, and it's incredibly striking how the Whitehall of 1795 was more like SpaceX 2024 than Whitehall '24 is like SpaceX 2024. If you go back and read some of those stories about 1790s, it has a very it has completely Silicon Valley feel; people like Pitt as as prime minister and ministers called people in; they promoted young Talent; everything moved incredibly fast; their motto was the Marc Andreessen motto: build, build, build; they built extraordinary State capacity; they had um the political community and the Whitehall community uh completely working with private sector; they did they took procurement incredibly seriously; there were constant debates; people got jailed for stealing money and for screwing things up—completely inconceivable now, where procurement is uh complete and not a disaster, and everyone gets put in the House of Lords and given massive rewards for failure in 1719, 1800. We looked a lot of these um disastrous characters up. So that's one huge one huge shift; the America of 1950 is not in charge in in in Washington. But also, I think that [Music] the some of I think Leopold—there's a lot of interesting things in Leopold's essay; Leopold's a thousand times smarter than me—but I think a lot of these people have to go back and read some of the books that my old tutor read, and I think that if you read Thucydides and the outbreak of the Peloponnesian War and you read his account of how that started and the escalations involved, the idea that if America says, well, we're just going to build this super intelligence which is going to render all existing military technology completely irrelevant, but don't worry because we're the good guys, and then from that position of strength we're going to um explain to the you 1.4 billion Chinese how the world's going to be structured, but you shouldn't fear this because we're obviously beneficent and the world will just be happier if we do it like this, and that the Chin the Chinese are going to and go, oh yeah, okay, that sounds that sounds reasonable, um uh okay, we'll get back to us in a few years' time, and we'll negotiate or rather will accept your dictation to us about how the world's now going to be structured; that obviously is never going to happen in in a million years; it's not going to happen; and China will take Extreme Measures and extreme risks in order to stop that happening because they won't see this as some kind of joint benevolent um creation by the Western liberal democracies, etc., etc., of a new world order which is to everyone's Advantage; they'll see it as the latest example of the Western world's grotesque hypocrisy um and conniving nature which must be scuppered at at all costs, um and they will try and Scupper it at all costs, and therefore I think that um no, you don't have to go into a lot of the further-out scenarios about what the capabilities of these systems might be; we already know from looking at archives how things like the Cuban Missile Crisis worked out in a lot of detail; we know, for example, we can read transcripts of JFK talking to the Chiefs of Staff in the Cuban Missile Crisis; we can see the kind of assumptions they made, and then we can now look at archival things, and we can see all kinds of ways in which the conversation was completely wrong; we never they all of the military advice in Cuba was for JFK to invade; they had no idea about what was happening.

With the submarine incident that was kept secret for 30 years, they had no idea that there were already tactical missiles, nuclear missiles, in in Cuba with orders to fire. They had no idea that um Castro was writing to the Kremlin begging to have the war, because even though Cuba would be completely destroyed, it would be worth it for the triumph of the Socialist Revolution. Everyone was completely stunned when all of these things came out 30 years later. So we know, like we know from history, that we constantly don't learn from history how wars start and regimes constantly confused and then are baffled when the archives are opened up and clutch their head and say, "My God, it happened to us in 1914; it happened to us again with Hitler in 1939," with both sides completely miscalculating. Various things happened in Cuba, um, and if you imagine those same dynamics happening in inevitably again, but in an environment where the technologies are nuclear destruction, so like, you know, a million times worse potentially than than July 1914, and the time scales are 20 minutes. Okay, that's the current situation. So you now have nuclear staff and 20-minute time time decisions rather than like, say, two months in in July 1914. But then you imagine that already nightmarish problem being being overlaid with: are these people even in control of these things? Is it them that are issuing this press release, or is it the AI system that's issued this press release? Do they even control their own cyber operations, or is their cyber operation now like somewhat gone rogue, etc., etc.

Uh, if you look at the debate over Putin and Ukraine, without getting into the all the ins and outs of it, one unavoidable thing is yet again complete miscalculation. And and if you talk to the people in the Deep State in DC and in London, it remains the case that Putin is essentially a black box; no one has a clue what's going on in his mind or about about what's happening in the immediate circle of power around him, what their calculations are, and how they would respond to certain kinds of threats, certain kinds of things that um, you know, the chaos that inevitably happens with war; planes get shot down—is it deliberate? Is a mistake? Was it a signal? Was it complete blunder? Etc. So you imagine all these normal things that we know happened historically, and then you overlay on that questions of the increased speed from these systems being automated and then the lack of clarity about who's actually in control of what—are the people running the system, or is the system running the people?

It doesn't strike me that the way to go is to um start another escalating crisis with China. In fact, I think we should go in the exact opposite way. Um, I think we should everyone should read Lequin News' memoirs about this subject and should reflect that there was a bipartisan policy in Washington for decades which was there should there will be one China; it will be United; Taiwan will rejoin, but it should be peaceful, not through bloodshed. Biden blundered around changing that over the last few years, but because he's senile and gaffes and says things that he doesn't necessarily mean, it was a very confused thing where he would say things, and then it would be walked back by the White House, and then he would repeat it and be walked back, but finally after about four or five of these episodes, China's got the picture of—hard, you know, he is senile, so we can't be exactly sure what he's saying, but clearly the attitudes are changing. And if you look at the debate in Washington, it's obvious that the debate has changed there; right, there's much more of a feeling of we can't let Taiwan be unified; the the old consensus is gone. I think it's just another complete disaster, and if we go down the path of America saying we're not going to let Taiwan be unified, that is heard in China as we intend to destroy China. Um, and if you add the timelines of what Xi's saying on Taiwan and the army, the critical importance of semiconductors and TSMC, and then the AI timelines, which obviously got all the recipes for um the classic catastrophe, right.

Well, one one good thing just in the last few days on the positive side was that when Xi and Biden met, they actually issued a statement saying that they had agreed that nuclear weapons would not be turned over to automated systems, which at least is a recognition that they can see some of the challenges, you know, that you're describing in terms of time frames. We'll have to see, of course, how how things go. I think so. One sort of connected thing to that though, which I think is going to drive this this is um there's a there's a um you can see in Ukraine this uh kind of drone anti-drone um uh extremely competitive and extremely fast-moving armor race happening in real time with a mix of classified technologies and open commercial commercial technologies. And one of the outcomes of this is, as people predicted 10 years ago would happen, there is an incredible drive to autonomy and taking humans out of the loop, because one of the things if you talk to Special Forces people who've been over there watching this, one of the things that that obviously happens is on the one hand you have these things they you have people hundreds of miles away piloting them flying into this bunker and then blowing everyone sky high, but then you have um uh various kinds of defenses kicking in to take the drones. Well, if you can just drop it in a vague area and give it an instruction and say find an entity that looks like the following and then make your own decision and kill within the following parameters, right, that's all that is now already happening. Um, so there's a great the speed is critical in war and there and therefore speed is driving growing autonomy, and therefore I think you're going to see across this whole drone AI yes system this process remorselessly leading to greater autonomy and less and less humans in the loop, because the more humans in the loop the slower it is; physics versus chemistry again. Yeah.

Um, no, I mean if I look if you look at World War II, it's actually extraordinary what technological advances took place between 39 and 45. You know, when you're in danger of getting killed, it really concentrates the mind rather well. And unfortunately, absolutely so. Coming back a bit from the intern arena to maybe thinking more domestically, imagining, you know, within a given society um so we're sitting um at New College, a college, and you know our selling point is excellent tutors educate uh you know the young people of the future. Um, I think there's a question to my mind which is we're sort of going on as before; we teach what we used to teach, uh but in fact if someone asked me, you know, at age 15, what should I learn? What will you know what kind of a career might I have in the future, uh I wouldn't really know what to say. Uh, and I mean I mean that genuinely; it's not just me. Some uh physics used to have—Joe may remember a series of online discussions in AI, and the former uh Chief Technology Officer of Microsoft was one of the speakers. I I went back and looked at this, and he talked about with growing enthusiasm of AI and AI training, AIS, and how there was going to be this super intelligence explosion, and then finally one of the staff at the end said, uh, you know, that's all wonderful; it was a great talk. You know, what should we tell our students? You know, what should they what should they do to prepare for this world? So it's interesting, you know, he you can see that he has nothing to say, right, which is to say because he really doesn't think that there's much advice to give, but then he felt if he just said, "Well, they're toast," uh, you know, that wouldn't go down very well. So in the usual hou, you know, you say, "Well, you know, my grandchildren, they play with things that I never played with," you know, the general argument, but of course it is true that humans adapt and have adapted to extraordinary changes in our circumstances, but it's a little hard to get around the idea that our retraining time isn't years; it's often longer. If you're young, maybe it is, you know, a few years, uh and that ultimately it's often from major technological revolutions that the next generation—it's your children who grow up in the new world. But of course if the time frame for changes is less than 25 years, right, then you know as a physicist, you know I always like to compare things, so the time frame for retraining and the time frame for technological changes have crossed, and these are two different worlds in some ways. So I'm just wondering are we going to have a—so let's say Anthropic and OpenAI unveil in two years this amazing model which is a better physicist than most, you know, physicists alive, a better mathematician maybe than most mathematicians alive, and so on. What happens then? You know, do people just say, "Let's go on"?

Will it be like specifically specifically with education you mean? Yeah, which is what will will will will the young—the question of what is what are you to prepare yourself for—does that does that become an issue at some stage? Does that generate or do people just go on? Um, it's possible that it almost vanishes as an issue, mhm, because I mean I've spoken to quite a lot of the kind of like if you think of like the top 30 people or so in the world who who deal with this technically, mhm, and they have very different views about it. There's clearly no kind of um there's no sort of agreed answer. Mh, the main thing one gets from talking to them is that these things are just highly unpredictable, right? There's no real we don't have any theory as to why this machine learning works the way it does. Um, scaling has just turned out to be an empirical fact for some number of years; we don't know when it'll suddenly conk out, if it will conk out. I mean, obviously has to conk out eventually because laws of physics, but um it doesn't have to conk out anytime soon. Um, so in that environment, no no one can really give a good picture of like what skills are going to be useful in 10, 20, 30 years' time. It seems a general sort of theme that people say is like it's more likely, I'm afraid, that you will be redundant before your hairdresser, um but other than sort of very vague comments like that, people don't really seem to have much much sense. So possibly what will happen is that that it defaults to just people, young people being told, "Don't think don't don't do education on the basis of careers and what you think is going to be useful and make you more money, because it's essentially completely unknowable what that turns out to be. You could easily do something which we snook yourself. Um, just do what's interesting to you and what you love, what you love spending your time on, and if the revolution does work out is in the way that some of the companies think, then you're going to have a lot of leisure time anyway, so um why not spend your time focused on what actually moves you and you feel every day like it's just wonderful to be involved with."

So when 5% of the population at the right age went to college or had personalized tutors like John Stewart Mill and Bertrand Russell, I I see that there's that group, but modern higher education isn't that; you know, it's 40%; you know, it's it's sold as skilling, yeah, right. Most universities advertise, you know, uh courses and so on, yeah. So certainly getting back from a world of 40% college education to the 5% who maybe want to learn stuff for the—I mean it's again um that that so the main thing I guess is I can imagine end states which Nick Bostrom, from our our Oxford uh uh colleague, you know, has a recent book in which he tries to visualize what what if things go right, right. So you can imagine far away there's another equilibrium at which, you know, we're all enjoying life and and doing stuff, but the getting from here to there is a highly non-equilibrium process. So to take an example, uh, you know, uh yes, certainly, you know, I'll be out of work. Uh, so there's one problem with that, which is to say I also have the skill set to make trouble, right, which is to say so there's that thing about elite overproduction, yeah, this will be elite overproduction on steroids; you create—and so it's precisely those people who—yeah, so that's one. The second, for the hairdresser, I mean, I I thought about this for a while; the issue is, you know, even now, you know, if you want to solve a plumbing problem, you can actually point your phone and ChatGPT at it and you know ask for advice, and you can actually get pretty specific instructions and what to do, maybe not for all plumbing problems. What that does mean is that there's a distinction between the knowledge that a skilled workman work person, you know, work person has and the actual muscular strength or skills, you know, to do it. So at the very least, even those of us who become redundant will, I think there will be a huge downward pressure on unskilled wages; you can just, you know, yell out of the window and say, "Hey, you you, you know, come turn this for me," right, because you can sort of see what what has to be done. So the disruption I I don't even think is going to be that segmented, and and and if it is segmented, it's the people who are most likely uh you know to to want something to be done done about, right. So I think I think the elite overproduction point is a very good one, and I think it's also one to which Britain is is particularly um dangerously exposed, because if you look at the structure of British GDP now, more than many countries, it depends on a set of uh white-collar services in companies like McKinsey and Deloitte and whatnot, and you know various kinds of things—the City, accountancy, law, business consultancy, etc., etc. Now I, you know, I I know now people working in some of these frontier companies, age 30-ish, who've got friends and sometimes housemates with jobs in these companies; they earn, you know, a couple of 100,000 quid a year; they they're one of like the top 1% of earners, but the frontier AI people say to me pretty much everything they do is like already automatable, and within a few years close to 100% of everything my housemate does is going to be automatable, and for sure no one is really prepared for this, right? And you've got a whole set of people who who've studied hard; they may have gone to accountancy school or whatever; they've they've worked their way up through McKinsey, and they're expecting their life is going to go on a certain trajectory; it's not at all; it's going to be profoundly shocking if if Sam Altman drops a model that suddenly basically just automates all of that away very rapidly. A lot same's going to happen with media; I mean, already happening with the media, right? I mean um the industry is already dying extremely fast, but also um you know GPT-5 or something is going to be indistinguishable; it's going to be able to produce um at least uh the same kind of um uh political media that currently takes up a lot of humans, which of course is part of the reason why so much of the political media has become extremely hostile to the tech world.

Just parenthetically, um, one of the reasons why the Russia-gate hoax was spread so much after Brexit and Putin in America is because the New York Times tried successfully to pin blame for Trump winning on Facebook as a kind of preemptive move, and a lot of the British media joined that hunt because they thought um rather than ask questions about how the New York Times published Hillary's emails, far better to try and blame Facebook for it; the fact that all of the actual hoax didn't make any sense to anybody that understood advertising, the whole thing was completely bogus, didn't matter. So you could see already even a few years ago the old media companies trying to knap the emerging tech companies. Inevitably, these kind of power battles are going to emerge again, right, because a lot of money and power is at stake. The old media companies think correctly; people like Rupert Murdoch are looking at this, gnashing their teeth, saying we've built all of these things, and then some 30-year-old kids going to come along and destroy the entire value of these businesses and basically just like completely replace it potentially with new media companies. So that's going to cause um complete chaos, I think. On the University point though, controversial point to make in this room, but we talked before about this about this cycle of of political media academic elites being repeatedly shocked in the last 10 years. So I'll give an example of something which I think is another one of these things which is coming in the same way. If you talk to what are disparagingly known as low-information voters in Pennsylvania 3 years ago, they told you that Biden is obviously senile; watch him walking around the stage; I had a grandfather like that; it's unarguable, right? What did the elite um insider journalists of the New York Times and CNN tell you? They said the president is super super sharp in in in in private. So the low-information voters pay no attention to the news were actually correct, and the highly paid experts in ver commerce were were completely wrong. Now if you talk if you look at the same groups now, group A says there's far too many people go to universities; a lot of people are going to massive debt to get degrees that are basically pointless or and actually even worse than pointless, like a waste of their time because they spend three, four years at a young age and they get into massive debt, um and there's just no like actual economic rationale for what they're doing; they've all been told to do that by the kind of prevailing culture in elite circles in London. Um, if you voice that sort of attitude, everyone laughs at you and thinks thinks you're an idiot, and um if you go to all the think tanks, they'll say, "No, no, no, we must discuss how we get even more people going to universities and doing even more of these degrees." I think that the low-information voters are yet again correct and are going to be proved correct, and they're ahead of the curve on the on the AI thing. So I think what will happen, hopefully what happens with places like this, I think is that fewer people will come; the quality will go up; well, we hope.

Um, so uh one more aspect. So supposing um Anthropic succeeds in building this PhD-level, postdoctoral-level, you know, AI researcher and then proceeds to run a million of them in parallel. So Dario says he will produce the next century's worth of science in 5 to 10 years. Uh, for those of us who are scientists, this means that every copy of Science and Nature will have these seminal papers, right? So we won't be able to do what we do now, which is look at and go, "Huh, not worth it," right? That's just uh and so on. Each one of them will need to be studied, and we won't be able to, you know, I mean, to actually internalize a breakthrough takes time; you have to maybe there's a new piece of mathematics; there's a whole whole set of phenomenology. Um, so so one is that disruption, which is to say I don't see how you can credibly say that humans sort of understand what's been learned, but then economically what does that mean? Does that mean that the next 100 years of patents have now been created and in the current system and become available to this one firm, and then all sorts of other firms, you know, drop in value? Uh, so I mean that also seems like, you know, information often arrives through the stock market, right, and so on. So so how does that play, and and what kind of demands for—so this goes back, but so in the case of education, you could say it's human capital and its value, right, and now you could talk about, you know, this kind of uh intellectual capital, know that's generated. So there again, how do you see the—I mean, how do you see the leading edge of it? Will there be—and then I'll turn to the question of ownership of these AI companies, but let's maybe just start with: do you think this is a plausible worry and under this scenario and and and how might that play out? I mean, if you if you read the stuff from people like Dario and Sam and Demis and those teams, I mean, they envisage this being patchy, right? It does seem to be um like not predicted 5 years ago, but it seems possible that these things actually end up being um continue to make rapid progress in in areas like mathematics and physics, mh, whilst possibly stalling out in other in in other kind of areas. So it it looks likely that it's going to be um in the short term patchy, where they suddenly shock and they do amazing things in in certain areas, but then possibly don't make progress in in other things. But I think that once they once they get to the point, if if they get to the point that Dario talks about in his paper—well, sorry, I I'll rewind—already there is a situation in which um all of the three-letter agencies in America, people like the NRO, uh CIA, NSA, etc., etc., uh have already begun discussions with all of these companies; that this is this has already been happening; same thing is happening in Britain with GCHQ and MI6 and MI5. Um, the reasons are sort sort of uh quite a lot of them are fairly obvious, right? It's to do with what's going to happen with drones, what's going to happen with cyber offense and cyber defense, um what's going to happen with um the potential for creating new kind of like biological weapons, chemical things, etc., etc. So those conversations have already begun. Also, there's a there's an obvious conversation which Leopold's—I say kind of refer to—which really until his essay came out was was not really a very discussed subject in public, but it it was this question about security of the labs. You you had a situation you've had a situation in which on the one hand some of the core entities of the intelligence services are starting to think incredibly hard about what the about how these companies and these products and these models are going to disrupt power and international relations and and war, yet if you go to these companies, they're kind of like classic Silicon Valley companies, right, with basically like no security and everyone in beanbags wandering in and out. Um, very good cybersecurity in lots of ways obviously at places like Google, because you don't want um uh uh you know hackers trivially hacking into Google, and they're obviously extremely good at dealing with things like that, but once you start dealing with power you're into a whole different ball game; you're into a ball game where intelligence services um run blackmail operations against your family, where intelligence services run honey traps against the head of your security to escape the cybersecurity defenses that you've got. There's a whole bunch of things that state-level actors can do to these companies which um is outside the normal realms of corporate behavior and corporate defenses and all the kind of normal things that people get involved with. So I think it's going to be inevitable uh it already sort of is happening that um deals emerge between these companies and the Deep State in various ways whereby the Deep State gets access to information about what's going on; information is shared; potentially negotiations are for people to be kind of embedded in these organizations so they can see what's going on, and at the same time um the Deep State will provide security of a kind that only states can only people like the FBI, the CIA, NSA, etc., can deal with um a situation in which a PLC Intel is is is honey trapping your head of security um and those kind of problems. So I think that that that's already starting to happen, um and the connections between the old freewheeling world in which a lot of these companies kind of um you know just had a normal kind of tech life in Silicon Valley in a certain kind of culture, that world is rapidly coming to an end for sure. Whatever else happens, that's those days I think are over, right? And maybe the economic stuff could generate deeper pressure for state control.

Exactly; it seems almost inevitable that that will happen as well, right? I mean, can you imagine what happened—so I'll give you an

Example: um, uh, someone I think you you know as well as I do, brilliant physicist called Steve Sue. One of his startups, um, it's called Super Focus, and it's working on uh, um, automate um, try customer support things. So imagine your say Samsung or someone—I don't mean that particular company, but just as a placeholder—imagine dropping all of its like brochures, Q&As, troubleshooting docs into this model. You train in model narrowly on all of that, and then what—how can it do customer support? Well, it turns out that, you know, the vast majority of those kind of things are basically automatable now. You look at a country like the Philippines; something like 10% or more of its entire GDP is basically that industry. So you imagine suddenly, um, you go from a world in which 10% of your country, if your economy is operating on something, and then technically like the vast majority, if not actually literally 100% of it, is can be gone, like can be gone like that. Obviously, there's going to be intense political pressure to try and slow that process down, as as has happened repeatedly.

Part of the problem, um, with this is Mark—um, there's a few podcasts that Mark and Reid, one of the kind of guys who helped build the internet and web 30 years ago, now a VC guy—one of the things he talks about is that whenever this cycle happens historically, it's much easier to imagine the things that will be destroyed and the jobs that will go away than it is to imagine what will replace them and what will come. So when you see these revolutions happen, you always see everyone warning, "Oh, well, that'll be terrible for so and so, terrible for so and so, terrible for so and so," and it is, and a lot of those predictions are correct, but almost no one/actually no one says, "But in 20 years' time, this will be, this will exist, and this will exist." So so these political arguments tend to be very asymmetrical, with a very clear set of losers, um, who will lobby and potentially with a loud voice uh uh for things that are not necessarily in the interest of the broader broader Society. Right. No, I think that's true. Again, it's the time scales that and read training times that that cause me, give me pause with, you know, the long-term gains, which is to say, so I'll ask you one last uh sort of question, and then we'll we'll open it up to a distinguished audience, which no doubt has things to say. So, um, coming back a little bit to uh your uh narrow political experience, you know, running campaigns, uh, even, you know, winning them when uh with with a high percentage, how is the practice of politics likely to change over the next 5 years? I didn't get the sense with the American campaign that, you know, that there was AI-based control in the background. Maybe there were some number of videos produced and that kind of stuff, but it seemed ultimately fairly modest. But in the face of these advances and general commercial availability of fairly sophisticated systems, then presumably the capacity even to pay for on commercial terms, term, so what might an election campaign look like 5 years from now? Um, um, so I think, so I'll make a general point first. I think a general point is that a lot of these kinds of tools kind of stretch out a phenomenon that already exists, which is a very small number of people are just much more capable of building certain kinds of organizations. Right. If you take it to the Limit, it's people like Elon or General Groves. There's not very many of such people, and their capacity to build things at speed and scale is um, it's not like a a 5% or 10% or 20% difference; it's like 100x or something kind of difference in capability. A lot of these tools kind of stretch that out even further because they mean that people like Elon can do even more, even faster, whereas the vast major—whereas if you compare, like take a whole set of tools, those tools, a small category of people on the power law like Elon, can then use as massive force multipliers, but if you give the same tools to the median MP, they're basically as capable as they were before—not very capable. So I think it'll stretch that whole process out even more for sure, because that's just the sort of long-term history of the of these things.

Um, I mean, one one example I would give is that uh, so basically everything that you read about the Russia hoax and me, Cambridge Analytica, Facebook advertising, blah blah blah, and Brexit is complete, like just complete nonsense. The the interesting thing technically that we did in the referendum was actually use AI stuff for polling, and there are possibilities to do polling much better. So the guy who did the models for me in 2016, guy called Ben Warner, then did models for me in the 2019 election, um, and they were by far the most accurate predictions. Um, he predicted 364 seats, and the result was 365 seats. And those kind of um, models for seat-by-seat outcomes are obviously very useful in elections because that you want to know where should your marginal pound be spent, where should your effort be spent, um, what physical locations do you send the candidate to, what are the actual crucial issues in the crucial places? Right. So if you can build models that are that are much more accurate of that kind of thing, that obviously they they have a lot of value. Now this guy who who did those for me is now doing a startup, and it sounds a bit sci-fi, but trust me, it actually um works. So he had the idea of, is it possible to run what you might call synthetic polling and synthetic focus groups inside large language model models? And it is possible. So, for example, if he were here now on his laptop, if you had a normal political discussion about election and you said, "Well, so and so is going to do an advert on blahy blahy blah, and they're going to put it in Pennsylvania, and they think that it's going to persuade uh women between 30 and 50 who are non-college graduates living in Pennsylvania of this message," what do you do? Okay, well, you hire some polsters, and they go off, and then you hire some focus group people, and the focus group people recruit people, and that takes time, and you have to get them in a room, and you have to have a discussion guide, and you and the normal kind time cycle for this is say a couple of weeks if you do it very very quickly, and you'll spend many many thousands of dollars on it. A lot of companies charge like between $5 and $110,000 for a focus group to talk to like six or eight people. If Ben were here now, he he could type that request into his laptop and say, "What do women between 30 and 50 non-college graduates in Pennsylvania, white or black or whatever, think about the following question?" And it will generate a focus group transcript of synthetic voters. And then if you compare that transcript of synthetic voters—by that I mean the people don't exist, right, the synthetic creations created by this, you know, linear algebra inside the inside the model—you compare the output of that with an actual focus group transcript of the same people, most people can't tell the difference between the two, and you can generate that in 30 seconds. So it looks like his idea will work, and if it does work, then it'll the whole multi-billion dollar industry for market research generally will be turned upside down, but also particularly so specifically your question about election campaigns; it will obviously transform election campaigns because, as in war, speed is critical. If you can sit there and go and get a rough answer—because you you want rough answers; it's all you need in politics—you very rarely does it matter like if the right answer is like 58% or 62%, right? What you're actually looking for are the very big questions. So if you look at the last election, for example, um, it was obvious in quarter 2023, quarter 1 2023, that the the three biggest issues in the US election were cost of living crisis, the southern border, and the president's style. Those are the three big things that you're trying to look for, and those are the three big things that ended up defining, define the election. So if you've got a an automated system that can look for these big things and roughly figure them out and figure these things out fast, then obviously it's going to change, um, it's going to change communication generally and political communication in particular. And this assumes that the model is fairly up to date; it's been fed recent information, recent newspapers, recent—now if you combine that with another set of technologies, right, for generating video, you're then very quickly in a very very odd world where one person can sit in a room, and they can say to system A, "Generate me an advert that persuades the following, the the 30- to 50-year-old women in Pennsylvania of the following proposition." Up comes a 30-second video. You then drop that 30, that 30-second video created by one model into your synthetic polling thing, and then synthetically poll the AI-generated content. It seems like a kind of—C, like if you said it anything like that 10 years ago, people would have just thought that's uh, I mean, maybe like in 100 years or something that would be possible, but that's basically doable now in crude form. Like this is not—no deployed products and everyone using this, but I've seen these things happening in front of me, and um, something roughly this is going to be happening, and that's going to be another—if you if you imagine, if you consider that the old media—now if you pick up the Times newspaper and and look at how it covers polling done in the old way, I.E., created in the 1940s way, its coverage is hopeless and incompetent repeatedly, right? And that's been around for like 70 years. So imagine how completely incapable the old media was going to be of explaining to people what the hell is going on in this new environment.

One other aspect of running a campaign is, as you say, to persuade 30- to 50-year-olds, but on the other hand, if you could persuade at the individual level, you'd be even better off. Yeah. Now a lot of commercial information is in principle good enough, right? Which is to say your phone company knows things about you, and and they do sell that kind of information in some aggregated form, but I I don't know for a fact, but my understanding is that in fact you could buy a lot of commercial information which could help you understand why much more narrowly—not just 30 to 50, but in that particular town or something—why. So uh, so then the the language model can generate, you know, it's it's customized appeals at some incredible rate, and then of course, you know, send off emails and so on. So is that in on the way? That for sure, for sure, that's basically happening now. I think I think people doing it are keeping it quite quiet, but I've talked to people just in the last month who are doing things like this, um, for sure that that's going to happen, um, and again it makes it it makes everything um uh faster and cheaper because because you can um personalize these things instead of having to do, instead of printing, you know, instead of trying to set up say 100,000 generic messages top and tailed to a set of MPs to try and influence them, you'll be able to send—you can today send—tailor—you can drop them into Claude or ChatGPT or something and get those models to basically tail, customize them, so the MP thinks they're getting an actual customized letter—doesn't even realize that it's all being produced by the by the model. So for sure that's going to happen. Um, I mean, there obviously there are some restrictions. Uh, America is less regulated than Britain, but there are still some restrictions on how you how you're allowed to combine data, different files, like photo files, financial data. I think in particular is it has various—M—regul controls on, but I don't know the details of that. Um, but if you think that's only if you care about obeying the law, yes. So someone is going to have to think about what if the PRC or Putin decided to do what didn't happen in 2016, but that now we actually are going to do it. They don't have to care about US law, obviously by definition, so they could definitely bring together every available data set in one system, um, in a way that no US campaign could do legally. So that's another thing that they're going to have to worry about—that foreign actors could um, what fore could potentially develop things that are actually more able than anything that exists domestically because of your own domestic regulations. Just one of many thousands of problems coming at the system. No. So Steve Rosen, whose work you know of, um, you know, strategic thinker, me, he—if he's just giving a talk a couple days back—and this point was, in the age of precision long-range strike, when you have peer competitors, you know, so that's back to a stalemate, at which point exploiting internal division is the thing to try and do. And from what you just said, that that will—the technology for that is here. Yeah, very good. So uh, I could keep uh, come occupied, but you've all been very patient, so uh, I think we should open it up. I see a hand up. Can I just ask, we are streamed online, the audience can hear us. You just microp app. Thank you for a really interesting and informative uh discussion. Um, I have a question, which is um, for for Dominic, which is, what do you think are the implications and risks of potentially leaving large parts of the World Behind in terms of tech and AI, um, particularly, for example, sub-Saharan Africa? And related to this, do you have any regrets about supporting Boris Johnson's uh decision to merge Divid and FCO? Thank you.

Um, on the first one, it does strike me as likely. So there's already a trend underway of the internet, internet kind of bifera at least, right? Into a kind of US version and the PRC version. It seems to me almost inevitable that that will continue, um uh in coming years, and the development of of the AI models will um will exacerbate that. I think increasingly that will be a like a core—well, it sort of already is a question. Um, so China's been extremely active, as I'm sure you know, in Africa on this front. It's had people actually deployed there for years on doing things like facial recognition and training models on on African faces. No one down there has being protected by GDPR or anything like that. Um, no one even knows really like what the hell the nature of the contracts are between the government of whatever down there and PRC. Right. It it seems so—I'm not at all expert on this, and I haven't thought an awful lot about it, but it seems likely um to me that increasingly countries will have to choose: do you kind of join in on the PRC infrastructure for all these things, or do you join in on the um the US infrastructure? And very few countries, um, possibly like say Russia and India aside, are going to have a lot of meaningful choices on on on a lot of these things, including the EU. Um, if you you look at the EU, uh, there are no relevant companies um at the frontier inside the EU legal structure, and you also have the EU has passed the EU AI Act, which, whatever one's view on how to—in in my opinion—uh different views on on the the longer-term questions about AI safety, the EU Act is kind of the worst of all worlds because it doesn't actually deal with a set of safety questions um that it purports to, yet it also massively kneecaps EU technology industry, and there's already driven lots of um young people to California. So in the EU currently says, "Well, we're going to be an independent player in this, but you can only be an independent," and it says laughably to me, "Well, we're going to be the global leader in trust in AI, but not in the actual technology of AI." That's not going to work; that's obviously doomed. So the EU um brought in GDPR with the intention of doing various things to Big Tech companies. They were told it would have the opposite effect; they didn't listen; it had the opposite effect to what they intended. It's now done the EU AI Act; that's a disaster. The—so you think that because of me and Brexit I'm I'm partial. I would just point out that Macron and others have been saying the same thing that I said during the referendum campaign. A lot of EU leaders themselves are now saying what Vot Le said in 2016—that we can't go on with the commission regulating like this, otherwise we're going to be turned into a a museum and nothing else. So you can see coming down the line—just to broaden your point—I don't think it's just about sub-Saharan Africa; I think this is going to be a global question. Um, there's going to be two huge poles. Russia will go will go its own way, though in lots of ways be Pro be probably like entangled in the PRC thing. India could imagine trying to do its own thing as well, but almost everyone else is basically going to have a choice of by American or by Chinese. Israel is an interesting case. Israel is an interesting case. You're right, but in lots of ways I think they'll they'll be probably they'll be a hybrid, right? They'll buy American, and they'll do their own customized things for certain kinds of security issues, and Ilas Atco has already set up a lab partly in Tel Aviv, so they they have a leading yeah person already. Israel, of course, Israel is is very very unique, um, and if you talk to the British intelligence services and British Special Forces, you also get a very very strong impression of just how amazingly unique Israel is. That's because they've got—obviously they've got a a bunch of extraordinarily smart people, and B they've got an actual existential situation in which they're constantly worried about getting killed, and they've got a power that might go nuclear, and etc., etc. So they've kind of got a set of incentives in Israel to build a whole load of things and to take building technological capability incredibly seriously, um, in a way that the EU just hasn't hasn't felt it at all. And you can see this in the response to Ukraine, right? I mean, so I had lots of these arguments in 2020 about drones in the future, uh, and at that point it was a much more kind of theoretical argument. A lot of people in the senior people in the MOD didn't want to hear it. Any rationalist, particularly in a place like Oxford University, would look at the last sit, the last couple of years' situation with Ukraine and say, "Oh, obviously the MOD must have changed now," right? Absolutely not; in fact, the opposite is happening. The MOD has doubled down on funding all of the things which are completely broken and useless, and it is funding that by stealing money from the future and stopping people building stuff with drones and AI in the future, exactly as you expect from pathological uh pathological MOD. So Israel does not operate like that. I highly recommend this book by Ed Lutwak and Itan Shamir on Israeli military innovation; it's really fascinating. Um, good. Yes, please. Thank you. You spoken a lot about like pathological, like you just said pathological MOD. You spoken a lot about how the fact the British state is almost sclerotic, and from what I could tell you don't think it's kind of full of very smart people, like you said Israel was. So I'm just asking, what what's the solution? You said, you know, nothing gets done in Whitehall because the Treasury and Cabinet Office control it. So what, how do you how do you change, you know, the Civil Service, the British government, to make it effective? So in in a nutshell, it the the there are there are lots of incredibly able people in the in in the British system, but they tend to be young. And what happens is that um young, smart, idealistic people come from places like this, and they start in the Civil Service in their early 20s. Then what happens is a reverse talent filter kicks in: the the most dynamic, able, entrepreneurial, hardworking quit by the time they're about 40 because they look around at this—at that point their ideal, their their idealism has been beaten out of them—they look around at their bosses and they decide they don't want to spend the rest of their life like that, so they leave. The worst kind of—e—of um of HR type then get promoted to the senior jobs, and that's why in Co you had so many things completely collapsed because you had a bunch of completely useless people as permanent secretaries. So the caliber of people at the top of the management of these organizations is imploded over o over a few decades. The in my opinion the core disaster was Northcote-Trevelyan, and the creation of the permanent civil service, and that is the thing that must be destroyed at all costs. That's the thing which is killing us; that is the heart of the pathology. And I recently came across a fascinating letter from Palmerston to Queen Victoria in roughly 1830s, and he says in that letter—he's comparing trends in Europe and England—and he says, um, "Well, obviously in Europe what's happening is there's the growth of these permanent bureaucracies that are taking over everywhere. Um, that system could never and will never happen in Britain." And if you look at how people like Palmerston actually ran government departments, then they were actually responsible—responsibility was real; their responsibility to Parliament was real. What we have now is fake meritocracy in the Civil Service, which actually excludes roughly 100% of the most talented people in the world, and completely fake responsibility of ministers to Parliament, where they're actually not in charge of anything, and you have fake Cabinet Government overall laid on top of that, where the ministers walk down the street and they smile for the cameras and blah blah blah, but—camera—now is Cabinet—now is completely scripted. People don't believe me when I say scripted; they think I must—there must be an analogy, but I mean it literally. So if you're with the Prime Minister and you have a meeting now—this is how a meeting happens—you go to the Prime Minister's study before the meeting, and he opens a folder, and inside the folder is a document called Chairman's notes, and that is an actual script, and and the script says, "Here's how you read it out to introduce the meeting," and then all these different ministers will read out the following things, and then at the end you read out, "This is the conclusion." So the conclusion is drafted in advance, right? So Whitehall now operates on the basis of completely—literally completely scripted meetings, where it's only a bad meeting if somebody goes off script, at which point the officials scurry around afterwards and figure out how to clear it up because people actually discuss the real issue. My God, like we—this is not how meetings work, right? So you have a whole set of people who are officials and ministers who've been trained to live in an environment which is the exact opposite of—if you go and had a me a meeting with Jeff Bezos or Elon Musk or with William Pitt in 1798—the idea—sorry, yes, 1798—thought I got my centuries confused there—um, the idea of Pitt sitting there and like reading out a script in bullet-point fashion, and then all of these clowns like reading out their bullet points and then him reading out his—I mean, would never have happened in a million years, but that's the way that we are actually governed. It's inseparable from Northcote-Trevelyan. Northcote-Trevelyan has to be blown up. The same path—the reason why what we discussed before about why you see the same problems across the Western world is this same thing: every Western country has allowed these permanent bureaucracies to develop, and that's why we have this exactly the same pathological behavior everywhere. We should return to the Old English system, which is real meritocracy and real responsibility to to Parliament rather than the fake meritocracy and the fake responsibility. I assume everybody knows about the Northcote-Trevelyan reforms, right? Not just—not just D. Okay, good. Joe, and then talking about the 1790s, I think you there—sorry, talking about the 1790s, so the time when BR—there was this called National Industrial—you know, things like the dockyard, Chatham and ports with these huge massive places integrated into into the state with—and now we kind of don't really think that Tech—they're kind of very separate from the state, but with Elon Musk kind of being so much part—

Of the new Administration, can you foresee a thing where, at least in America, there is this sort of national industrial thing where these kind of places that are the tech leading are somehow also integrated in the state, or closely integrated into the state, the way, say, the Dos are and maybe Israel is? I don't know.

Yeah, yes, you, you. I think you kind of easily imagine that. Um, I mean, it's obviously completely unclear at the moment how the whole experiment has got is going to work out with a section of Silicon Valley Elite deciding that enough is enough; essentially, like the system is so bad we're going to have to put aside 30 years of thinking that we just get on with building our companies and ignore those people thousands of miles away. The view in Silicon Valley always was that being on the west coast and thousands of miles away from DC was actually a great blessing because we can just do our own stuff and they ignore us and leave us alone. But precisely because of the issues that we've been talking about, these worlds are just inevitably colliding now; they neither side can leave each other alone. And what happened is the Democrats decided to go for a subset, a set of Silicon Valley, and Silicon Valley, which wanted to be left alone, decided, okay, Two Can Play That Game; we're going to take you out, which is what's happened. What happens in government, though, of course, is a very different question.

Um, now if you actually have people like Elon go to the White House, the center of power, and say, for example, "We are not going to try and reform the US Air Force to make it friendly towards drones," we know how the process of these reforms works out: everyone just shares PowerPoints for years to come, exactly the same as what happens in the cabinet office when you try and talk about AI. It's completely pointless; that's not how Elon works, right? Um, so there's only basically two ways in which someone like Elon can operate. Option one is you go into the Air Force and fire the vast majority of the people and restructure it completely—very, very hard to do in government. A lot of things you, you can do; he could do a Twitter or legal in various ways. But the other option is you just say, "That's okay, we're going to do it differently. All the money that US Air Force was going to get for drones is no longer going to US Air Force; instead, we're going to create the Drone Force. It's a separate legal entity; we will empower it with a different set of laws; it will be exempt from all of the normal, um, uh, procurement rules and blah, blah, blah, blah, and it will actually build like Silicon Valley does—I.E., super fast and optimized for speed and engineering, not optimized for Congress, all of the insane DEI stuff." Which they put in parentheses; if you look at what happened on the chips, for example, Biden tried to put tens of billions into chips; less than 1% of that money was spent, why? Because Congress insisted on putting in all kinds of DEI communism into the system to buy off various Democratic, um, uh, pressure groups and trade unions and whatnot. Close brackets. So you can, like, to me that's like, that's, that's the big question.

Um, um, not, uh, you know, is there some entity like Doge set up outside the government that gives some advice, because that will not really affect very much. The core question is, um, and it affects the deeper questions about the AI staff: does a subset of Silicon Valley essentially move to DC and sort of take over at least parts of how the system works, and therefore you have this new Elite, political Elite emerging and a much more entrepreneurial culture, culture and a legal shift in how a lot of these institutions work? So if that happens, then America will look extremely different in a few years' time, and the gap between what's happening in America and what's happening in Europe will grow even faster. But of course, the whole point about why I describe these things as pathologically is they act like—in, as an official said to me, "Dominic, you need to understand, you are a virus, and we are the immune system; it's our job to expel you from the system as quickly with as little damage as possible." And that's how these permanent bureaucracies see the world, and that's how they will see Elon and the whole Silicon Valley Network that is talking about going there. Um, and those people play, you know, it's for, it's for the highest stakes; it's for power in the most powerful country. So, um, a part-time Silicon Valley approach, I think, won't work because it won't be able to break through how those things work. But the ability of the people is such that if they actually decide to go for it and they go to Washington and they say, "We're going to rip all this up and we're going to start rebuilding it," and they have a White House and president that reinforces that, then all sorts of extraordinary things are obviously possible. Thank you. One, one more, sorry. I, okay. All right. Well, maybe we should switch sides for a minute and then come back. Yeah, hello.

Um, just to confirm that Dom's story about scripts is true, and I, I, I've written those scripts myself, but it, but it's quite rare for the script not to have been agreed with the minister in advance, uh, but that depends on the minister. Yes, exactly. They're happy being NPCs. Yeah. Well, I wouldn't necessarily use that language, but, but the story is completely true, um, and it still goes on. Um, my question was slightly different in that, um, I, I, I think of this problem as an economic historian, and one thing that's true of industrial capitalism for hundreds of years is that capitalism is internally self-mediating, and that if you're in control of capital, it's hard to realize gains without employing people. So the deployment of capital is the combination of technology, people, techniques, and all kinds of things to try and realize a gain. Um, what AI raises the Spectre of is that you can raise gains without needing other people, or needing far fewer people. So that implies that you'll have progressively more concentration of wealth, and you'll have a separation of interests whereby people in control of capital don't need anybody else to realize upside, which is new; that's never happened before historically. My concern from a societal level, and I think this comes back to your first two questions, is like, what's the likely view, what's the benign view? Um, if you have large swaths of the population displaced from productive activity, they're no longer contributing, and also you have—especially in Europe—States constructed on the basis of taxation, and most of that taxation comes from income, social contributions, um, fiscal transfers; very little tax actually is derived from capital itself. How do you maintain stable Societies in that sort of future? Um, do you arrive at a situation where people try and exercise political power to prevent these things from happening, or uh, do you have a simply more redistributive Society where the state gets serious about taxing capital? That, that's the sort of future I think of; it's hard to know which one of those two will happen, um, but I, I, I'd be interested in both of your views on that one. So I'll make a physicist point, which you've sort of heard me make, say, along the way. You know, when I open an economics book and I look at it, you know, it reminds me of thermodynamics, uh, which is not shocking because I think, well, some of the founders were actually people who learned thermodynamics, and then you ported the theory over, but that's near equilibrium, right? So in economics you're often having arguments about we move this variable and, you know, the new equilibrium, you know, and their linear elasticities which you use in some general sense, right, to, to figure it out. Uh, and in physics also if you, uh, you know, if I slowly heat up a glass of water, it goes through a succession of near equilibrium states to which thermodynamics applies, but if I were to suddenly drop in there, you know, I mean, if it suddenly raised the temperature by a million degrees, it's a completely non-equilibrium process; it may eventually settle down to something, but in between you have a description which is now completely much harder and much more different. So again, I'm just bringing the set of metaphors I have as a physicist, you know, to the discussion, which is to say I look at these time scales and I go, "We're heading for a very non-equilibrium regime," and maybe you can tell me that there is an equilibrium state very far away which involves maybe some of what you're describing, but if the process of getting there is vastly quicker than normal equilibrating processes, then, then I do have to wonder what that, you know, sort of journey looks like. So that's sort of the picture in, in, in, in my head. I mean, you could imagine in principle ways of slowing it down. I mean, if you thought of harm, you know, sort of people being laid off as a harm, you could—in economic jargon—have a Pigouvian tax, right? I mean, a carbon tax is supposed to be that, or something, but as of now I don't think these are things that anybody's discussing, partly because of the international competitive idea that, you know, if you don't get to it, then, so, so all these things are tied together I think in a way that's very, anyway, but over to you for maybe, yeah.

So, so, um, couple things. Point one, I think, um, this agent thing is already happening, right? One of the most interesting things that's happening now, I think, inside these companies is experiments with, um, kind of agentic companies; the concept being that, um, uh, you can, that the models can essentially create, or possibly not create, um, they, they given some high-level goal, and then they're told, "Build a company that can go on the internet and do the following things," um, and with little other than setting some goals inside OpenAI or Anthropic or whatever, the agentic company then goes onto the internet, starts beavering away, and builds itself. We've even seen a weird, weird kind of example of this; so there are some kind of very sort of fringy experiments that have already happened and sort of leaked out. One interesting thing to Google is, um, a guy set up a kind of crypto chatbot; he put it on Twitter; the crypto chatbot tweeted at, at Mark Andreessen, one of the top five, um, most influential, uh, Silicon Valley, um, VCs, and it asked Andreessen for funding. Out of curiosity, Andreessen publicly on Twitter—you can read the whole Twitter exchange—why did $50,000 in crypto, in Bitcoin or something, and this entity then, um, uh, this, um, crypto bot has then, then started marketing memes and then created a meme coin currently valued at something like $300 million. Now, who knows, maybe that'll all plummet to zero tomorrow, but the point isn't like what happens to that exact thing; it's just these weird things like are already happening on the internet, um, and these are just kind of like fringe little experiments at the moment, but inside OpenAI and, and the big companies they're experimenting and assuming that that this is going to be a normal thing. So at the moment people think, "Oh, OpenAI is a company." Shortly, OpenAI, OpenAI might be a company with a thousand companies inside it, all off doing what, doing what you're talking about.

Um, my second point very quickly is, um, is the, is, is just going back to the, and recent point, as a, as a caution. Historically, we always think, we always see how things are going to be destroyed, and we never see how things are going to be created. Mark would say—if he were here—every single previous revolution has the same warnings; every single time. Humanity is so creative that people go on and they do new things; that's always happened so far without exception. So we shouldn't just assume that, um, or at least we should be very cautious about thinking that, uh, "Oh, these things are just going to destroy all of this activity, and humans are going to have no role," because that's not how things have played out before. The obvious answer to that is intelligence is intelligence is a quantitatively different thing; therefore, history might not play out the same, but I'm just putting the, the, the counterargument. The third thing is the point, um, that, that you mentioned, the international competition point is critical, I think, because this is driven, a, this has been central to the whole process since kind of Isaac Newton and the Industrial Revolution and everything a few centuries ago, right? Every country has had, had this question; you know, you can read endless things in the 19th century about people talking about the influence of railways, the influence of the telegram: "This is the devil; the devil is afoot; no good will come of it; atheism will spread; chaos will come." You can read letters from people like Metternich at the time predicting what will happen if all these dynamics continue. If you then fast-forwarded him to 1940s Europe with, um, Auschwitz and whatnot, he would say, "See, all of our fears of a century ago were completely vindicated; these, this automation process, these technologies destroyed the old world; they destroyed Christianity; they destroyed our old constitutions, and, um, you've got industrialized slaughter of millions, um, we were, we were right." But no one's been able to get off the ladder because if you get off the ladder, then you're basically enslaved by your, by, by other state actors. Um, and if you go back and read Vannevar Bush's famous essay in 1955, "Can We Survive Technology?", he points to this dynamic and, and says, you know, "There's not, there's not really any way out of it." The theoretical way out of it is, um, is some kind of one-world government, and therefore it's not surprising that some of the fringe elements of the AI safety movement are essentially now positing a kind of Stalinist one-world government as the only way actually to control AI, which is sort of logical in one sense, but raises the obvious question that, um, we saw how Stalinism played out—death on an unimaginable scale—and having a global Stalinist government able to restrict and control technology everywhere and anywhere, um, is, uh, for sure going to be catastrophic. So that's the fundamental, um, that I think that's the kind of a kind of fundamental dilemma that no one has a way out of, and probably there is no way out of it.

Probant, do you have a question? Yeah, yeah, you haven't talked very much about what governments should do, uh, about, uh, AI so far, so therefore the question: if you were special advisor to Rishi Sunak, what would you advise him to do? So the first thing I would do is I point out to him how he has destroyed his own capability on AI without even realizing what he's done, because of how GCHQ and, and, and the Cabinet Office work, uh, and I would say to him, "Step One is obviously to reverse, uh, the blunder you made on day one without even knowing you've made such a blunder, uh, so you recreate an actual, um, high-performance elite team with these kind of skills around the PM." What COVID showed extraordinarily clearly is the incredible difference it makes between if you have a PM in a normal cabinet room with, like, 1914, a bun, like, 20 different, uh, guys with very similar education, pens and paper sitting around who don't understand graphs and don't understand technology, then you're going to have disasters, um, versus what happened when we brought in people that did know these things, and the, and the extraordinary difference it made to the conversation because the politicians then could actually have actual real experts in the room explaining to them what these graphs meant and, and, and whatnot. So I would say before you get to any kind of serious policy, the, the mo, the critical thing is you have to have people with technological expertise near the PM, but you can only do that if the PM clearly takes it seriously, because all of these people can make massive amounts of money and do all kinds of fascinating things, or, in, in companies or be in a place like this, which is very conducive and lovely for people to live in; no one wants to be, no one like that wants to be in a non-job around the PM. There's lots of officials who are happy being in non-jobs around the PM, but the technically able, technological people don't want to be in that, in that thing. So you have to have a PM that actually wants, takes it seriously; a PM that puts his own power and responsibility behind taking it seriously and then has these people there. The second thing I would do is I would, um, I would approach the AI problem slightly obliquely, and I would say, "We've just had COVID, um, across the Western world; that was a fiasco. We're going to create the actual entity that should exist to deal with future pandemics, because for sure, um, we are going to face future pandemics, and possibly pandemics in which AI plays a role, given that these models now are, people are experimenting with these models in terms of creating bioweapons and other things, and this is like a live issue inside of National Security Council and other parts of, um, similar sorts of entities in, in London." So I would say, "Let's build the actual super-duper pandemic entity that should exist but is prevented from existing by current Whitehall." Such an entity inevitably will integrate AI expertise into it, and then you will have a set of people who are thinking about possibilities for death on a large scale and war and terrorism and how Whitehall works and how power decisions are made at the center of power and how AI models work for prediction and for helping all kinds of other capabilities, and you'll be training a whole set of people on a daily basis to figure out how do, how does AI and, and power and death, how do we actually manage these things in a live setting? So every day these people will be waking up and looking at reports coming in from the Congo or whatever, saying, "The new Cade of blah, blah, is duh, and the model predicts blah and blah and blah, and what do we do about that?" So you'd have this, it very, very iterative, um, process, and it's a sort of thing where you could, it's so important, right, because you act, you could literally save a billion people's lives; it's a very, it's a great job for a set of people if it's taken seriously. So you could imagine the PM saying, "Okay, we're going to build something, but it's going to be outside all the normal horror show; it's going to be outside Whitehall HR entirely and outside Whitehall procurement and outside all of the, all the things that make high-performance entities illegal inside Whitehall." Here's this mission; it's going to be part of the PM's office; don't—and if you created that, you would then have something which, um, which I think would be an enormous asset and would be training people to think properly about all of these questions and bringing together theoretical questions and very, very practical questions, engineering questions and organizational questions. That's my best solution so far to this. What about policy? Policy comes from having people; so I think the most important thing, you know, if you look, um, if you look at the last few years, I'll give you an example of policy: the Online Harms Bill, a total fiasco. The Online Harms Bill is there to deal with a much, like, a, a sort of in almost infinitely simpler problem than the ones we've been dealing about today, but if it's completely beyond Whitehall to deal with that, they've created an Online Harms Bill which is completely farcical in almost every way. So my point is it's sort of pointless thinking about policy while you have Whitehall structured the way it is because it's just programmed to go inevitably wrong. I don't want the current Whitehall to regulate or do anything with AI because it's just absolutely definitely programmed to completely mess it up. You have to think at a more fundamental level; the fundamental level is true expertise and actual high-performance organizations. If you create that, then you're in a position that you've got something that could actually generate good policy.

So there's one last question there, and then we have, we'll move to the reception, and you can certainly ask other things, but yeah. That, that person, why should it, why should we view the government being resistant to AI because they're resistant to change rather than being resistant to an implementation of AI which will largely, to me, benefit those disproportionately who are wealthy? So I didn't quite understand the question; say again. Why should we view the government as being resistant to AI because they are resistant to change, yeah, rather than being resistant to an implementation of AI that seems like it will largely and disproportionately benefit those who already have power and are, all three seem to be quite wealthy, sort of the Silicon Valley elites? So you're suggesting that what's happening at Whitehall is that the people in charge are thinking through an approach, and their approach is, um, doing certain things on AI would help billionaires; we don't want to help billionaires; therefore, we will do what we're doing. Is that what you're suggesting is currently happening? Um, sort of a resistance to an AI which has largely been driven in the interests of those at the top rather than having more of a holistic view, societal-wide? I think you just, you, you enormously overrate the amount of thinking that's, that, that's going on about these things. I mean, if, if your model of these guys was correct, then they would be acting in various ways to scupper billionaires with AI, right? But they're not. I, about hesitancy; why seem hesitant about figures like Elon Musk and such when they say we want to implement things this sort of way? That, that your model of this can't possibly explain why the PM would nuke his own capabilities inside his own office without even realizing what he's doing. It's not, most of these things are not, um, if you talk to like the median MP or the median cabinet minister or the median Shadow cabinet minister, they're not thinking about any of these things; they're not, it's not a subject of, of interest there. And also, I mean, this question, people in this room, there is also an element of the established science world and the established AI world which has various power and budgets at the moment and is very keen on telling political people, "Don't worry, this is just not really very fast, and this is a lot of Silicon Valley hype, and we don't really need to get them very, very concerned." So there are also elements inside the kind of traditional scientific world bureaucracy which, for their own reasons—fundamentally to do with power and money as usual—don't want the government changing things, and therefore they are also telling MPs, "Oh, a lot of this stuff is overhyped, and, and you don't have to engage with it." So you've got a bunch of MPs who, anyway, their default program is just wake up, see what's in the old media and babble about that all day rather than think about sort of questions we're discussing today, and then they're told by some respected chap from the Royal Society, "So don't worry about these Elon Musk characters," so it's all, you know, a lot of California nonsense. Then, of course, they can, they can do what they've been doing for the last X years, which is just sort of ignore it and not take, not take a lot of it seriously, and then get super surprised when certain things happen. I think people are worried about Elon Musk figures; my point—well, they're worried about Elon Musk-like figures for sure, but not in any sense because they, because they have, um, an accurate model of Elon and an accurate model of AI and they're putting these two things together and then coming up with some kind of accurate model of what's going on in the world; they just, a lot of politicians just have a, "Oh my God, the billionaires," but a certain kind of billionaires, right? So if you look at, for example, the billionaire class in the last election was very, very strongly for the Democrats, not for the Republicans; Democrats raised far more money from billionaires than the Republicans did. So it's which class of, which class es of billionaires is also a critical thing, and this goes back to an important, I mean, very, very complicated thing, but it's where the Russiagate hoax about Trump and Brexit collides with psychological dynamics inside the Democrat Party over the last few years, which itself generated this backlash from part of Silicon Valley. Part of the reason why what's just happened has happened is because after the Russiagate hoax spread, a whole set of Democrat politicians said very, very clearly to Silicon Valley, "You guys better make sure that all this AI stuff is politically acceptable," and by politically acceptable what they meant was espouses AOC.

Like politics in public, hence while you see these models over the last few years have all come out, they're basically like AOC parrots. As soon as you get them onto the subject of everything to do with sex, race, history, etc., you had the ludicrous thing recently with Google getting, making a complete global spectacle of itself with its image generator thing and making everyone black from the 1780s and whatnot. Um, that's that was to do, that was precisely because of behind-the-scenes uh pressure. Same kind of pressure and same kind of dynamics were what led to um embeds from the CIA and FBI and whatnot inside Twitter actually doing live editing and live censorship of things on Twitter at the behest of various agencies inside the White House.

So that whole thing has created this backlash inside Silicon Valley, which is a lot of the Democrats essentially—sorry, it's connected to one other thing which is very important. Two weeks before the last before the last election, John Kerry, Presidential nominee for the Democrat Party in 2004, publicly stated on TV the First Amendment was a historic mistake, and after we win the election, we will fix that mistake. If I or anybody predicted that such a person would have said this even 5 years ago, everyone would have thought that was completely insane, couldn't possibly conceivably happen, that Democrat leadership would describe the First Amendment as a historic error. But it shows the direction of travel of a large fraction of the Democrat Party, and a large fraction of the Labor Party, and a large fraction of academia. These things are all are all connected up, and that's part of the reason why Elon and and another and a kind of counter-elite network inside Silicon Valley decided to um uh.

So two years ago, I had summer here, I was here in summer, and I had lunch with my old tutor, Robin Lane Fox, and he um quoted to me an amazing passage from Cicero where Cicero is lamenting the elites of the time and saying that they're all sitting at their behind behind their walled gardens, tending to their fish ponds, and if they keep doing this, the Republic will be destroyed. I went over to Silicon Valley with this ringing in my mind, and I said it to some of these people and said, "You guys are going to have to get involved, and you're going to have to put aside your fish ponds." Lo and behold—for those of you who can't see on camera, Robin is now clutching his head at this thought—um uh, lo and behold, some of these people have put aside their fish ponds and decided to throw their weight in. And this is a fundamental reason they that um the democ—a lot of these people, remember, most of these people voted for Obama twice and voted for Hillary—a fundamental reason why they switched is because they've come to see the Democrat Party in the last four years as a party that sees the First Amendment and free speech as a mistake to be corrected, and they've come to see that as far more dangerous than Trump. And that's part of the motivation of um what an explanation for what's just happened in America.

Maybe I'll say one last thing, and then we'll finish, which is, you know, I hope AI turns out to be one of these issues that in fact transcends the usual tendency to polarize along a right-left axis. Just to—there's an interesting case of California; it was this bill, SB 1047, for creating liability for very, very high-end, you know, compute being used for things. Two people who were for liability were Elon Musk and then the company Anthropic. Other people in Silicon Valley were on the other side and did not want that, and eventually, you know, knew some sort of—in other words, Elon was in fact on the other side from what you might—I hope this is more the future in the sense that it's a species-wide challenge, and it would be a shame if it ended up being something that, you know, was only just ended up becoming another, you know, there's the major becomes polarized like everything else; it goes like immigration or DEI, and a lot of these things. I think it would be a shame. With that, uh, maybe thank Dominic for um [Applause]