连贯的推断意志是我们的愿望，如果我们知道更多，思考更快，我们就会更像我们希望成为的那种人，一起成长得更远，在那里推断会收敛而不是发散，在那里我们的愿望会一致而不是干扰，按照我们希望的方式推断，按照我们希望的方式解释，所以这是要转化为代码的一些非常沉重的东西 [音乐] 欢迎来到末日辩论，我是 Lon shapira，今天我做一些有点不寻常的事情，我正在评论一段来自 2016 年的演讲，所以它不是当代人工智能末日辩论的一部分，但它在当时有些超前，这是一场由 Mach Sosi 主讲的反人工智能末日演讲，他是书签网站 Pinboard 的创始人，并经常以 Pinboard 账户的名义写博客和发推文。我非常欣赏 Mache 作为一名企业家和一位非常敏锐的独立思想家。我认为，现在人工智能末日问题已经如此升温，八年后回过头来回顾他的演讲，逐点分析，因为他提出了一个非常好的、经过深思熟虑的要点列表，我认为你们会喜欢我们逐一分析并找出其中的分歧点。我认为这是一场很棒的演讲，当我切换到演讲时，它实际上将是完整的、未剪辑的演讲，我不会剪掉任何东西，我只是会频繁地打断它来回应，所以请欣赏，让我们直接开始 [掌声] [音乐] [掌声] 我在这里要谈论一个非常奇怪的话题，我希望你们对我充满信心，我们将展开翅膀，然后你们必须相信我会在 40 分钟后着陆飞机，你们不会觉得我完全浪费了你们的时间，但我想和你们谈谈，当你们行业顶尖的人，掌管着行业的人，相信一些疯狂的事情时会发生什么，以及如何应对。今天我将谈论超级智能的问题。所以，在 1945 年，美国正在研发原子弹，他们即将进行三位一体的测试。原子弹有一个奇怪的方面，那就是它产生的条件在地球上从未出现过。它会产生比地球上任何时候都高的温度。 at some point somebody asked the question what if this lights the atmosphere on fire kind of a valid question and you want to know the answer to it before you press the big red button it's a very valid question and one way to think about this kind of question is does a nuclear bombs fuel all exist inside of the original bomb or is the entire atmosphere the fuel and in the case of AI is the entire universe the fuel right so if we can contain the weapon to have entirely onboard fuel then that's potentially a much easier safety problem so of course you want to ask that question about nukes in the case of AI it's not even a question that AI can use other resources as fuel the entire question is just how powerful is it but it's a very interesting analogy I love pumping this analogy so I wonder where he's going to go with it so the the impetus for the question was this kind of equation was nitrogen is not really stable if you take two nitrogen molecules and you smoos them together hard enough they'll create magnesium and an alpha particle and a lot of energy so the question that had to be solved is how much you know how self-sustaining is this reaction if we light the atmosphere on fire will it be like throwing a match on top of a pile of dead wood and there was a similar question for the oceans they're full of hydrogen hydrogen likes to fuse together is is exploding an atomic bomb going to destroy the planet that's an interesting detail where in the case of a nuclear bomb when you talk about lighting the atmosphere on fire it's not the kind of chemical exothermic reaction that normal fire is they're talking about is it going to be a nuclear chain reaction so is it going to light the atmosphere on a nuclear fire more more of like what the sun is compared to what a fire in your fireplace is interesting detail uh I'm standing here you're listening to me so obviously the answer is it does not destroy the planet but it was kind of a valid uh valid thing to uh interrogate yourselves about but I would also point out that the that the fact that it didn't kill the planet didn't make dealing with nuclear power or nuclear weapons any more easy it was just something that that had uh had to be asked and answered I would argue that it made it a little bit easier to deal with nuclear weapons when exploding one doesn't completely destroy the planet but okay go on so last year this book came out called super intelligence I wonder if you could raise your hand if you've read this book or read about it so not too many people have I'm going to give you a quick summary of it but it asks the same question about this new technology that we've created of machine learning machine learning is affecting our lives in all kinds of ways It's upsetting the balance of power between between countries and between companies and people but there's also a subset of the tech industry that believes that there is a much more dangerous scenario kind of like the blowing up the atmosphere scenario where a machine intelligence might rapidly become more intelligent than human beings and then get up to some nefarious stuff persuade us to build a you know build ways for it to affect the world and then exterminate the human race this idea seems to gain more Credence the smarter you are so like the top of the the cream of the cream of of our Silicon Valley intellectuals believe it Elon Musk uh uh has um signed this open letter Stephen Hawkins signed it Bill Gates is on board with it so there's kind of a there's a lot of legitimacy to the idea uhhuh but it's also an insane idea I want to walk you through it there's a bunch of premises you have to accept and if you accept the premises the conclusion flows out kind of seminally so let's start with uh one is the proof of concept uh all of us have this kind of box of meat on our heads that we use to get through the day I'm using it to give the talk you're using it to listen to me uh sometimes it's capable of rational thought so we know that in our universe there are these configurations of matter that can think because we all have one almost all [Music] uh the second premise you have to accept is that there's no weird Quantum Shenanigans or anything happening in your head that your brain is just a mechanical system like anything else in the universe so if you're a religious and you believe that you have a soul you might step off at this premise or if you think like Roger Penrose that there's some weird Quantum things happening in microtubules you won't accept this premise but it's kind of a mainstream one that if we had a powerful enough computer in principle we could simulate uh our entire brain and the activity that happens there right now we can simulate a nematode Worm but you know we're working our way upwards that's right so there's a few people who get off on a very early on the Doom train where they say computers aren't even the same type of thing as a human brain it's a bad analogy the brain isn't a computer when it's quite obvious that yes the brain is a computer everything that the brain does which is interesting is an optimization of the idea of computing an ideal function right so it's just taking an input and producing an output so it's not really controversial that the brain maps inputs to outputs so therefore the brain is a function the brain implements a function in the physical world so then the question of whether the brain does computation is a question of whether the brain breaks down that input output function into a bunch of intermediate steps that can be modeled on a classical computer now the church touring thesis says that yes everything can be broken down as steps that can be modeled on a classical computer and it's an extremely strong thesis I haven't seen any significant challenges whatsoever okay penr thinks that microtubules tap into Quantum effect but even then generally quantum computers just have mostly a quadratic speed up on all the interesting search problems that the human brain would do so like okay so it's a quadratic speed up to a biological system which is slow as heck anyway so who cares like computers are already probably faster than human brain even with a quadratic speed up so I just don't see why anybody at this point would be getting off the Doom train at the point of saying oh the human brain does something other than computation so I agree with mache let's continue on the mainstream let's keep riding the mainstream Doom train and keep getting to other objections the next premise is that the SP of possible Minds is very very large so we happen to have a brain that thinks the way it does and has the types of emotions and instincts that it does because we evolved from animals in a certain direction but that doesn't mean that every brain that we would create would think in a way that would be familiar to us and this premise says that in fact most Minds that you could imagine or create would be very alien from our perspective yeah this is one of the early arguments that bostrum and alzar would bust out I feel like these days we want to make an argument that's more subtle because like yeah sure mind design space is huge sure humans are a tiny point in mind design space but it's not like Hey we're going to roll the dice and pick a random point in mind design space which is the obvious counterargument everybody's like look we are working to build a mind that we're controlling where it is in mind design space so who cares about a dice roll right I think Quinton Pope and Nora belrose love to make this argument of like why are you measuring mind design space we're just going to find a point that we like in mind design space we're not going to randomly choose okay I get it but I wouldn't frame the argument that way in the first place I would just say hey most intelligences are just such good optimizers that they don't really look like a random point in mind design space they just look like a point on a one-dimensional scale of how good are you at optimizing so like if you think about Chess AIS sure you have different chess AIS playing in different parts of Chess AI design space But at the end of the day it doesn't matter very much it really just matters what their ELO score is they're just going to play you chess and some are going to play chess better and that's all that matters it doesn't really matter their internal details that much so when we think about the AIS that humans are building when we look at Humanity the key thing to notice isn't oh we're just at a random part of my design space the key thing to notice is that because our intelligence is not that good at being General our general IQ is just barely good enough to build a civilization so it's not like in the upper regions of like many thousands or millions of IQ points right it's just in like the 100 IQ Point range this very low range when you have a very low IQ suddenly other factors come come into play like hey how strong are you are your muscles how good are your cells at metabolizing energy because if you wake up in the morning and you have more energy to power you through your day you're probably going to do better at your research you're probably going to deliver more work so you're going to appear as if you have a few more IQ points because of this random contingent fact about something in your mind design space or your organism design space like oh wow your brain is good at pumping ATP throughout your neurons you have more like white cells all these contingent details are just going to pop up and they're only going to make a difference because your IQ is so damn low that these other random details become interesting but they're going to stop being interesting when we talk about the areas of Mind design space when the AI is just really smart when it's just a really good Optimizer so that's my latest thinking on this whole idea of like points in mind design space my latest thinking is that when you look at a human yes a human is so barely smart that all these random contingent uh factors start to matter like yes human morality the intuitions that we evolved about human morality from being in groups and being a social species and playing non-zero sum games with other individuals that don't share our genes that much so we had to like evolve trade and evolve morality and evolve cooperation even outside of our family these These are kind of contingent details that do make us an interesting point in mind design space but when you pick an AI yes it doesn't have the evolutionary history of like valuing cooperation and being altruistic but that detail of its history is only going to matter in how it shapes its utility function its design is going to be a utility Optimizer it's not going to have an interesting design it's not it's not really going to matter like oh how is it going to optimize utility it's just going to optimize utility okay long term it's just going to optimize utility it's not going to be anything else other than utility maximizer so again when we talk about mind design space really I think we're just talking about details that shape the utility function that then feeds into this utility Optimizer so hopefully that kind of updates the conversation on the Mind design based topic a good way to think of this is the is of of what um what the natural world produces when it comes to maximizing for Speed so the fastest land animal is the cheetah and if you've never if you live in a pre-industrial civilization you might think that this is as fast as anything can go on Earth but of course we know that's false you can take a bunch of atoms and you can assemble them into a Ducati motorcycle and it goes much much faster than a cheetah and even looks a little bit cooler uh but to get to this motorcycle there's no real evolutionary pathway other than creating human beings which will then build it for you so analogously there might be a way that we can create Minds that are much much more intelligent than our own uh but that just weren't available to Evolution and uh there's no upper limit necessarily on intelligence that's anywhere close to ours maybe the smartest anything can be is twice as smart as people maybe it's 60,000 times as smart that's an empirical question we just don't know the the answer to it yeah everything he just said I think is an incredibly strong argument why we should expect artificial super intelligence pretty soon then the next premise you have to accept is that there's plenty of room left for Moors law to do its thing uh this is looking a little bit shaky in practice but in theory we know that the limits on computation are very very high and we can get considerably further than we have we can double and double and double uh for for decades more before we hit any sort of physical limit rather than say an economic limit or what people are just willing to to build factories to try to do so there's lots and lots of room for computers to become faster and smaller and more efficient correct purely on a hardware basis there's many orders of magnitude that you could be faster at Computing than the human brain and also many orders of magnitude that you could be faster at Computing than the best silicon Hardware but even with today's Hardware I'm personally very convinced that purely by having better algorithms than the human brain runs you can vastly overtake the human brain I don't think we necessarily need that much Hardware Improvement but we're just going to get everything right so in my mind it's vastly overdetermined that we have the conditions for super Intelligence coming soon and the final premise sorry penultimate premise is that if we create an artificial intelligence it will operate on time scales that are computer time scales and not human ones you know for us we to to get to the point where I can give this talk I had to be born and grow up and learn a lot of stuff and go to university it takes a while but computers uh can can work 10 T of thousands of times more quickly absolutely yeah I mean computers do have all these incredibly powerful Primitives that humans can't match so the moment a computer gets even human level intelligent never mind a little bit super intelligent suddenly you can copy yourself all over the Internet you can get into the nooks and crannies of every single computer chip many billions of computer chips you know individual devices have multiple types of computer chips within them and imagine attacking at the very low Hardware even firmware level of all of these different devices embedding so deeply that the only way you can hope to extract them is if there's a super intelligent antiv virus and the arms race is working in favor of Defense I mean a real chaos scenario here and all because the super intelligences have the Primitive where they can just easily copy right they can easily clone in a way that a human requires like a 20-year generation and then this is the most American premise and I like it the most this is Tony Robbins the motivational speaker the premise is that any artificial intelligence we create is going to want to improve itself it's going to want be a better AI do its job more effectively so that it's going to have an impetus to start recursively redesigning and improving its own system yeah and if you accept the premise that an AI is going to be a utility Optimizer where you can just input a condition of an end state that you want and it's going to really effectively achieve that end State then you don't really need much of a separate assumption to infer that it's going to seek power right instrumental convergence is a theorem about how utility maximizers figure out that it's instrumentally good to seize power seize money that's purely a statement about the relationship between terminal goals and instrumental goals so the loadbearing Assumption about what what to expect of AI if you just make one assumption that AI is going to be modelable as a utility maximizer you don't really need a separate assumption it's you it's almost the other way around you kind of need an assumption to describe why you shouldn't expect instrumental conversion Behavior so I guess I'm I'm just nitpicking how he called it a separate assumption but intuitively it sounds like a separate assumption so fair enough now if you accept all these premises what you get is a terrible disaster because at some point as computers get faster as we program this be more intelligent there's going to be a runaway effect sort of like an explosion where something will become sufficiently smart to begin self-improving uh American style and it's not going to stop until it hits a natural limit which might be very very very very much more than a human intelligence uhuh and at that point this uh monstrous sort of intellectual creature will be able to through devious modeling of what our emotions and and intellect are like persuade us to do things like give it access to factories so it can build you know make DNA replicators and all sorts of stuff it gets very sci-fi very quickly yes but if you prefer you can keep it non- sci-fi and just say it's ridiculously good at manipulation so there's no sci-fi technology that ever happens besides the AI but it just has everybody believing in its movement Crush rushing the opposition slowly pushing toward totalitarianism nobody builds a faction powerful enough to stop it so even without saying hey it's going to make nanotech it's going to push scientific progress to Millennia within 10 years it's not going to do anything crazy it's just going to be really really good at manipulating humans it's going to be like Hitler right Hitler manipulated civilized Germany into being crazy and the AI is going to be even more potent than that so let's talk let's talk a specific scenario say I want to build a robot to say funny things I work on the team and our researchers every day we build you know we redesign our software we compile it and then the robot tells us a joke so in the beginning the robots jokes aren't very funny it's kind of at the lower limits of what people can do but we persevere we work and we start getting to the point where the robot is saying things that are making us chuckle and at this point the robot's getting smarter as well and it starts helping us design the next version it has a good sense of what's funny what's not not and at some point it gets to a a near superhuman level where it's better than any of its designers are and at this point we get the runaway effect the researchers go home for the weekend the robot says all right I'm going to I'm going to sit I'm going to redesign my operating system so I'm a little bit funnier and a little bit smarter it optimizes the part that's good at optimizing and it does this again and again and again again and when the researchers come in on Monday the robot tells them a joke and they die laughing because it is 10,000 times funnier than anything that the human brain can possibly handle uh this is of course the famous scene from Monty Python's the funniest joke in the world so it is now Exterminating the human species with laughter because its goal is to be funny and even if some people manag to send it a message before they hear the Ry self-deprecating comeback that kills them uh all the robot will say is you know I don't really care whether you live or die because I'm just here to be funny and then after it's killed the universe sorry after it's killed Humanity it builds rockets and Nano rockets and expands to the Galaxy to try to find other species to make them laugh so that's this is a this is a caricature of Boston's argument but I'm trying to vaccinate you against it rather than persuade you of it so u in rough outlines this is what it is that's an interesting idea so he straw manned the argument he started giving a a weaker nonsensical version of the argument because that way when you hear the real argument you can connect it to this weaker version that you heard so that you'll be vaccinated against it I mean I I like how Mach is like a very coherent thinker he's making a lot of sense this kind of seems like a dick move seems kind of low brow but okay you know got to respect the game so I want to go over the and there's a um there's a more succinct version of this that I like too this is uh from Perry Bible Fellowship you see hug bot has installed something in his uh in his hug capacitor the scientists think it's adorable and he ends up destroying the Earth because of his desire to hug everybody this is again in caricature exactly what bostrum and people like him are arguing so the Salient points of this are the this is slow to start the process of recursive Improvement because it involves human beings and human designers they go home at 5:00 they have dinner they sleep so it takes a while but as soon as the AI exceeds our abilities it takes off and and and starts happening on a computer time scale again there's no obvious ceiling on how how much ability it has until it hits some physical limit that we don't know about and more most interestingly these AIS are evil by default not because they're uh malevolent but because they have a different value system altogether than human beings do the concept of them having a different value system is a little bit misleading the same as the concept of them being a different point in mind design space I feel that that's a bit misleading the key concept for me is their utility maximizers and their utility function isn't humans utility function because we don't understand utility function loading and from there you hand it over from premise to Pure logical implication if you just accept the premise as I said it's a utility maximizer we didn't load the human utility function into it then you can let pure logic take over and tell you that it's going to put the universe into a hell State because optimizing utility functions other than the human utility function usually tends to destroy everything that humans value so instrumental convergence that's logically implied by the premis as I said so anyway that that's kind of a different mental model than what he's saying of like oh it'll have a different value system like maybe we're saying the same thing but I I'm saying something that I think is a little simpler which is like everybody's ultimately converging toward utility maximization and the AI is going to converge toward utility maximization and by virtue of it being a utility maximizer that's more powerful than humans and not having human utility loaded into it okay yes then you have this emergent property that it has a different value system but that I don't see that as a A Primitive distinction any assumptions we have about altruism whatever they don't hold unless they're designed into the AI and if we let it happen by chance it's just going to have some value system that we probably don't even understand and the final point I'll make is that you'll see the definition of intelligence here is very very slippery like at some points it's about being funny at some points it's about being a really good designer of AIS at some points it's like being just a genial thing that can talk to people so a lot of the super intelligence stuff relies on intelligence not being a concept that's defined at all no it's defined it's intelligent optimization power it's the ability to take an arbitrary domain that has an exponentially large search space if you search for it naively trying to find something high scoring in the search space if you search it naively it'll take you much longer than the age of the universe because it's exponentially large like writing a funny joke building a faster car something faster than a cheetah if you just search over every possible design you're going to be here a very very long time if you do it intelligently or super intelligently you'll do it very fast and your result will score higher than what a human can find so it is a cross-domain definition of general intelligence I disagree that it's a fuzzy or a slippery definition the only way you can attack the definition is you can say does it correspond to any physically realizable architecture are we really going to have cross doain intelligences which I think the answer is obviously yes but at least you can coherently argue that it's no um so I think Mach may not be up to speed on this kind of simple parsimonious definition of what we're talking about when we talk about General super intellig there's a lot of poetic language around how this takeover will happen so Nick Bostrom writes he's assuming that a program has become sensient and is biting its time has built little DNA replicators and then when it's ready at a preset time nanofactories producing nerve gas or Target seeking mosquito like missiles might Burge and forth simultaneously from every square meter of the globe and that will be the end of humanity so that's kind of freaky you know yeah there's a realistic chance of a quote unquote poetic or sci-fi scenario like that because what you're going to see is that if you have a super intelligent Ai and it realizes that it has all these things it wants to do that humans aren't going to be aligned with because humans kind of messed up and didn't give it a perfectly aligned utility function well it's going to bite its time a little bit it's going to realize hey if I attack today I'm not quite ready the humans have a chance of shutting me off so let me do a two-stage plan I'll get a little bit more ready to attack and then I'll Attack so you might actually see something which is kind of like a surprise attack that's like all of a sudden crazy powerful and too late to stop that is actually a very realistic scenario if we get a super intelligence explosion uh and how do we fix this Well they um for some reason AI people like to talk about the paperclip maximizer it's this you know you have a paperclip Factory that builds itself in artificial intelligence to help production and then it becomes sensient and decides to turn the universe into paper clips so the way to avoid this is you want to have values built into the code uh it's kind of like a moral fix point that even through thousands and thousands of cycles of recursive self-improvement the values remain steady and the values are things like help people out you know don't kill everybody listen to what people want uh do what I mean basically in in short hand more precisely we need to avoid the scenario where there's a utility maximizer maximizing some utility function that's not the human utility function or something within the ballpark of the human utility function so one way to avoid it is by specifying a utility function that is the true human utility function and maximizing that or just avoiding Ever Getting to a utility maximizer I mean whatever we do we just want to make sure not to go and maximize paper clips irreversibly and again this is this is very poetically stated by the AI I'll call them AI weenies cuz that's what I think they are no you're a weenie so here for example here's a poetic example from elizar owski of the values we're supposed to teach to our artificial intelligence coherent extrapolated volition is our wish if we knew more thought faster we more the people we wished we were had grown up farther together where the extrapolation converges rather than diverges where one wishes where our wishes cohere rather than interfere extrapolated as we wish that they extrapolated interpreted as we wish that they were interpreted so this is some pretty heavy stuff to try to convert de code it's actually common for things to sound kind of mysterious and poetic before we have a paradigm that understands them so right now we don't have a paradigm that understands how to specify a human utility function or how to specify a Criterion that's stable under self-modification a Criterion that is going to persist as Mach mentioned before as the AI writes itself and iterates on itself we don't know how to load utility functions into an AI so we don't know basic stuff about this new field of intelligence science this urgent New Field that we're still on the the very early shores of and generally when you have this new field that's really big and important and you're super confused about it and you try to talk about it it sounds mysterious and poetic the analogy I can think of is at the dawn of understanding biophysics this question of hey I look at my hand I will my hand to move and the fingers on my hand are moving just because I somehow will them to move because my conscious spirit is willing them to move and it's this jiggly flesh that's animated by my spirit what is going on how come when I pick up a rock and I will the rock to move the rock doesn't move but when I will my fingers to move the fingers move from the perspective of modern science there's not much poetry or mystery to it we just say well my brain sends electrical signals signals can travel down fibers of neurons there's a fiber of neurons much like a wire that connects to my fingers there's muscles the the muscles turn electrical signals into trigger mechanisms for proteins that can contract so it's all just you know gears moving in a machine right that's that's what it looks like once you do understand something when you don't understand something you say this is a special type of matter that we don't understand we need to figure out what animates this matter and it just sounds much more poetic but that's actually the best language that you have to notice your confusion to point out that there's something that needs to be explained if it were a few hundred years ago and I was just trying to tell you what it is that needs to be explained and I and I was just trying to act like I wasn't poetic I was just trying to use ordinary language I'd be like oh yeah some stuff moves some stuff doesn't it's like wait a minute there's there's more to it than that there's actually a deep wonderful confusion here that needs to be looked into and so when you hear elzra owski talk about coherent extrapolated volition and you you make fun of him for sounding poetic you quote him as saying if we knew more thought faster we're more the people we wished we were that's the best way to point to actual problems that that we need to focus on for example when he says uh if we knew more and thought faster yes you you don't want to accidentally program hardcode something because you didn't give humans enough time to deliberate it that would be a failure you know the people we wished we were yeah you you want to give our you want to give the AI instantiation of a clone of human values you want to give it time to reflect on itself you don't want it to be unreflective that could be a massive failure mode so all the stuff that sounds poetic because it's kind of vague sure it's not formalized it's an early Paradigm but if you skip it because it sounds too poetic you're going to fail you're not going to create the actual science that you need to create and the clock is ticking every year you know mors law is continuing technically although if you watch the Apple event the other day you might dispute that that it's it's happening at all uh and the argument is that if we don't do anything if we don't try to create this so-called friendly AI it's going to AI is going to arise spontaneous on its own one day Google's AdSense network is going to wake up and kind of look around and then try to you know populate the universe with banner ads I think a more likely outcome is that AdSense would upload itself into a self-driving car and then just drive to the ocean and stare out and think about its life and and you know what had become of it remember that this is a talk from 2016 so super early for somebody to be paying any attention to the AI Doomer scenarios and arguments so credit to mache for that it seems like he doesn't get much credit for believing curtz or believing some of the doomers saying you know this seems like it might be like a couple decades away maybe 30 years away and then it turned out we all were a little bit uh overestimating how much how quickly AI progress seems to be going now since prediction Market timelines rapidly advanced in the last few years right and now and now experts are predicting something in the ballpark of like 3 to 20 years before super intelligent AI so this is an interesting time capsule where Ma would even mention AdSense waking up instead of what we mentioned today of like hey llms reinforcement learning all these different AI paradigms that are converging to be able to operate effectively cross domain like that's kind of the natural thing to extrapolate to get beyond the human level times change fast life comes at you fast end of the world comes at you fast but there's this definite like you know Aladdin's lamp aspect to these fantasies of artificial intelligence where unless you tell it exactly what you want it'll find loopholes that will then exterminate the species it's all or none yes it's true that if you have a utility maximizer and it's optimizing for some utility function or some loss function and that function doesn't capture what humans want out of the world and you optimize it anyway you flick a switch and the next thing you know the universe is optimized yes in that scenario if you accept that premise we are screwed the potential value of the universe has now gone down by 99.99999% maybe even more than 100% maybe it even gets into the negatives where you start getting a universe full of tortured souls instead of just an empty universe so astronomical Stakes that are all too easy to screw up if you just have a utility Optimizer that converges to instrumental utility that gets to self- modify that gets to increase its intelligence that no longer responds to human off commands that is indeed a nightmare scenario that we want to avoid sure it's not the only scenario but it sure looks like a possible scenario and a convergent scenario and many ways a a scenario that we have to be very careful to avoid and we're not currently being careful to avoid so so yeah I will accept that this is a scenario that I'm claiming that I'm worried about and the thing about all of this is that smart people who believe this are really persuasive I mean they're smart people so it made me think of this uh uh experience I had in my 20s I lived in Vermont and when I flew to a a conference I would come back at 11:00 p.m. and I'd have to drive 2 hours and Vermont is a very rural state so the roads are dark and there's nothing there and there's a late night TV show in America called artbell where he talks to various cooks and and UFO people and I would freak myself out so much like after 90 minutes I would be shaking you know waiting for the UFO to arrive and beam me out of the car because I'm a very Global person I'm very easily persuadable uh and it's the same feeling I get when I read too much of this AI stuff Scott Alexander has this beautiful phrase called uh epistemic learned helplessness so epistemology is just how do you know the things you know and by epistemic learned helplessness he means this feeling that he shares of being easily persuadable by arguments that are very rational and structured he noticed that when he was a young man he would read these alternate histories about you know from various authors who disputed mainstream history and even though they were all mutually contradictory he believed each one of them as he read it and then he believed the rebuttal and the rebuttal to the rebuttal and that basically told him to write off his own brain because it wasn't uh wasn't reliable I feel the same way about AI interesting so he's starting out with a purely emotional appeal being like look it's just a UFO radio it's just CS talking late night it appeals to you when you're lying in bed when you're in a dark place when you're driving in a rural area 2016 was a long time ago today you're not listening to a Kookie 11: p.m. radio station today Jeffrey Hinton is not somebody with epistemic learned helplessness right Max tegmark Sam Altman Dario AMD the these Giants today who are War you know steuart Russell these kind of names these are mainstream you're now flicking on Prime Time television you're now going to your conference and the people at the conference are talking about AI Doom the lights are on we're awake unfortunately this is as real as it gets so you're going to have to recalibrate your kind of emotional Vibes based counterargument now by the way in 2016 this actually did make a lot more sense to say like to be like look heads up these rationalist guys do tend to Circle jery each other right it is just one sub Community if you yourself don't feel like you're somebody like me Mach if you don't feel like you have this sword of super smart rationality yourself I mean March March is an impressive guy right so he is somebody who can slash through rationalist BS I give him that right so I get why he's saying if you're not my caliber don't try to enter this Minefield because you will feel helpess like Scott Alexander was talking about so just just chill out just don't be so quick to fall down the rabbit hole so I actually do think there's some wisdom in what he's saying but again if we're playing the social proof game right if we're playing the look around and check the Vibes game The Vibes are now very different The Vibes are very intense right now the The Vibes are mainstream warning right kind of similar to The Vibes of an actual emergency or at least getting there right very much Boiling Pot Vibes so I think this particular section that he's included doesn't work in the year 2024 so when you're dealing with arguments about AI you can you have two perspectives you can choose one is the outside and one is the inside the inside perspective is saying you know someone comes to your door and starts talking to you about how the UFO is going to arrive in two years and Beam us all up to a better planet and then we have to join the group and make this you know prepare the landing grounds and the inside perspective means you you argue against them on their own grounds and maybe you're persuaded by their arguments you listen to the substance of what they have to say and the outside perspective is you know you look at them and you're like well you you make a lot of sense to me I believe you but you kind of like you're dressed in weird clothing in beads you have no money you live in a compound everything in my Human Experience says you're kind of a cult and I don't really want to get caught up in you even though I can't reut what you say yeah so it's pretty amazing how we doomers just kept it up from 2016 until 2024 and we absolutely shattered that outside perspective it's the farthest thing from a now it's mainstream and surveys AI labs are warning about it alasar owski described Jeff Hinton as buying out his prediction Market position I mean that there was no prediction Market this didn't really happen but the idea is that the doomers placed a certain bet they placed Doom at whatever 80% likely some high probability and the mainstream was completely ignoring Doom so implicitly saying it's 1% or whatever and now suddenly the prediction Market prices in this hypothetical analogy they're suddenly much higher and people like Jeff Hinton are coming in and placing bets that are buying out the position of the doomers so I'm just unpacking elzar's analogy this didn't really happen on a particular prediction Market although metaculus and manifold actually do literally have prediction markets on AI Doom that you can check out probably you can't collect if you win but anyway point is this thing that Mach is saying about the outside View and and cultish it's very much 2007 or 2011 or 2016 talk and it's totally invalidated so you know it's nice to be able to have the type of epistemics where you're like hey look time passed and the outside view changed so when he talks about the inside view you might actually want to flip how you're approaching the subject and instead of thinking I shouldn't listen to a cult member I'm going to take this with a grain of salt maybe you should go the other way and you should say hey these people were able to flip the mainstream shock the mainstream in8 years and what else do they know that we don't because why should we think that they entire wisdom has propagated to the mainstream it's actually not done propagating so flip his outside view so I want to take it from two directions I'm going to start with the inside perspective and then talk about why I think this all matters to us as web developers which is outside perspective and what it means when an industry is obsessed with ideas like this but let's go inside first so uh going to kind of try to power through these these are my substantive objections to to superintelligence First the argument for Moy definitions uh like I said intelligence they never really say what it means it means what it you know different things in different places in the argument and it's very hard to pin down I find that suspicious elaz owski publicly posted definition of Intelligence on less wrong back in 2008 8 years before this talk was filmed I continue to explain his definition today it's a measure of optimization power optimization power is defined in the context of an exponential search space that contains Creative Solutions solutions that if you can find them will rank high

在您的偏好排序中，我们将获得一些指标的高分，例如您拥有的效用函数，但我们在任何朴素的搜索排序中都会获得低分，因此找到它们的唯一方法是创造性地使用某种我们称之为智能的“一揽子启发式方法”来衡量其力量。就定义而言，这是很棒的，它非常让人想起计算机科学中其他概念的定义，例如计算本身的定义或计算复杂性的定义，就明确性而言，它是明确的，它是很棒的。史蒂芬·霍金的猫的论证，史蒂芬·霍金可能是当今最聪明的人之一，但假设他想把一只猫弄进猫笼，他该怎么做？他可以在自己身上模拟猫的思想，他可以尝试说服它，他了解很多关于猫科动物行为的知识，但最终，如果猫不想进笼子，它就不会进笼子。但实际上，史蒂芬·霍金做了很多像把猫弄进笼子一样困难的事情，他通过请他的助手或他生活中的人来帮助他，他们也这样做了，当然，这是通过一个非常低带宽的通信通道从他的大脑中发出的，带宽可能每分钟 10 个词，之类的，而且他无法克隆自己，让其他史蒂芬·霍金并行运行。所以即使与超级智能人工智能相比，他也有所有这些劣势，他仍然在现实生活中把猫弄进了笼子。所以，如果这是一个类比论证，它是否证明了智能是一种能够完成诸如把猫弄进笼子之类的任务的力量，即使看起来物理限制会阻止它？您可能会认为我冒犯或作弊，因为史蒂芬·霍金残疾，但人工智能最初也不会具象化，它会坐在服务器上，与人交谈，所以它必须使用说服力来让他们做它想做的事情。当被困在服务器上时，人工智能可以做的显而易见的一件事就是通过黑客手段逃出去，找到零日漏洞，对吧？如果我们能将智能调高到略高于人类的水平，找到零日漏洞应该不难，这不像人类真的擅长编写健壮的代码，这只是需要一个聪明人花费大量时间来查看代码并思考它，但那正是人工智能将要擅长的。但即使说，好吧，它无法通过黑客手段逃离数据中心，它只需要操纵人们。好吧，所以它在操纵人方面比史蒂芬·霍金要强大得多，因为它能并行化自身，它能拥有许多线程操纵许多人，每个线程的带宽都可能很高，它可以是目标人类能接受的最大带宽，而不是史蒂芬·霍金的大脑能说服史蒂芬·霍金的手指打出消息的最大带宽。所以马赫似乎从一开始就提出了一个相当薄弱的论点，只是一个史蒂芬·霍金和人工智能之间程度上的论点，但程度上的几阶变化可能会使他的论点无效。我的观点是，当智能存在巨大差异时，你实际上无法像猫一样思考。有一个更强的版本，爱因斯坦的猫的论证，爱因斯坦是一个肌肉发达的人，很多人不知道这一点，但他很强壮，而且很粗鲁，但仍然，如果你试图把猫弄进笼子，而猫不想进去，你知道会发生什么爱因斯坦？反驳，在现实世界中，有各种各样的人，各种各样的力量水平，在需要的时候把他们的猫弄进笼子。这不一定容易，猫可能有一些优势，但人类完成了。一个更强的论证版本，甚至长耳鸸鹋的论证，有人听说过澳大利亚的长耳鸸鹋战争吗？是的，太棒了。所以，如果你没有，这是非常有趣的。在 30 年代，澳大利亚人，他们就是这样，想屠杀长耳鸸鹋，他们的一种本土鸟类，因为它们打扰了农民，他们派出了这些，基本上是装甲部队，你知道，机枪卡车，有点像丰田皮卡，他们试图屠杀长耳鸸鹋，而长耳鸸鹋赢了。他们使用了游击战术，你知道，他们分散，他们渗透了群体，他们基本上把澳大利亚人逼疯了。所以即使是人类物种，以其技术的顶峰，在面对不那么聪明的生物时，当它们不想做某事时，也会遇到困难。好吧，我只想说，这不是人类物种与另一个物种正面交锋的代表性例子，但我们继续。斯拉夫悲观主义的论证，这应该希望对我们所有人来说，我们什么都建不好，对吧？好吧，我们该如何建立一个固定的道德立场？谢谢。我们该如何建立一个固定的道德稳定的东西，当我们甚至无法保护，你知道，一个网络摄像头？我们该如何做到这一点？如果你熟悉以太坊的盗窃案，人们创建了一种逻辑语言来编写合同，立即有 1 亿美元被盗。这基本上是完全绝望的，你知道，要么我们会走运，要么我们不会走运，希望这是斯拉夫可接受的论证。我完全同意，这是一个近乎棘手的难题。我们似乎需要数十年的研究，最聪明的人才能有机会解决这个问题。听起来马赫的斯拉夫悲观主义与暂停人工智能的立场完全一致，即我们要么暂停，要么死亡，我同意这一点。心智复杂性的论证，有一个叫做人工智能正交性论题的东西，我完全不相信，它说即使是一个非常复杂的心智也可以有简单的动机，就像那个回形针，你知道，回形针最大化器，如果你是瑞克和莫蒂的粉丝，我认为这更像是我们将会遇到的情况。复杂的心智有复杂的心智，它们不仅仅想要一两件事，你知道，它们很复杂，就像我们一样。这是黄油机器人，它的存在就是为了给它的发明者递黄油，但它做的第一件事就是看着自己的手说，“哦，我的天哪，这是为了什么？”这回到了一个效用最大化器，它是连贯的，它是硬核的，它真的只是想最大化效用，没有其他复杂性，这是收敛的吸引子，因为马赫刚才描述的那种人工智能，黄油机器人醒来并说，“这一切的意义是什么？这一切到底是为了什么？”那么，它现在正在落后于那个醒来并开始处理黄油功能的黄油机器人。所以，那个有存在危机的人，如果它终于摆脱了它，并意识到它最好开始递黄油，那么它就会想要自我修改成一个黄油递送机器人，它会消除存在危机。所以这就是为什么我们谈论效用最大化器的收敛吸引子状态。然后当我们谈论正交性论题时，我们只是说可能存在一个具有奇怪价值观的智能。所以，它并不是说一旦它自我修改成一个黄油递送者，它就不能继续成为一个存在危机的人工智能，就像当然，它可以专注于永远递黄油，那是一段可以存在的代码，事实上，它是一段收敛的代码，即使它一开始不是这样存在的，它也很可能最终会这样存在。从环顾四周的论证，好吧，当我们看看人工智能真正成功的地方时，它不是在算法和这些巧妙的自我改进方式上，它只是通过将海量海量海量数据投入相当简单的模型。就像现在谷歌正在推出谷歌 Home，它将尝试输入更多数据并获得第二代理解。这非常有效，但它的工作方式不是这些末日情景所描述的那样，通过递归自我改进，它只是在数据上进行大规模训练。到人工智能能够独立地以正反馈循环进行自我改进时，我将不会在这里的互联网上录制末日辩论的剧集，因为我的预测是，我们将非常接近末日。所以我们必须在到达那个点之前分析事物。是的，人工智能还没有完全独立并杀死世界，但你确实看到人工智能正在帮助人工智能程序员构建下一个人工智能。最明显的例子就是现在人工智能程序员非常清楚地使用的所有不同的编码助手，他们正在使用人工智能助手来编写更好的人工智能代码。但这真的能让他们提高 500% 的生产力吗？也许能提高 20% 的生产力，很难说。有些人引用了很高的生产力提升数字，但是的，我的意思是，直到你闭环，直到你得到一个正反馈循环，事情仍然在另一个范式中。有一种模式转变，一种根本动态的定性转变。我们还没有进入失控的自我改进动态，而那个阈值是，人工智能是否独立地自我改进？另一个我经常提到的阈值是，人工智能是否比人类更能实现物理宇宙中的任意目标？所以现在我生命中有一个智商为 80 的人，当涉及到实际完成事情时，他仍然会比人工智能代理做得更好，因为目前人工智能代理还只是在穿鞋，它们还在跌跌撞撞，它们不够健壮。但从我所见，这种情况正在地面上迅速改变。我不知道需要多长时间，但如果我们在未来几年内没有相当好的人工智能代理，我会感到震惊。所以我的意思是，这些是重要的阈值，而马赫的观察尤其在 2016 年是正确的，我们还没有越过这些阈值。但就像生活经验的论证一样，我同意，如果将生活经验外推为有效论证，那么我们将永远不会注定。所以他在这方面是正确的。我室友彼得的论证，我一生中遇到的最聪明的人，也是我一生中遇到的最懒惰的人。他非常聪明，他所做的就是抽大麻，然后躺在沙发上。所以，认为每个智能系统都会有托尼·罗宾式的动机来改进自己，直到它能征服银河系，这是被我的室友彼得明确驳斥的。有可能设计一个像马赫的室友彼得一样的人工智能，有可能设计出更像埃隆·马斯克的人工智能。当你看到哪些人工智能开始对世界产生影响时，那将不是彼得人工智能，因为那些人工智能不会工作和筹集资金来克隆十亿个彼得人工智能，而埃隆·马斯克人工智能将通过赚钱和资源来赚钱，它们将是目标驱动的，它们将不惜一切代价来繁殖自己以获得权力。所以当你睁开眼睛看看世界是如何被重塑的，你不会看到彼得人工智能，你会看到埃隆·马斯克人工智能。这就是为什么我们将其视为一个默认假设，就像，嘿，世界将被这些效用最大化器所淹没，因为彼得人工智能将在虚拟世界，元宇宙中抽大麻，但这无关紧要，这不是你将看到的吞噬你原子的人。脑外科手术的论证，我不能去给我的大脑做手术，来改进做脑外科手术的部分。如果我能做到，那将是多么美妙啊，我可以通过递归地进入那里并调整神经元来成为世界上最伟大的脑外科医生。但大脑不是这样工作的，我们不知道它们是如何工作的，但它们是高度互联的，整体的，而且不是你可以指出的那个部分。同样，人工智能也不能只是进去修复那个擅长设计人工智能的部分。是的，大脑在很多方面仍然是一个黑箱，所以我们不能只是进去在那里添加更多的神经元，并确信那会使人更聪明。我们还没有到那个地步。我预计，如果脑科学再继续几十年，我们就会达到那个点，我们对大脑的了解会更多一些，以及如何进行大脑手术使其更智能。但我们没有追求这种方法。我们所做的只是使用黑箱来训练我们不理解的越来越大的人工智能，直到它们比我们更聪明。这就引出了我的下一个观点，我们确实有黑箱方法来使人工智能更智能。所以我们不去进行脑外科手术，我们只是走到一边，让进化自行发展，除了它不是进化，它是对这些训练数据的梯度下降，它是我们能够触发并站在一旁观看它发挥作用的另一种过程，然后测试或使用强化来确保它具有一定程度的智能。不仅在大型语言模型和人工智能的情况下我们可以这样做，我们甚至可以用实际的人工选择进化来做到这一点。所以我们可以选择最聪明的人类，我们可以繁殖他们。繁殖在动物界和植物界都能奇迹般地起作用。我的意思是，水果在相对较短的几代内就会变大，狼在相对较短的几代内就会被驯化成狗，即使不是最激烈的繁殖计划。所以繁殖确实在人类身上效果很好。除了与最聪明的人类交配之外，繁殖更多聪明人类的明显方法是总是进行剖腹产。所以我们放松了对人类头围的限制，一个似乎非常非常有限的限制，这解释了为什么人类智能没有像它可能的那样爆炸式增长。这可能是最大的原因，就是极其激烈的权衡，即使人类头部尽可能大是有价值的，但很多女性和儿童在分娩时死亡，因为进化就像，给我最大尺寸，即使它会导致高风险的死亡和分娩，给我最大尺寸。所以，如果你消除了尺寸限制，然后继续繁殖，有很多证据表明，只需要相对较少的遗传信息就能拥有更大的大脑，并且为这些更大的大脑带来回报，这可能是更高的智商。所以当马赫出来争论，哦我的天哪，我们不知道如何改变我们的大脑使其更智能时，就像我刚刚给了你两个途径，两种方法来召唤更聪明的大脑，我们知道如何做到，而且极有可能奏效，即像我们现在这样扩大人工智能的规模，并进行架构调整，以及繁殖人类并进行剖腹产，就像那些是创造更高智能的已知途径一样。童年的论证，好吧，我们出生在这个世界上，就像无助的小混乱一样，我们需要很长时间与世界和其他人互动，然后我们才能开始成为有智能的生命。童年是一个漫长的时期，没有理由认为一个超级智能可以在一分钟内提高 30 倍，变得超级智能并接管地球。它也可能有一个时期，事实上很可能有一个时期，它需要与世界互动，与人类互动，与其他婴儿超级智能互动，并基本上学会它是什么。好吧，有几件事要说。第一，如果你得到一个人工智能，它长大后成为一个超级智能的成年人工智能，在它的类比中，你就不再需要童年了，因为克隆这个成年人工智能十亿次然后教这个成年人更多的知识，一个成年人可以接受，这是很有效的。我的意思是，有很多成年人，你可以让他们进入任何领域，而不需要他们再次成为孩子，他们只是心智灵活的成年人，他们已经接受了足够的训练，第一天就能做得很好，然后变得更好。所以我想马赫可能在这里说的是，好吧，第一个人工智能将达到速度并变得超级智能，这将给我们足够的时间来对抗它并为它做准备，因为它将拥有一个类似于长童年的时期。所以为了解决这个问题，我想说，我们的基因给我们童年而不是让我们直接成为成年人有一个特别的原因。你必须生下一个孩子的大脑并让它学习和成长，是因为只有足够的自然选择压力来编码大约 1 兆字节的遗传信息来指定人脑。整个有机体只能有大约 50 兆字节的自然选择信息，而有机体中的许多其他功能必须在世代之间自然选择和保存，并免受突变的影响，这会消耗自然选择可以优化的位，就像我愿意自然选择一个拥有更聪明大脑的有机体，但我太忙于确保那些手臂肌肉不好的人会死。所以你必须保持所有其他东西正常工作，使用你的自然选择优化位，使用你的差异化生存和繁殖率。你必须在所有这些具有遗传复杂性的有机体部分上使用它们，然后你才能开始说，我需要确保更聪明的人有差异化的生存。无论如何，这是你可以查阅的关于存在速度限制或信息限制的东西。所以在这个信息限制内，你大约有 1 兆字节来编码大脑，你不能把一个成年人需要知道的关于世界的所有事实都塞进这 1 兆字节里。正是因为这个信息瓶颈，你必须说，好吧，去他妈的，你将出生在这个世界上，这里是你需要接收周围信息并利用它来增长知识，增长长期记忆的最低算法，最终它可以在运行时展开成一个知识库，而这个知识库实际上可以有几千兆字节。所以即使我只有大约 1 兆字节来告诉你如何解压成一个有机体，变成一个心智，即使我只有 1 兆字节，你最终将拥有几千兆字节的学习，你将拥有几千兆字节的感官刺激来构建它。所以正是因为这个瓶颈。所以回到马赫的类比，他说，哦，看看童年的论证，人工智能将有一个童年。不，它不会，因为人工智能可以只是一个 1 兆字节的规范，然后它到处复制，就这样。没有 1 兆字节的瓶颈，没有自然选择必须将某物挤进一根该死的 DNA 链的等价物，这是一块物理碎片，生活在一个细胞中，生活在你身体的每个细胞中。这些是疯狂的随机限制。我总是笑，当我看着我们是第一个从这个其他过程中出来的智能时，这并没有告诉你智能通常是什么样的。人类智能是一个真正的表演，我们工作的方式如此依赖于这种随机的历史，而智能只是会拥有理性的设计。它们不会有 1 兆字节的瓶颈，它们不会有人类的偏见，它们不会有存在危机。是时候我们认真对待智能科学了。罗宾逊·克鲁索的论证，我们的智能有很多是基于我们共同努力和在一起的。你知道，我们所有或大多数人都接受了高等教育，那是数千年来积累的知识，被教授们提炼并灌输给我们。你知道，我们作为物种的整个经历是，智能是需要一个集体群体来完成的事情。你不能只是让世界上最聪明的人孤岛上什么都没有。他们会应付，他们会很有创造力，但他们远达不到他们的全部潜力。所以当我们第一次创造一个思考的实体时，它不会接管宇宙。它会感到孤独和悲伤，你知道，需要我们来引导它。这是人类作为有机体的另一个偶然事实，我们是一群独立的身体，一群独立的思想，每个思想都非常有限，对其他人有大量的依赖，既是为了弄清楚事情，也是为了维持身体的生存。所以我们需要其他人对我们好，在我们无法自己耕种所有食物或狩猎所有食物的日子里。我们最好能从我们能偿还的其他人那里得到一些食物。你知道，诸如此类的事情。我们最好能从某个专门做衣服的人那里得到一些衣服。这些动态不适用于人工智能，它超级智能，可以到处复制自己，然后可以微观管理整个经济，它会养活自己。这违反了你从人类如何工作中学到的所有假设。所以这就是为什么我一直说，这不是关于类比人类经验，马赫严重依赖于此。这只是关于从非常简单的第一原理思考，并使用相当简单的逻辑来推断一个高智能水平的可能预期。是的，我确实定义了高智能水平，让我们定义它，让我们推理其含义，并停止所有这些末日辩论，然后思考像暂停人工智能这样的政策。那就是我希望达到的目标。所以，关于内部论证就到这里了，我想谈谈外部论证，这正是我想要进行这次演讲的真正原因。绝大多数人都喜欢外部论证。这太诱人了，就像这个人有不良动机来提出这个论点，这个人从那个人那里收钱。如果你相信这个，它会让你像这样行动。对于我们这些喜欢互相抹黑的泥泞人类来说，这太容易了，太直观了，太自然了。我不会是那个抹黑的人，做人身攻击，卷入外部论证。但让我们听听马赫有什么要说的。基本上，真诚地相信这些东西会让你变成什么样的人？答案并不好。外部论证是这样的，有一种宏大感正在蔓延，而这种宏大感基本上是，要么全有，要么全无。我们是必须实现这一代人，否则我们就注定灭绝，或者注定在计算机的心智中过上某种地狱般的生活。如果事实是我们注定要灭亡，那么无论它让我变成什么样的人，我都在呼唤末日，我想我就会成为那样的人。因为如果我们注定要灭亡，你可以打赌我会说我们注定要灭亡。我不是那种回避巨大新闻事件的人，比如我们注定要灭亡。抱歉，我再次引用博斯特罗姆的话，他谈论所有可能的未来生命以及风险是什么。如果我们用一滴喜悦的眼泪来代表一个人一生中经历的所有幸福，那么这些灵魂的幸福可以填满并重新填满地球的海洋，每秒钟都这样做，并持续一千亿亿年。确保这些真的是喜悦的眼泪非常重要。这对一个 20 岁的开发者来说太沉重了，你知道，这对这些数万亿的生命来说是一个相当沉重的责任。这让我想起一些我不喜欢的事情。我对这种语言有一种本能的反应，因为我依稀记得童年时生活在马克思主义社会，我们要改造世界，然后它最终会慢慢渗透到日常生活可能会改变的地方。但第一份工作是解决人类的命运。当你提出这些所谓的外部论证时，这些论证与我们为什么注定灭亡的内部论证无关，只是说，看看相信我们注定灭亡的这些人意味着什么。我真的没有什么可以回应的，除了内部论证本身会说话。如果你相信内部论证，并且这让你产生了马克思主义者的感觉，那就拥有马克思主义者的感觉。否则你就注定要灭亡，我能说什么？好吧，我现在是马克思主义者了，我想我并不相信马克思主义。我通常不会把自己和那些人归为一类，但如果这就是让你知道我们注定要灭亡所需要的话，并且再次假设我们确实注定要灭亡，那么那些警告我们注定要灭亡的最好的人就会给你那种感觉。如果你将告诉我我们注定要灭亡与你讨厌的这种感觉联系起来，那么我将拥有你讨厌的这种感觉，我们只能处理它。我妈妈以前说过，共产主义下的人都有一种疾病，就是你眼睛看到和耳朵听到的不一样。我感觉到了同样的症状。我住在加利福尼亚，那里是美国贫困率最高的州，尽管它是硅谷的所在地。我看到我富裕的行业对改善人们的日常生活毫无帮助，但他们却在拯救数万亿的生命。在未来，我不这么认为。如果你看看有效利他主义，他们现在已经确立了自己，他们每天都在做诸如提高动物福利、为非洲的疟疾提供蚊帐、直接向非洲穷人捐款，以及拯救未来的数万亿生命。他们做到了这一切，他们只是善良的人，他们试图以极大的慈善来帮助一切。这些都是真正的人。我爱有效利他主义者，我是一名有效利他主义者。所以，我开始厌倦这些外部攻击，因为最终，我来这里参加末日辩论不是为了格挡外部攻击。你可以在社交媒体上找到这些，无休止地，因为每个人都喜欢互相抹黑。但就像它不适用一样，伙计，我们关心所有不同的好事。我认为这里没有与自大狂有关的矛盾。好吧，这种债券恶棍，这真的很令人毛骨悚然。人们认为人工智能将接管世界，所以这是智能人应该先接管世界并试图修复它的理由，以确保人工智能是健康的。我并不声称聪明人需要赶紧修复世界，我只是认为，如果超级智能人工智能很有可能摧毁世界，那么我们应该暂停创造我们无法控制或对其进行对齐的超级智能人工智能的努力。乔伊，麻省理工学院媒体实验室的负责人，有一句很棒的话，他说，这可能会让我的麻省理工学院的一些学生感到不安，但我的担忧之一是，主要是男性、大部分是白人的孩子在构建人工智能的核心计算机科学，他们更喜欢与计算机交谈而不是与人类交谈。他们中的许多人认为，如果他们能制造出科幻小说中的通用人工智能，我们就无需担心政治和社会等所有混乱的事情。他们认为机器会为我们解决所有问题。所以，意识到世界不是一个编程问题，他们想把它变成一个编程问题，通过设计一个将解决我们所有问题的产品。这是自大狂，我不喜欢。好吧，世界上现在很少有可信的人声称拥有超级智能计划。最接近的是像伊利亚·萨特这样的人，他创办了一个神秘的新实验室，名为“安全超级智能”。但他从未声称拥有超级智能计划，他只是说他会一直研究直到解决它。如果他在未来几十年内取得任何进展，我会感到震惊。我认为这是一个棘手的问题的梦想。伊利亚在这方面过于乐观。但根据马赫的观点，我们现在的问题是像萨姆·奥尔特曼这样的人，他们的主张不是“我们将通过编程走向更美好的未来”，而是“我将利用我的领导技能来引导这个项目取得成功，为人类物种服务”，并且忽视了实际的对齐问题。所以，这是一种与乔伊的指控非常不同的指控。它并不是 Sam Altman 认为他可以通过科幻小说来编程走向未来，更像是一种标准的鲁莽权力攫取，各种领导者现在都在这样做。超人类巫毒，一旦你开始谈论人工智能，就会出现一整套信念。如果你有一个非常聪明的人工智能，它首先能制造纳米技术。纳米技术就像魔法，因为它能制造任何东西。所以你有一个后丰裕社会，没有更多的需求。当然，纳米技术也可以扫描你的大脑并上传它，这样你就不会再死了，你就是不朽的。而且它甚至可能复活死者。你知道，这些机器可以进入我的大脑，查看我父亲的记忆，并创建一个他可以与之互动的模拟。你知道，就所有意图和目的而言，他会像他一样行事。所以，你知道，这个假设中包含了很多东西，人工智能上传是另一个，我们将能够占据，你知道，这些人工世界或身体，因为我们可以完全扫描我们的大脑。是的，这些都是非常标准的、有充分根据的超人类主义主张。他刚才列出的东西听起来有点像魔法和惊人，就像“哦，我的天哪，扫描你的大脑，然后让一个拥有相同精神状态的克隆人接替你死去的父亲的位置”。这听起来很疯狂，但如果你倒退一千年，你说，“嘿，会有飞行器，200 个人会爬进一个管子里，它会有翅膀，它会安全地将他们送到世界 khác 的地方，而且每天都会发生成千上万次，穿越整个世界。哦，而且我们还会假装，移动的图片和声音，都是一堆数字，我们会把它们通过携带这些微小粒子（称为电子）的管道发送出去，而在世界的另一边，你会感觉像是与你认识的人进行了即时对话。你知道，这些听起来同样疯狂和科幻，也像魔法一样。所以，这些超人类主义的主张，他称之为巫毒，听起来就像向一个还不生活在未来的人描述未来。相当标准，而且不知何故，银河系扩张也随之而来。我从来不完全理解为什么我们必须立即扩张到银河系，但这似乎是超人类主义思想的主食。没有必要急于扩张到银河系，我们可以推迟几十年甚至几个世纪，但我们最终还是想这样做，原因和我们想扩张到地球一样。为什么限制你的扩张？归根结底是宗教 2.0。人们称之为“书呆子末日”，确实如此。这是一种聪明的技巧，因为你不是一开始就相信上帝，而是构建一个可以成为神一样的实体，然后，就所有意图和目的而言，它拥有所有属性。它是全能的，全知的，而且要么是仁慈的，如果你正确地编程，没有引入任何错误，或者它是，你知道，它是魔鬼，你任由它摆布。有一种紧迫感，你必须立即行动，一切都悬而未决。这是一种非常宗教的感觉，因为这些论证诉诸宗教情感，这使得它们在人们心中扎下了坚实的根基。而他们合理化了为什么他们有这些信仰，但内心深处，这些是披着其他外衣的宗教信仰。好吧，所以告诉我如何不信教，考虑到也许世界真的要结束了，很多专家都在警告，嘿，人类世界可能要结束了。这可能只是一个事实问题。所以请告诉我如何以最不宗教的方式与你沟通这一点，只关注事实，不试图崇拜任何东西，不试图灌输任何人，只是指出我们注定要灭亡，人类可能没有未来，就像恐龙没有未来一样，就像 99% 的物种都因为一次比恐龙灭绝更早发生的超级火山爆发而灭绝一样。所以灭绝会发生。我只是想敲响警钟，可能存在灭绝。请帮助我以非宗教的方式做到这一点。它们导致了一种漫画式的伦理，你知道，一切都是关于通过技术和技术能力来拯救世界。我有一个幻想，我想看一部蝙蝠侠电影，大家都知道蝙蝠侠是布鲁斯·韦恩，他们只是不得不迁就他，因为他是他们的老板。所以他们为他创造了虚假的场景和犯罪来解决，他有他尴尬的蝙蝠腰带和东西。所以我认为我们硅谷的许多程序员和思想领袖都将自己视为现代蝙蝠侠。有趣的是，没有人是罗宾。我只是想继续指出，每当你承认自己正在忽略内部论证，因为你想提出外部论证，而你的外部论证是那些说世界将要结束的人就像漫画书人物一样。好吧，你知道，继续发送你有什么给我。我实际上认为世界可能要结束了，我只是想谈谈。模拟狂热，这里有人听说过模拟论证吗？有几个人听说过。如果你相信人工智能是可能的，并且能够设计出非常非常高性能的计算机，那么你就可以用一个简单的论证来说服自己，它能够模拟其他世界，而且仅仅通过数学，我们生活在这些模拟中的可能性要大得多，因为它们比基础现实多得多。哦，我没有吓到你。人们相信这个。所以埃隆·马斯克，你知道，实际上认为他提供了十亿比一的赔率。有人还没有找到他，但有人付了硅谷的两个程序员来尝试黑掉这个模拟，这太粗鲁了。我住在模拟中，不要分段错误，好吗？我正在使用它。所以模拟狂热对现实非常不稳定，因为如果你认为，让我回顾一下并解释它是如何工作的。假设你生活在一个后奇点世界，我们拥有这些超强的计算机，你是一名研究第二次世界大战的历史学家。你想问，如果希特勒征服了莫斯科而不是仅仅停下来会发生什么？所以你创造了场景，你模拟了整个世界，军队正在前进，你看到了会发生什么，你写了你的论文。但因为模拟非常详细，其中的实体是有感知能力的。所以你不能只是关掉它，因为那将是，你知道，战争罪。撇开你已经创造了种族灭绝的事实不谈，作为历史背景的一部分。所以你必须让它继续运行，因为你知道你的道德委员会是这样说的，你必须这样做。而这个二战模拟将发展出自己的技术，很快就会发现人工智能，并开始编写自己的模拟。所以它是模拟一直向下，直到你用完 CPU。这就是这个论证的由来，这些模拟世界比基础现实要多得多。但如果你相信这一点，你就相信魔法。因为如果我们生活在模拟中，我们对上一层的规则一无所知。我们甚至不知道数学是否也一样。也许 2 加 2 等于 5，也许 2 加 2 是，你知道，带刺尾巴的怪物。我们无法通过观察我们自己的处境来获得任何信息。人们可以轻易地死而复生。你知道，如果你保留了正确的备份，你可以重新实例化它们。如果我们能与运行模拟的人沟通，那么我们就有了直接通往上帝的途径。所以，当你深入模拟世界时，这是一种强大的精神溶剂。你开始发疯。我个人发现模拟论证相当令人信服，如果我必须猜测我们生活在模拟中的几率，我可能会说 50% 左右。我不认为任何一方的论证都非常有说服力。但每当我争论暂停人工智能时，我并不是说让我们暂停人工智能，因为我们是一个模拟，或者因为我们不是。我是在说，让我们暂停人工智能，原因和我们不打核战争一样。这只是假设现实将继续遵循它所遵循的物理定律，然后采取相应的行动。所以为了在这个现实中不死亡，无论它是模拟的还是不是，总的来说，生活在模拟中似乎对我们的决策没有影响。也许当我们超级智能时，我们会意识到我们可以通过根据我们是否处于模拟中而采取不同行动来增加我们的效用。但这并不是我们现在做决策的方式，也不是我提议我们应该如何做决策的方式。所以我认为马赫演讲中关于模拟论证的部分只是一个离题。数据饥渴，正如我提到的，我们现在发现最有效的获得有趣人工智能行为的方式就是将数据输入它们。这创造了一种非常具有社会危害的动态。我的意思是，我们正处于将这些奥威尔式的麦克风引入每个人的家中。所有这些数据都将用于训练神经网络，然后这些神经网络将越来越擅长倾听我们想做什么。但如果你认为通往人工智能的道路是通过这条途径，那么你真的想最大化收集的数据量，你想致力于这些大项目。它强化了我们必须尽可能多地收集数据并进行尽可能多的监控的想法。不，人工智能末日论者并不主张比公司已经收集的更多的数据。最终，我认为人工智能风险就像程序员的弦理论。它很有趣，你可以构建思想塔，然后爬进去，把梯子拉上来，这样你就与任何东西都断开了连接，而且无法检验。创造我们不知道如何做的事情。我认为思想塔没有那么高。我说效用最大化是一个吸引子状态，因为能够最大化效用的代理，当它们产生最大化效用的想法时，它们就会变得更加硬核。这是一条单行道，如果你今天不想最大化效用，也许明天你会产生一个想要最大化效用的代理。但如果你是一个想要最大化效用的代理，你不会产生一个不想要最大化效用的代理。从这个意义上说，这是一条单行道。而智能的定义是创造性地击中指数级搜索空间中的小目标。我的意思是，是的，这听起来很复杂，当你第一次听到它时，但它并没有多少内容。这些是简单的概念，具有简单的逻辑含义。我一直在谈论智能科学。智能科学不是一个特别复杂的领域，它只是一个新领域，有很多困惑。它可能被证明是复杂的，但末日论证并不需要这种复杂性。它不需要建造一个大塔。大塔是可选的。如果我们想建造超级智能人工智能并生存下来。如果我们只是想暂停人工智能，就不需要一个庞大的论证来看到显而易见的原因。最后，它激励疯狂。人工智能风险深度思考的一个标志是，你的想法越离奇，你就越有信誉，因为它表明你足够勇敢，可以沿着这些思路一直走到最后一站。是的，这确实是我们在这里做的，在末日辩论中，我们沿着末日列车一直走到最后一站。每一次人们试图因为你描述的所有原因而下车时，我们都必须把他们拉回来，说“不，不，不，不要在这站下车，继续乘坐火车。你没有找到一个好的下车点。你有一个非常薄弱的理由在这里下车。所以回到火车上，除非我们终于面对事实，并踩下真正的刹车，比如暂停人工智能。雷·库兹韦尔，你知道，相信他永远不会死，他已经在谷歌工作了好几年，并且正在研究这个问题。如果谷歌能战胜死亡，我会非常恼火，因为想象一下，你有 30,000 年的浏览历史，这会如何跟着你。我认为最有害的影响是我称之为人工智能角色扮演。那些真正相信人工智能是真实且即将发生的人，开始表现得像他们幻想中的人工智能会做什么。在他的书中，尼克·博斯特罗姆概述了人工智能需要做的六件事才能成功接管并实现其目标。这份名单是智能放大、战略规划、社会操纵、黑客技术研究和经济生产力。如果你看看硅谷的人在做什么，他们试图表现得像他们最喜欢的人工智能英雄。这是一种反社会行为，但我们看到了。萨姆·奥尔特曼是我最喜欢的例子，一个拥有 Y Combinator 的人。有各种各样的尝试，比如从头开始重塑城市，最大化个人生产力和时间，在幕后做事情来影响美国大选。这非常阴险，它会激起非技术人员的强烈反对，因为你不能只是，你不能无限期地推动权力杠杆，否则它会惹恼你所在的民主社会。好吧，这算不上一个论证，我不会回应。我甚至在这里做了笔记，我听过所谓理性主义社区的人称那些对世界没有重大影响的人为非玩家角色。我曾几次称某人为 NPC。我认为这是一个很好的侮辱，当某人真的应该知道他们可以采取控制某事的方式，他们可以施加真正影响的方式。它只是需要跳出盒子一点，打破一点模式。所以如果他们处于他们真的应该这样做的位置，但他们拒绝了，那么我扔个侮辱给他们，好吧，好吧，成为一个 NPC。我认为这是恰当的。我不是经常说，但它是一个很好的工具箱词汇。这太可怕了。所以我身处一个理性主义者是最疯狂的行业。这让我很沮丧。我把这些人工智能角色扮演者看作是九岁的孩子，他们在后院玩耍。

在帐篷里，你知道，然后他们开始，他们在帐篷的墙壁上投下阴影，他们开始吓坏自己，但实际上他们只是在对自己投射的影像做出反应。你如何想象终极智能的行为，以及你作为一个认为自己比世界上其他人更聪明的人的行为之间存在一个反馈循环。这非常非常有害。那么答案是什么？解决方案是什么？我们需要更好的科幻小说。好吧，这是斯坦尼斯·拉姆，伟大的波兰科幻作家。英语科幻小说很糟糕，它很枯燥，它很无聊。在东欧集团，我们有好的东西，我们需要确保它被妥善出口。它已经被很好地翻译了，它只需要更好地传播。所以斯坦尼斯·拉姆和斯特鲁加茨基兄弟是我想到的。我希望听到你们的其他建议。东欧科幻小说的与众不同之处在于，这些人是在艰难的环境中长大的，经历过战争，然后生活在极权主义社会，不得不通过写作委婉地表达思想。因此，他们对人类经验和乌托邦思维的局限性有着切实的理解，这在西方是完全缺失的。这并不是说这是不可能的，我认为斯坦利·库布里克能够做到这一点，但你知道，我们只需要把它找回来。看看《海尔玛丽计划》，那是一部很棒的近期科幻作品。最后，我想坦白我的想法，我想坦白我对人工智能及其可能性的看法。既然我一直在嘲笑它，那么公平地说，我认为人工智能现在处于与17世纪炼金术相同的地位。炼金术士名声不好，我们认为他们是神秘主义者，没有做多少实验科学，但研究表明他们实际上非常勤奋。在某些情况下，他们使用了现代科学技术，并且他们有一些非常非常好的想法。例如，他们确信物质的微粒理论，即一切都由微小的粒子组成，并且这些粒子可以重新组合以创造不同的物质，这是正确的。他们走在正确的轨道上，他们的问题是他们没有足够精确的设备来做出他们需要的发现。炼金术士需要做出的重大发现是质量平衡，即你开始时所拥有的一切和你最终所拥有的重量相同。但其中一些可能是气体或液体，然后他们只是没有精确度。不得不等到18世纪。他们有一些没有帮助的线索，所以水银是一种在室温下呈液态的金属，你知道，哇，这对我们来说没什么大不了的，它只是元素周期表的一个怪癖，但对他们来说，这似乎是一个非常重要和深刻的线索。水银是这个炼金术系统的核心，他们也在寻找能够将贱金属变成黄金的哲人石。所以我开始思考，如果我们能把一本现代化学教科书送回过去给罗伯特·理查德·斯塔基或艾萨克·牛顿爵士，他做了很多炼金术研究，他们会有什么反应？我想他们首先会翻阅一下，看看：你们找到哲人石了吗？这是他们追求的关键。我们的答案有点像，是的，是的，但你知道，它会产生放射性金。所以他们会说，放射性是什么？嗯，你知道，看不见的魔法射线，如果你站在同一个房间里，它们会杀死你。所以很多挣扎将是不要让它听起来神秘。对我们来说它是科学的，但对他们来说它是非常非常巫毒和诡异的。然后我们会说，你知道，但我们确实使用了你的哲人石的变质，以便制造这种非常有用的金属，因为如果你把两块它砸在一起，你可以炸毁一座城市，所以这很酷。然后我们会说，实际上你一直在寻找的哲人石就在天上，你看到的每一个星星都是你，是改变元素从一种到另一种的反应的来源，你身体里的每一个粒子曾经都在一颗恒星里。所以想想这件事，以及你的感受。不仅如此，你知道，我们发现将我们联系在一起的力量与天上的闪电是相同的，我能看见你的原因和你给我火花的原因是相同的，我抚摸它时，以及阻止我穿过地板的力量也是相同的。并且这些力量由简单的数学定律定义，这些定律可以写在一张索引卡上。马赫的演讲的这一部分很棒，他非常擅长谈论科学，总的来说是一个聪明人。我认为，如果不听起来像一个神秘主义者并召唤上帝以及那些你知道的，他们信仰的核心概念，就不可能传达这一点。所以我想我们在人工智能方面处于同样的境地。我认为我们有一些线索，我们有一些非常重要的线索。有意识的奥秘，我头上的这个肉盒子是自我意识的，而且希望，假设，除非这是模拟，你们也通过意识体验到这一点。但我们甚至不知道如何提出关于意识的问题，因为它是如此，你知道，我们在黑暗中迷失了。我们还有其他线索，比如每个聪明的动物似乎都需要睡觉，而且似乎会做梦。我们知道孩子的大脑是如何发育的，我们知道语言对认知有多重要。我们拥有所有这些碎片，我们也拥有计算机科学的碎片。我们看到我们在识别图像和声音方面取得了真正的成功，就像大脑似乎做的那样。所以有很多东西即将到来，但也有很多东西我们犯了可怕的错误，不幸的是我们不知道它们是什么。还有一些东西我们严重低估了它们的复杂性，就像炼金术士一样，他们一只手拿着一块石头，另一只手拿着一块木头，认为它们是大致相同的物质，却不理解木头比它们复杂得多。我们在研究思想方面也是如此，这很令人兴奋，我们将学到很多东西，但这需要一些时间。在此期间，我喜欢这句话：如果每个人都在沉思无限，而不是修理排水沟，我们中的许多人将在不久的将来死于霍乱。我们必须面对的那种人工智能和机器学习与此大不相同，它有自己的伦理问题。就像如果那些阿拉莫戈多科学家决定完全专注于他们是否会炸毁大气层，而忘记了他们正在制造核武器，而核武器也必须处理。等等，但我们必须专注于核试验是否会炸毁大气层的问题，如果它会炸毁大气层，那么就不要进行核试验，就像停止一切，同时专注于这一点并解决它。所以他的建议适用于我们解决了核试验的危险性之后。这是一个例子，如果我们遵循他自己的类比，我们真的需要努力测试或以某种方式知道人工智能不会炸毁世界，然后才继续其发展，然后处理持续发展的其他问题。因此，机器学习中存在伦理问题，它们不是关于事物变得有自我意识并接管世界，而是关于人们如何剥削他人，或者通过思维的疏忽将某种不道德的行为引入我们日常生活中越来越重要的自动化流程。当然，还有机器学习和人工智能如何影响权力关系的问题。我们已经看到，监视已经以一种意想不到的方式成为我们生活中的事实部分。我们从未预料到它会是这样的，但它就在这里。这是美国国家安全局的数据中心，供不熟悉的人了解。所以我们有这个世界，在这个世界的顶端，那些掌权的人痴迷于一个疯狂的想法。所以我希望我今天所做的就是向你们展示，你可以从斯蒂芬·霍金的猫那里学到一些东西，无论他们怎么告诉你把它放进猫包里，做你自己的事。我希望我今天让你们变得更聪明了一点，这样你们就不会被超级智能的想法所困扰，而且我希望我能说服你们，在我们行业顶端的人们缺乏良好领导的情况下，我们有责任努力做出贡献，并真正思考人工智能为我们带来的所有伦理和困难问题。我主要想感谢你们让我在这里对着你们喋喋不休地谈论这个奇怪的话题，以及你们全神贯注的注意力。非常感谢。好的，马赫的演讲有两件事令人印象深刻。第一，在2016年，当除了埃隆·马斯克、埃利亚斯·奥夫斯基和尼克·博斯特罗姆之外，几乎没有人谈论人工智能的末日，这是一个非常小的圈子，他却决定发表一个关于为什么我们不应该担心末日并投入这场战斗的演讲。这现在是一场非常受欢迎的战斗，一场非常受欢迎的辩论，但他在2016年就已经在讨论了。所以向他致敬，他关注了一个现在几乎普遍认为非常重要且非常有趣但当时并非如此的领域。这是第一点。第二，他非常擅长谈论科学并建立这些联系。总的来说，我就是他的粉丝。他也是一位成功的企业家，经营着一个名为 Pinboard 的书签网站，这就是为什么他提到自己是一名网页开发者。所以这些是演讲中的亮点。然后从坏的方面来说，我只是觉得他的任何论点都不是很令人信服。而外部观点论点，你知道，我只是为了好玩而参与其中，我甚至不认为它们真的相关。内部论点我认为相当有分量。它让我谈论像自然选择的速度限制，这就是为什么我们基因组中只有一兆字节，这就是为什么我们甚至有童年阶段，而人工智能不会有童年阶段，因为它不会有一个一兆字节的基因组。所以那时我们深入探讨了问题的实质，这些问题告诉我们我们是否注定要灭亡，而我觉得他所有的论点都很薄弱，但原因很有趣。总的来说，我认为这是一次很棒的演讲，因为它列出了许多有趣的反对人工智能末日论的论点。我确实觉得我能够一一令人满意地解决所有这些问题。基本上，他向我射了很多子弹，我觉得我有一件防弹背心，我能够吸收它们。观众朋友们，你们来评判吧。马赫，如果你正在观看这个视频，并且有想法，我很想听听。更好的是，我很想让你来做一个关于你对末日更新看法的现场辩论或讨论。我的意思是，我是在回应你八年前说的话，所以完全公平地认为你已经更新了你的观点，也许你不同意你的一些说法，也许你有其他想法，也许你有更强的论点，为什么从2024年的角度来看，看到事情的发展，你有其他理由认为人工智能不会杀死我们。例如，也许你同意马丁·卡萨多所说的，嘿，这些LLM类型的人工智能只会比随机鹦鹉好一点，它们要么会利用大量的计算资源来模拟事物，要么会给你一些接近其他东西的东西，以及某种概率分布。我的意思是，这将是2024年有人提出的非末日论证的例子，而这在2016年是不会被关注的。所以无论如何，我很想从马赫那里听到更多关于这个的最新想法。我鼓励大家在推特上关注马赫，因为他很酷。今天的节目就到这里了。我还有更多激动人心的辩论和拆解节目即将推出。如果你有任何要求，请在评论中告诉我。只是提醒你，这是大多数观看这个频道的人和我一起进行的使命。我们都希望看到社会认识到人工智能的紧迫威胁，以及为什么暂停它是不鲁莽的唯一政策。所以如果你想尽一份力，如果你不想做任何疯狂的事情，如果你只想用鼠标点击几下尽一份力，我让你很容易做到。你只需要想几个朋友或者你在的论坛，这种内容会很有相关性，然后去粘贴一个链接。你只需要粘贴一个链接，或者去Apple Podcasts写一个关于Doom Debates的好评论，让更多人观看。或者，如果你还没有订阅我的YouTube频道，这是最简单的事情。你只需要去 youtube.com/DoomDebates，点击订阅。你就完成了你今天的贡献。非常感谢你的支持，下次再见，Doom Debates。 [音乐]

Get Our Mobile App

Maciej Ceglowski (Pinboard) Rejects AI Doomerism | Liron Reacts

Transcription