Is the Universe a Code? with Nick Bostrom

0:00This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online makes sense? There's no place like Chrome. Check responses set up require compatibility and availability varies 18 plus. For the first time ever, Yamavar Resort and Casino at San Manuel is giving away a new Rolex watch for every Club Serrano card deer. Play with your Club Serrano card and on September 20th, you could be one of five winners of a Rolex watch. Plus, all winners will advance to the finale at Palm's Casino Resort Las Vegas for a chance to take home a rarity. The second Mustang Dark Horse ever produced on September 26th. Two properties. Six winners. Only at Yamavar Resort and Casino. Your California to Vegas connection. Details at Yamavar.com must be 21 or older to enter participate. Please gamble responsibly. You have fallen into event horizon with John Michael Gaudier.

1:25In today's episode, John is joined by Nick Bostrom. Nick Bostrom is a philosopher at the University of Oxford known for his work on existential risk, the anthropic principle, human enhancement ethics, superintelligence risks and the reversal test. He received a BA in philosophy, mathematics, mathematical logic and artificial intelligence from the University of Gortonburg in 1994. He earned an MA in philosophy and physics from Stockholm University and an MSC in computational neuroscience from King's College London. During his time at Stockholm University, he researched the relationship between language and reality by studying the analytic philosopher W. V. Quinn. In 2000, he was awarded a PhD in philosophy from the London School of Economics. His thesis was titled Observation or Selection Effects and Probability. Also in 2000, he held a teaching position at Yale University and later became a British

2:29Academy postdoctoral fellow at the University of Oxford in 2002. All the versed event horizon makes a show for a few reasons. We love to do it. We wanted to listen to a show like this and another reason. What we do to help give other people something they can relax to and learn from. That's why, when people tell us they fall asleep listening to the show, most of the time a multiple playthroughs, it doesn't bother us at all. It's actually extremely gratifying. And knowing that we're quite happy that today's video is sponsored by Endel. If you need help falling asleep or just want to relax, you can use Endel, an environment-based app that takes everything we know about sound and combines it with truly cutting edge technology. This all results in providing you with personalized soundscapes that helps you focus, relax and

3:29get a satisfying night's sleep. Why are we a science-based show recommending Endel? Because we know it's informed by science, created with science and backed by science. So for a great way to support event horizon, relax and get help sleeping. Be one of the first 100 people to download Endel at endelg.io. forward slash event horizon. And you'll get a free week of audio experiences. Thank you to Endel for sponsoring today's video. Remember to subscribe to Event Horizon so you never miss an episode. Nick Bostrom, welcome to the program. Pleasure. Nick, you have given me over the years many existential crises, which I absolutely love. I love nothing more than a good existential crisis, but also the Fermi Paradox. And I spend no small amount of time thinking about the Fermi Paradox and why we just see the

4:29great silence when we look out into the universe. So my question for you is that with technological development, it's clear, I think, at this point, we could say that we could create some very, very dangerous technologies, molecular nanotechnology, artificial intelligence, super intelligence, things like that that could pose an eventual threat, eventual, not only to us, but if it escapes, it could pose a threat to anyone in the galaxy as this enormous super intelligence expands should it choose to do so. Was this deep in the Fermi Paradox? Because it would seem to me that civilizations would put up warning posts, you know, sending out setting messages that hopefully somebody could decipher and say, we should not build this technology. But we don't see that. Always see as a silence. So is it possible in your mind that we really are alone in the galaxy and that we need to watch what we do technologically in order to not go extinct?

5:31Yeah, it's possible. I mean, if we received a warning message like that, do you think we would heed it? I would say very likely not because of geopolitical situations. Everybody loves a good weapon and these things tend to get developed. And if you look at the development of the nuclear bomb, there were really no skids put on it. It just happened. So that's what worries me most is that we will develop technologies like this regardless of the dangers. And so I think there are quite strong competitive reasons for humans racing forward. And then similarly, if there are alien civilizations for them to develop technology that would allow them to colonize space. And since we haven't seen any, I think then that evidence for they're not being any. So to me, the Fermi Paradox has never really been that paradoxical in that we have at least one good explanation. And that might be additional explanations, but one would just be that it's very unlikely

6:34for any given planet to develop life at all, let alone technologically advanced life. And it seems to be that everything we know is perfectly compatible with this hypothesis. If the universe is large enough, I mean, if it is in fact infinite as it appears to be, and there would be infinitely many planets, and even if the probability for any one of them producing intelligent life was astronomical is small. With an infinite number of roles of the day, it would still get some coming up on this unlikely value where intelligent life develops. And so that would be even if we assume a great filter that makes it unlikely that any planet results in life, we would still expect there to be a lot of life, but just very widely separated. And that seems to match what we observe. Now, it could also be that there are additional possibilities that would explain the Fermi paradox that we're in some sort of zoo, for example, or more kind of complex explanations

7:35could be devised, but at least one. And so it seems to me a paradox is where there is kind of two things that are in conflict with one another. And it's hard to see how you could reconcile them. But in this case, I don't see what is exactly opposing what in a theoretical explanatory sense. In the case of an infinite universe, which we've measured essentially that the geometry of the universe is flat, so it looks like it very well could be infinite beyond the observable universe bubble that we have. In an infinite universe, how does this affect the concept of it all being a simulation? It would seem to me to run an infinite universe you would need infinite resources, right? Yeah, if you actually wanted to implement an infinite universe in sort of with local granularity. But to generate a universe that appears from some vantage point as if it's infinite, you wouldn't need infinite resources.

8:37If you just speaking crudely, hung canopy around us with sort of little needle tricks through it that you let light shine through to depict the stars. A more sophisticated version of that kind of thing would create the illusion for the people inside the theater that they were actually living in an infinite universe. And it would take only a finite amount to run. And we could kind of get some sense for the possibility that very realistic appearances could be generated with a small amount of resources if we just reflect on our own brain's achievement every night when they produce dreams that in some cases appear quite realistic and convincing. If that can be done by our own humble little brains, then imagine what could be done by a technically mature civilization running planetary-sized supercomputers. So all the things that we see certainly could be generated by such a device.

9:39I'm so essentially it would render that what you are looking at. And you would just see a superficial sort of universe looking out there. But within your own sphere, the Earth essentially, the simulation would become complex, but the further you look out, the less complex it would be, which actually is what we see. Because we lose resolution the further we look. That's right. I mean, you could also make it seem, I guess, such that it would appear complex at greater distances, as long as we weren't actually experiencing all of that complexity. Like if we just received some low bandwidth signals that were in man super-civilizations in our neighbor galaxies, that would make it appear as if there were maybe more complexity little further out. But our observations could still be rendered relatively cheaply, in that the main cost would be simulating our own brains and then at some crude level, the world around us,

10:39and then this low bandwidth signal from the neighbor in galaxy. Now biases, and you've written on this the anthropic reasoning. We always color our thinking whether consciously or unconsciously, with the fact that we are human, and we are humans with human brains thinking on these subjects. So how does one go about removing anthropic reasoning from thinking about things like technology or aliens? Yeah. So anthropic bias is a very particular kind of bias, arising from the fact that all our observations are filtered by the precondition that there be some observer in a suitable time and place to have the observation. And in most everyday context, this doesn't really arise, but there are particular contexts in which it becomes important. Such as when we were thinking about the Fermi paradox, as we were talking about, you

11:40might naively think that we have observed one planet in detail our own and here, intelligent life evolved and so that would be evidence that the emergence of intelligent life is easy, that it tends to happen on planets as long as the microscopic variables are roughly in the right range. But that I think would be to fall foul of an anthropic bias in that there is this obvious observation selection effect that guarantees that even if intelligent life is extremely rare, it would still be the case that every observer would observe themselves in a place where this possibly very unlikely thing happened. All observers will observe themselves having evolved from some planet, no matter whether that is very likely to occur on any one planet or very unlikely to occur on any planet. It looks as if the observational predictions of both of those hypotheses are the same and

12:43match what we observe and so our observations don't actually distinguish the two. At least not the mere observation that we came into existence on this planet. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50 page restoration block or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online makes sense? There's no place like Chrome. Check responses set up require compatibility and availability varies 18 plus. For the first time ever, Yamavar Resort and Casino at San Manuel is giving away a new Rolex watch for every Club Serrano card deer. Play with your Club Serrano card and on September 20th, you could be one of five winners of a Rolex watch. Plus all winners will advance to the finale at Palm's Casino Resort Las Vegas for a chance to take home a rarity. The second Mustang Dark Horse ever produced on September 26th, two properties, six winners, only at Yamavar Resort and Casino. To your California to Vegas connection, details at Yamavar.com must be 21 or older to

13:47enter participate. Please gamble responsibly. Now in regard to an even bigger issue, the anthropic principle or the fine-tuning problem that you're intimately related, so this universe appears from what we can see to be fine-tuned for the development of not only life, but just matter itself. You change a few parameters and matter can't exist. But this universe, which is all that is, barring a multiverse, which we can't test, then that would seem to suggest that it is a simulation, right? Or is that a step too far? I think it's a step too far. I think it certainly is primafacial puzzling that the constants of the universe are such that if they were very, very slightly different than we would have had a lifeless and maybe a matterless universe that things are as it were balanced on a knife's edge. It seems a fact that Christ out for some sort of explanation and different explanations

14:50have been proposed. Of course, these have the explanation that our universe was the result of intelligent design. You could imagine if our universe were created by an intelligent being who particularly wanted life to exist, then that being would adjust the constants to permit life to exist. So that would be one type of explanation. Another is the multiverse hypothesis where our universe is fine-tuned, but the whole ensemble of universes that exist need not be fine-tuned. You could have a very simple mechanism that just spews out all kinds of different universes, and that could be a very simple hypothesis because it doesn't need to be building in some very complex and precise criteria for what it is producing. And then given that multiverse is then stating a wide range of different parameter settings, then you invoke an observation selection effect to explain the fact that we find ourselves in a universe that appears fine-tuned, namely only those universes who contain observers.

15:52And so all observers, given such a multiverse hypothesis, would see this apparently fine-tuned universe that we see. So those would be two possibilities. Like a third might be some kind of simulation hypothesis, then of course you still have the question of the basement universe, the universe in which the computer running the simulation is built. What about that universe? Is that fine-tuned? And then you might ultimately again have to resort to a multiverse explanation or an intelligent design explanation to account for the fact that that universe is fine-tuned. I mean, yet the further alternative would be, and we don't know whether this is possible, to actually come up with a very simple fundamental theory that would just a few axioms implied at the precise parameter values that we see without using an ensemble of universes as part of the explanation. Some sort of superduper symmetry that just makes it pop out that the gravitational constant had to be just so, and the other constants had to have values just in.

16:56That would kind of be surprising, but at least if we map out the space of logical possibilities for what an explanation could look like, then that should also be included. Now the idea of a simulation, you have made the case that three criteria could be set forth to define and at least ask the question, do we live in a simulation based on the concept of an ancestor simulation. Now do you think that our technological development and our mindset as we are, would we ever create our own ancestor simulation and could we? Well, I would say that if we do, then that would dramatically increase the probability that the simulation hypothesis is true. That is, if we get to the point where we are creating our own ancestors simulations, we should conclude that we are almost certainly living in a simulation ourselves. And it looks like it is possible to do this, not for us today, but that we have a path

18:00that we could pursue a technological trajectory that will eventually lead to the development of capacities that would enable us by using a tiny fraction of our resources to create vast numbers of detailed computer simulations, including simulations of simulated brains that would be conscious of their simulated worlds and such that the experience of these simulated minds would be similar to the experience that we have. I think that's a physically possible technology that technological and mature civilizations will develop. One question I have for you, and this is somewhat off the wall, is say it's a fantasy simulation, in other words, this simulation that we might live in is someone else's fantasy. It is their ideal virtual reality, and that we might seek to create ideal virtual realities for ourselves. In other words, video games, so to speak, on a next level basis that we descend into virtual

19:01reality and sort of forsake life in the universe. But is it possible, perhaps, that the reason for creating a simulation is virtual reality and we simply live in somebody's video game? Possibly, yeah, I mean, sure. A lot of things are possible. In this scenario, I guess you could distinguish the type of simulation that would just be for the simulator to observe from the outside, like I don't know, Sim City or one of these world box simulations versus a computer game where the player has an avatar inside the world and is sort of actively shaping it and participating in it. Both of those are conceptually possible. Now, for scientific reasons, creating an ancestor simulation, say, for example, you are some entity, a Boltzmann brain in a dead universe and you're like, what was this universe like? I'm bored, everything's black, and I just have come into existence, I've popped in, and I'm going to simulate what this was. Do ideas like that? I mean, I guess what I'm asking here is that why would you create an ancestor simulation?

20:06I mean, would it just simply be academic? In other words, you want to try to reconstruct what once was in a universe, say, the far future when the red dwarfs and everything are blanking out in this universe, any civilization that's there might be like, well, what was this like? Because it would be extraordinarily hard to study cosmology, for example, when you can't see everything in a huge expanded universe, would that be a motivation for creating an ancestor simulation? Think if you're a Boltzmann brain, you're in a bit of a pickle, in that you probably won't last that long. Most Boltzmann, this would be sort of thermal fluctuations that just happen by pure chance to take the shape of some sentient brain. But a free-floating brain in the middle of some intergalactic gas clouds wouldn't survive that long. In fact, almost all Boltzmann brains would have observations that would be to some high

21:11degree chaotic, I think, for every mind that has the relatively coherent shape that ours do, as a result of having evolved and having to be functionally adapted to survive in the world around us. For every such Boltzmann brain that does have experience, there would be trillions and trillions of ones that were just more haphazard. Since the Boltzmann brains wouldn't have been filtered by any kind of functional adaptiveness criteria. And so if we were Boltzmann brain, I think we should expect to see something more chaotic, like the kind of smallest possible experience. If you started with a single atom and then started adding atoms at random, what's the smallest possible cluster that would at least produce some sort of sentience experience? And maybe it would be something analogous to watching static on the television, but

22:17rather than a big high-resolution screen, just a few pixels in some sort of blurry, diffused consciousness. I think you would get astronomical and many more of those than you would get a high level, highly organized anthropomorphic-style minds like the ones we possess. One wonders what effect time would have on that, because if time is inherent to this universe space-time, and it may be for the Boltzmann brain a blink of an eye, its existence, and it just packs in a whole bunch of simulation into that very brief existence, but to us, it seems like trillions of years. Well, I'm not sure why subjective time would flow faster for a Boltzmann brain if it were made of the same stuff as we are made of. If it had biological neurons and stuff like that. If it sort of spontaneous materialized as some kind of super dense neutron star pattern,

23:19then, yeah, sure, maybe it would do more in one minute than we would do in a minute. Although by that token, it would also be more likely that the minimum viable product would last for a lot less than a minute. And so again, I would expect the majority of experiences to be of the minimal, very short-lived, very chaotic kind that I described earlier. Now, simulation theory itself, do you think, in your view, that it's testable? I mean, could we ever even determine if this is an ancestor simulation, or is it going to be something that we forever wonder about, because we can't eliminate the possibility? Yeah, I think there are different observations that would provide evidence for or against the simulation hypothesis. At one extreme, we have, say, the possible observation of seeing a window popping up in front of you with text explaining that you're in a simulation and offering you, like,

24:20further information that you could click on. That would be pretty convincing for the simulation hypothesis. Another type of evidence that would increase the probability of the simulation hypothesis is if we ourselves continue down the path to developing the technology needed to build them, the closer we get to technological maturity, and the more we retain our present interest in creating these types of simulations, the more likely it is that we will create our own ancestors simulations, and that then would eliminate the two alternatives to the simulation hypothesis. And if your listeners have read the original simulation argument paper, you remember there are these three alternatives that it tries to show at least one of them is true. And one is that almost all civilizations at our current stage of development go extinct before reaching technological maturity.

25:21So that hypothesis would go out of the window if we ourselves reached technological maturity that would suggest that that wasn't so hard. The second alternative is that out of all these civilizations that do reach technological maturity, almost none of them remain interested in using their planetary size supercomputers to actually create large numbers of ancestors simulations. But again, if we reached a point where we were doing this kind of stuff that would also eject this second alternative and that would buy the simulation argument, only leave the third, the simulation hypothesis itself. And you can check the simulation argument for the reasoning behind this, some simple probability theory. But what that means is that anything that makes it more likely that we will reach this point where we are creating our own ancestors simulations would also be empirical evidence in favor of the simulation hypothesis. And of course, conversely, the absence of either seeing windows in front of us announcing

26:25that we're in simulations or observations of things that make it less likely that we will make it through all is evidence against the simulation hypothesis. For example, if we discovered some extremely dangerous technology that looked like every civilization would discover at some point before they developed the capability of creating ancestors simulation. Some technology on the way there that would inevitably destroy whoever discovered it. If we find evidence of that, then that would increase the probability of the first alternative and so on. Who would remove the reasons we have for believing the simulation hypothesis and reduce the probability of that hypothesis? Now, another idea that you have put into the debate is the idea of a singleton. And broadly speaking, a singleton for all its pluses and minuses, I mean, such a thing

27:26could be absolutely amazing or it could be an absolute nightmare and a mortal dictator or an artificial intelligent dictator. But one thing that it could do is impose ethics and say no ancestors simulation because it's unethical. You're creating beings, so to speak, in an artificial environment and you just can't do that. That be the answer to one of the three that they simply, all civilizations simply never create ancestors simulations because they feel that it's unethical and something makes a singular decision, a singleton that stops it. Yeah, that's the second alternative, this strong convergence towards refraining from creating ancestors simulations. That could be perhaps implemented if there were a very strong convergence to develop singletons so that each technological civilization became a singleton. And then additionally, we'd have to postulate that all these different singletons scattered

28:27through the universe arrived at the same conclusion as to what they should do with respect to creating ancestors simulations, i.e. refrained from it. But that certainly is one way in which the simulation argument could be true. We think that that could be a case of consensus by just pure logic. In other words, you reach a singleton and the answer to the question is always the same. In other words, they go through the logic of it and they say, we don't do this. The next civilization over that's developed singleton also arrives at the same conclusion and that there are absolute truths, absolute conclusions possible in the universe that singletons will arrive at unanimously. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block or finally break down that long article you've had open for weeks.

29:27Gemini and Chrome is here for it. Ready to make anything online makes sense? There's no place like Chrome. Check responses set up require compatibility and availability varies 18 plus. For the first time ever, Yamavar Resort and Casino at San Manuel is giving away a new Rolex watch for every Club Serrano card deer. Play with your Club Serrano card and on September 20th, you could be one of five winners of a Rolex watch. Plus, all winners will advance to the finale at Palm's Casino Resort Las Vegas for a chance to take home a rarity. The second Mustang Dark Horse ever produced on September 26th. Two properties, six winners, only at Yamavar Resort and Casino, your California to Vegas connection. Details at Yamavar.com must be 21 or older to enter participate. Please gamble responsibly. Yeah, that seems possible. Might appear incredibly unlikely, but we need to remember here that the single-transient question that would have the ability to do this would presumably be super-intelligent. That if they could develop the technology to create huge numbers of ancestor simulations,

30:31they would also have the ability to increase their own intelligence or develop machine super-intelligence. And so the likelihood that there could be some such very strong and striking convergence is increased, I think, by the fact that this would all be super-intelligent minds and there might kind of pretty reliably tune into the same considerations. So if there is some consideration that actually decisively disfavorites them, creating ancestor simulations, they might discover it and fully realize its ramifications. So in the development of a Singleton, whether it's a human Singleton where everybody is on the same page, so to speak, and everybody's augmented and brain augmentation, everything else, and everybody starts arriving at the same conclusions. Do you think that that leads to a universe, essentially, of singular thinking? And that that could solve the Fermi paradox and that everybody just concludes, nope, nope, nope, don't go too crazy with technology, don't develop AI, don't go out into space and create galaxy spanning empires, node, Dyson spheres, things like that.

31:34So we're going to focus on maintaining your resources for as long as you can and just hang on. And that's the simple solution of the Fermi paradox is that eventually everybody thinks the same. Yeah, I think the possibility of a scenario where everybody thinks the same, but some of these fundamental questions is increased if we postulate super-intelligence. Because one source of variation is error. People make different errors and so they come to all kinds of different beliefs and opinions about text with super-intelligence that would be less likely. So the increased chance to do is a convergence. I don't think it's sufficient though to have a convergence about empirical beliefs about how the world works. There would really have to be some convergence of motivation, right, so that they would all be motivated to stop whether building super-intelligence is or colonized space. And again, it's conceivable that there could be a fairly strong such convergence if there are instrumental reasons that were discovered for refraining from these things.

32:35I would say that with space colonization, it's a little bit perhaps harder to see why exactly they would convergently arrive at a decision to refrain from this. With simulations, I mean, there are like kind of ethical considerations and maybe game theoretic reasons for why this could be convergently discouraged. But it seems just such a waste to have all these cosmological resources out there going to waste. I mean, we are talking about conserving energy, right? Then you should switch off the lights when you go outside and all of this stuff that we are talking about. But look out there in the universe. There are suns, illuminating empty rooms, vast amounts of neg entropy just being flushed down the toilet on a massive scale. Like at some point, it seems some civilizations would want to go out and make some better use out of this. And I don't know, maybe there is some clever reason we haven't thought of why this would be a bad idea from the perspective of all these different values and goals that these civilizations might have. But it's a little bit harder to see that, I think, than to imagine that there could be

33:39some such reason for refraining from creating answers to simulations. The elephant in the room is yet another option on the list and that is that everybody goes extinct before you get to this stage where you can create an ancestor simulation. Do you think that this could be the downside of a singleton, meaning that the extinction of humanity occurs as a result of the singleton and that that's it. And nobody ever gets to the level that you need to or this could go for any any technological trap that sits in the way where there's just some stage in technological development that a civilization hits causes its own extinction and cannot ever get close to the power of being able to create an ancestor simulation. In other words, you die before you get to that point and that's the rule. Yeah, so that's like, yeah, I'll turn it to one. It would have to be a threat of a particular kind. So there are some risks that existential risks that might be pretty severe but are unlikely

34:42to affect everybody uniformly throughout the universe. So maybe, for example, in a pessimistic frame of mind, you could persuade yourself that humanity is likely to destroy itself because we will develop some very powerful weapon and then the world being what it is, we will fight some big war with it and that will be the end of it. And maybe that's a fairly likely way for us to come to an end. It's unclear, however, whether that would be a plausible way for the first alternative to be true because that requires it to be kind of close to universal failure of all civilizations at our stage throughout the universe to reach technological maturity. And you would take with some failure modality like war that there would be at least a few civilizations here and there that avoided that particular failure. Like maybe they would already have had one conqueror who conquered the whole world. So they wouldn't be fighting any wars anymore or maybe they would have evolved to be more peaceful or they would have created a sufficiently strong united nations like structure to abolish

35:48war. If there are millions of these civilizations, you would expect at least a few to have ended up in a better situation with respect to international conflict. So that would be an example of a risk that might be that are pretty big as in terms of its probability of being something that will destroy us but still pretty unlikely to be the explanation for why non reaches technological maturity throughout the universe. Now what does, all right so you think within philosophy you think about existential crises essentially and put up warning signs of what we might look for as we develop technology and put breaks on it and be careful what we do especially with artificial intelligence. Do you think we are at the point where we need to make those decisions now and that if we don't we are in grave danger or do you think we have time to change the mindset and start just thinking through our technological development or is it too late?

36:51Well, I mean it's not that either or we are I guess doing a little bit of thinking as it's vicious about our tech radical development. As a fraction of global resources though I gotta say it's a pretty small fraction that is devoted to trying to ensure our own future. We are pretty much living for the moment by the seat of our pants as they say. Yeah, but on the positive side there is a lot more quality adjusted thinking I would say these days than there were even 20 years ago. So this whole field of study I created this Future of Humanity Institute we were founded in 2005. I've been thinking about these things from before but I remember in the in the late 90s really what existed in terms of the infrastructure for trying to think through the implications of future technological developments. These kind of radically transformative technologies that post backs risk. More or less was like a couple of internet mailing lists. Some people hanging out

37:52there and chatting a little bit and there were a couple of institutions that were doubling but really very limited. Now you have a rapidly growing ecosystem of effective altruists, rationalists you have various academic institutes in different places you have specialized research labs focusing on AI safety within both some big tech companies and also as as nonprofits you have some big foundations who are funding this stuff and there's a lot of talent flowing into this field. AI alignment specifically and this broader kind of consideration of macro strategy and X-rays more generally. Still a very small fraction of global GDP but the growth rate has been significant. But yeah I think still we are more at the stage where one might be thinking about how to notch things on the margin rather than fantasizing about what kind of the ideal setup would be if one somehow could reform our global epistemic and governance structures

38:57from the ground up. One concept that actually what I think it dates from 2000s but one concept is the idea of the singularity, rake her swiles singularity. How do you think that affects this and what are your thoughts on the idea of a singularity? Do you think that that's how it's going to go or do you think that it's going to be a lot more messy and complicated than that? As a singularity I think could be quite messy and complicated if you zoom in. The term means different things but in one meaning it just means things happening really fast and so just because things happening very fast on a calendar timescale doesn't mean that that couldn't be a lot of stuff going on if you sort of zoom in. It might just be that all these complex things going on unfold more quickly than we are used to. So I do think it's likely that there will be a period of very rapid change coinciding roughly with the development of machine super intelligence. I don't think this is certain or not. It could also unfold over a somewhat longer period of time but relatively fast take-offs scenarios

40:03quite likely to make. Now traditionally there have been other components sort of rolled in to this concept of a singularity that I would want to break out and add an analyze separate. So some people have associated the singularity hypothesis with the claim that there would be some predictability horizon beyond which we cannot see the future becomes unknowable. It changes so fast it's remained and then we have no ability to extrapolate or predict what happens beyond that. I think that's quite a separate claim and we need a lot of qualification. And another component that was rolled into this I think particularly by Kurzweil was this notion of exponential growth. The idea that you can forecast when the singularity will occur and how it will unfold by plotting bunch of technological fields on log paper and seeing that you have an exponential there and that that's how things will continue. That may or may not be true in the relevant sense. I am a little

41:04bit skeptical but in any case it's a completely separate and independent claim from the claim that there will be some period in the future where progress is extremely fast. And so I think we need to disentangle these three different senses of singularity and then we can debate each one on its merits. We are barreling headlong into two technologies over the next few centuries anyway that seem to be infinitely dangerous more more so than nuclear weapons and one is molecular nanotechnology and the other is artificial intelligence. And one could even toss in human augmentation to make humans super intelligent because if we do that then we are no longer homo sapiens we are something else. So in other words homo sapiens goes extinct and you get something new you know the technological human. So what do you how do you view this do you do you think that these existential risks are what's going to end us was opposed to above and beyond things like climate change or the things we

42:07talk about right now in the present nuclear weapons whatever do you think that a great filter lies ahead of us with the technologies that are coming and are being developed rather rapidly especially artificial intelligence. Well if by great filter you mean something that possibly could explain the Fermi paradox that is some radically improbable step that hardly any civilization makes it through not even one in a million then probably not as for super intelligence is concerned it seems that even if we failed in that step what that would result in is on the line super intelligence that is super intelligence that doesn't care about humans or human values that wouldn't explain the Fermi paradox right because then that super intelligence itself might go out and do the colonizing and render itself visible to other civilizations out there. So even if super intelligence were a big existential risk it wouldn't be a great filter in that sense molecular nanotechnology I mean I think

43:08that is a significant source of existential risk it's a little harder to see how it would be an explanation for the Fermi paradox and the filter in that sense in I mean if for now other reason it's looks feasible to get super intelligence maybe before we get atomically precise manufacturing and nanotech and once you have super intelligence you might then like the super intelligence might then see the danger and avert it or or you achieve super intelligence in five minutes later you have molecular nanotechnology right but then then it would be maybe the super intelligence that would govern how that were used and if the super intelligence formed a singleton and it had enough savvy to foresee the consequences of certain ways of using the nanotech it could then steer clear of that risk it seems but still yes I think nanotechnology would be a source of some existential risk and synthetic biology too I think we should maybe add to that list of large existential risks I think depending on how you carve the cake up there might be other ways where you would get big

44:11slices I think for example a lot of existential risk arises ultimately from the possibility of conflict the fact that humanity is splintered into different opposing factions with now overarching reliable ways of resolving our global coordination problems and to produce global public goods and that manifests itself for example in militaries that we're spending a lot of money developing the capacity to kill the people in the other factions that then develop their own militaries to kill us and if you sort of zoom out and look at humanity from from the outside that that looks like a kind of tragedy a tragedy even if these weapons were never used and and and and multiply so if even when they are used we see global warming like it's a kind of failure to protect global commons overfishing we see it in many different ways that there are these costs to not having a reliable methods to resolve our differences particularly at the international level and so

45:11if you carve up the risks say risks by accident risks by conflict risks from nature on independent of human activity I think the risks from conflict would be a big chunk this episode is brought to you by Google Chrome you think you know a browser but Gemini and Chrome that's new it can help you with practically anything on the web like restoring a vintage motorcycle from a 50 page restoration block or finally break down that long article you've had open for weeks Gemini and Chrome is here for it ready to make anything online make sense there's no place like Chrome check responses set up require compatibility and availability very 16 plus Yamavab resort and casino at San Manuel is bringing the biggest laughs to the stage break taboos with Ali Wang on August 28th and 29th enjoy Ralph Barbosa's dry humor on September 18th and 19th and don't miss Nikki Glacier's unapologetic comedy November 19th tickets on sale now at Yamavata theater dot com only at Yamavab resort and casino celebrating its 40th anniversary you in must be 21 to enter

46:18now you mentioned something interesting adding to the list artificial biology now this is an existential threat that I don't think very many people think about or know about and it's the idea of creating say something like an artificial plant that out competes all natural plants and the natural plant goes extinct and eventually you end up with a completely artificial ecosystem made up of of creations in other words custom made plants because when you can think about genetics you cannot do nature and create superior plants essentially this is probably one that we're already dealing with don't you think another with GMO type crops and things like that for example corn pollinating all over the place you know I think a cornfield can populate for two miles radius so we're we may already be in that existential problem right yeah and and in the air the virus and so forth I think it's the field of synthetic biology is advancing so quickly and it's just a

47:19large space where creative scientists will find ways of doing all kinds of stuff and some small subset of those ways might be ways to do stuff that's extremely dangerous if we are lucky existential dangerous and and right now we don't really have any way to stare around those discoveries I have this in a different paper the vulnerable world hypothesis this metaphor of great containing balls representing possible inventions different technologies ideas that we might discover and human history as the process of reaching into this urn and pulling out one ball after another and so far the balls we have extracted have sort of been white balls or maybe various shades of gray they have been mostly positive and some technologies have been mixed blessings but we haven't so far pulled out the black ball from the urn like a discovery that is so dangerous that it destroys the civilization that discovers it but if there is a black ball in the urn then

48:21it looks like we will eventually pull it out if science and technology continues on a broad fund and whilst we have become quite good at pulling balls out of the urn we haven't really any capacity to put them back in we can invent things but we can't uninvent things so our strategy such as it is seems to be basically hope that there is no black ball in the urn and that's particularly true in synthetic biology there is not at all the same safety consciousness and culture as say there is a nuclear physics after Hiroshima and Nagasaki physicists and regulators realized in their gut that nuclear weapons were a dangerous thing it was not just fun creative science but they needed to be tied over site and controls the same is not yet the case for in biology and there the ethos is still putting a lot of emphasis on like democratizing access to science favoring open publication celebrating creativity and there is not that gut level awareness that some

49:23of what is discovered could turn out to be really dangerous and so I think that's an area where we need to level up. In other words always as what happened at the beginning of the nuclear age always look for the fallout before you said you set off the bomb itself and there was some very reckless activity in the development of nuclear weapons that that was just would not happen today but we only know about it through trial and error so in essence if we pull out the black ball from the urn we don't really have the luxury of trial and error it's either going to get us or not and the debate may not happen unless you have it before and you try to identify the black ball right? Right yeah now I mean I kind of suspect that the level of safety consciousness before and during the Manhattan product was was greater actually than it is today in synthetic biology that there were people I'm thinking Leo Zelard who was the first person to

50:27realize the possibility of a nuclear chain reaction who tried actively to suppress publication of relevant results with partial success and he was later persuaded to reach out to Einstein for them to gather to write the letter to President Roosevelt that then led to the instigation of the Manhattan product only because they fear that Nazi scientists might otherwise develop the bomb and get there first and so again it was this competitive dynamic that caused these people to override their concerns but in some areas of reality there is not even that it's not as if they are driven by this tragic fear of being up against ultimate evil that they must defeat in the form of a Nazi it's rather that it's kind of fun to discover things and sometimes thinking has not gone much farther than that. Now my last question for you and I'll get to that in a moment but I want to I want to turn to everybody's attention towards your homepage nickbostrom.com and there'll be a

51:29link in the description below by the last question for you Nick is all right so you think about existential crises things that could happen to us bad things and you as I said earlier you put up warning posts and tell us that maybe we should think about this before we take the leap which one scares you the most? Well that maybe is a super intelligence AI risk but it's a double edged sword well maybe that's not the good category because or metaphor since the sword is double edged sword is sort of dangerous on both sides but what I mean is that AI is also a source of great hope at the same time as it's a big fear I think if things go well with AI it could also be the solution to a lot of other existential risks and the key that unlocks a much bigger better future for humanity so I think this transition to this machine at El Nacer will be associated with great

52:30existential risk but ultimately it's a portal that we will need to and should pass through and our focus should not so much be on whether or not we should do it as more how can we put ourselves in the best possible position using the time remaining to maximize the chances that that this goes well right Nick we are out of time thanks for pairing with us I very much appreciate it and I look forward to reading more of your ideas because they are some of my favorites I like I said I love a good existential crisis and you definitely call attention to great many of them oh thank you for the conversation

54:29you Yamaha resort and casino at San Manuel is bringing the biggest laughs to the stage break taboos with Ali Wang on August 28th and 29th enjoy Ralph Barbosa's dry humor on September 18th and 19th and don't miss Nikki Glazers on apologetic comedy November 19th tickets on sale now at Yamaha theater dot com only a Yamaha resort and casino celebrating its 40th anniversary you in must be 21 to enter band rule is now available everywhere in the US so you can get in on the sports action no matter where you are including these places may this your most rewarding season ever with fan rule agent location restrictions apply product

55:32availability varies by market see fan rule dot com for eligibility gambling problem call 1-800-Gemma