Eörs Szathmáry - Major Evolutionary Transitions and AGI's Impact (Stewarding the Flame, Episode 7)

0:00This is Daniel Fajelli, you're tuned into the trajectory. This is the seventh episode of Stewarding the Flame, where we talk about what it would look like to encourage the flourishing of the greater living process of which humanity is a temporary part, which may be unfolding and unfurling beyond us. Our guest this week is Eor-Sothmari. He is one of the co-creators of the major transitions and evolution framework along with John Maynard Smith, who's no longer with us. Eor-Sothmari teaches at Hungary's Center for Ecological Research and is a leading theoretical biologist. The major transitions and evolution highlights how key jumps in the evolutionary process had to do not just with a new body plan or a new capability, but with things that used to be individual components, bubbling up to become larger collectives, which then act

1:04in the interest of that greater pattern. And as you'll discover in this particular episode, it may be the case that we're going through another one of those right now. There is no one with, I think, a cleaner view of what these pops in fundamental shifts of the regime of power and intelligence on planet Earth is beyond Eorce, given the nature of his research and the fact he's also studied AI. So I've got a lot of comments for the end of this episode. As usual, I'll save them for the end of this episode. Without further ado, let's fly into the meat and potatoes. This is Eor-Sothmari, here on the trajectory. Glad to be able to dive in here and get into this broader question of stewarding the flame. You and I have had some interesting philosophical discourse off the microphone, and now we get to dive right into it. But I want to make sure that the audience and even myself get to refine the initial view of an idea that you're one of the developers of that I think is rather important of major transitions in evolution.

2:05When you explain this to folks with different kinds of scientific backgrounds who may be unfamiliar with evolutionary biology, what's your tertiary definition that we're going to poke around and get a little bit more nuance on it? Evolution is arguably more than just one form of gene replacing another. It's more exciting than that. It's about the evolution of organizations and levels of organization throughout almost four billion years of evolution. And together with the late John Maynard Smith, we identified a number of really very serious steps as they were in the original and the evolution of life. Verbal number of things you can identify again and again. So one is that a higher level unit of evolution is being formed from lower level ones. The second important aspect, new ways of storing, using and transmitting hereditary information

3:14arises. And usually they go hand in hand. And then we identified something about eight major transitions in 95. There was a revision since then. But these two things are common things, you know, formation of higher level units and also lower ways of dealing with different kinds of hereditary information. And while you are doing this in evolution, you know, along the lines, the lineages where it happens, complexity does actually increase whatever sensible definition of complexity are using. It does actually increase along these lines. Yeah. And so a few of them, I guess, representative examples that you folks bring up is the formation of the single cell from presumably sort of wriggling RNA or proteins, whatever the case may be. The, you know, that's one example. And another is what we presumed through Margulis was maybe the bacterium that swam its

4:18way into a single cell and became a mitochondria. Now we have sort of this cyborgish, you know, eukaryotes going on. And then of course we have multicellularity. And you folks go so far as to say sort of, I think it was symbolic language. I don't remember if the economic connections tied into that are just symbolic language, but this kind of coordinated factor above humans is another one. Let me know what we want to clarify here. These are our very nice examples of the original list. I just want to add something because it was realized later by David Kweller in an awfully nice book review that actually there are two categories of these major transitions. One of them he called egalitarian, which means that the things that are coming together are unrelated, like in the case of the eukaryotic cell. They might convoy on the host cell, they were unrelated, you know, they were related ultimately, but you know, by any sense, they were unrelated.

5:22So coming together, where you are unrelated is the egalitarian transition. And the other one is the fraternal transition, which means that for example a cell divides and they just stick together, they remain together. And this is also how an end colon is being formed. So these are the two ways of how to do business, so to speak in the major transitions. Of course, if you concentrate on something like human societies with language, that was our last transition, which Ariya has a genetic background as well, that's kind of a mix. So then when you come to human things can complicate it, right? I was going to say, yeah. Well, and we're going to, so what I'd love to do is start to refine this. I would encourage everybody listening and just, you know, you can ask GPT, you can buy the book, you can go on Google, look up kind of major transitions and evolution, get a sense of these different strata. I think, do you folks go down to something about even the level of molecules?

6:24Because there's a part of me that wonders, how far do we go down? Like now maybe you would pretend to know, I don't. What atoms are actually made of, whatever that means, what quarks are even made of, whatever that means. Like I presume I have only as much access as I have. And so who knows what had to come together to form, you know, the nucleus of an atom or what have you? We don't know. And of course, I'm not asking us to necessarily speculate, but it seems that there's this hierarchical strata, which is much more than genetic that causes, again, new ways to transfer information, new variation, et cetera. How far low did your system get and do you presume it could go lower? Obviously you can go as the epicywish, but what is important for us is when you get molecules that, for example, can act as replicators, which means that they react with certain substances and then, for example, they can make a copy of themselves because we are biologists.

7:30So we are interested in this evolution phenomena for which you need a number of things. One of them is a way of multiplication. You need something like inheritance, which just means like, because it's like, right? Somehow. And the third one is variability, which means that heredity is not exact. Now if it's true that the heredity, among the hereditary traits, you have at least some that affect either your capacity to multiply or your capacity to survive, meaning not to die, then the combination of the two, multiplication and survival is what we call biological fitness, right? And that can already appear at the level of some molecules. This is the important thing. If you think about it, life is both ontologically, as well as epistemologically rooted in chemistry.

8:32That's very important. We don't need to go further down than chemistry, but chemistry is very important. We, you and I, also, we are in a way applied chemical systems, right? Sure. Yes, yes. Of course, there's a great many arguments of, you know, if you know where all the chemical billiard balls are moving, can you explain, you know, the evolution of, you know, the, the, the dung beetle or something? But yes, obviously chemistry, I think we, I think, safe to, safely to say, you know, we're constructed there from, I guess what this seems to imply is that from a transitions and evolution standpoint, things that have those three qualities you talked about, that seems to, at least from our present observable and conceivable view of the universe, that seems to start at the chemical level, in a molecular level. In other words, going below that, we would, we would presume maybe there's a lot of bumping and there's a lot of moving, but there may not be necessarily these, these, inheritability,

9:35et cetera, et cetera, sort of components. Is this, is this correct to presume? Except that that is the idea of cosmological natural selection. Oh, yes, yes, of course, that, oh, why are we small in and that's small in the idea. Yes, yes, yes, small in the idea. It would be a very interesting case at a really ground level. I mean, I, being a system, I, we wrote even a nice commentary about it for nature. So, yeah, I mean, the, the, the, the moral is, which is very exciting, that if you satisfy these conditions, then be it, you know, the, the, the universe, be it cultural items, be more like us ourselves, the Darwinian type of evolution can unfold. That's the, you know, what's fascinating to me, we had David Sloan Wilson on, who obviously applies the concept of evolution generously to all sorts of various and sundry phenomena. But sort of is extremely reticent to presume that it may ever bubble beyond humanity.

10:41As if sort of like our present level is sort of the totality, the culmination of the, the entirety of the process, he considers that, that to be speculative, but that everything else is evolution to not be speculative. I happen to be more in your camp. This may happen in a cosmological scale. People can look up Lee's, Molyn. This, this may happen at scales below molecules, but at least at our present conception, it seems as though molecules are a place worthy of starting where we can observe that these phenomena occur. Okay, great. A little bit more granularity here. And I think this is all going to tie back to AI and kind of the future with biological and non-biological, you know, intelligent systems. Stanley Salta has this sort of uses the term individuation. I don't presume he created it. There's probably some other person who kind of coined it, but the idea being that like, things have an impetus to persist. And Spinoza would call this the canatus, the inherent impetus to persist. Things have an impetus to persist.

11:41An individual algae cell has an impetus to persist. A flatworm, like most of the time, almost all of its cells are essentially serving the shape of the flatworm more than their own individual cells. In fact, some of them are born to be boiled alive, you know, and pumped through the digestive track so that they can help it eat it food, eat food or whatever the case may be. Just like there's cells in my stomach and on the soles of my feet, born to die every however many seconds. And so the thing perpetuated is a higher order thing. The canatus is a collective canatus locked in where, you know, we have cancer, we have some exceptions, but my cells don't leave and go hang out with another person they think are more attractive. My cells are pretty well perpetuating me. When I look at language in humans, I look at coordination, but not necessarily individuation, right? When a cell took in the mitochondria, higher order individuation, flatworm, higher order individuation, it strikes me that maybe human civilization, we are way more coordinated,

12:48but I do not feel as if, and it could be the case that I am part of the Austin, Texas higher individuated thing, but I travel to France. And I used to live in California and I used to live in Rhode Island. I'm not part of the Austin thing. What are your thoughts on this? Like where do you draw the line between individuation and major transitions? Well, that's an interesting question. What I would like to say is that I think that there was a phase in the origin of humans when actually small groups of humans acted very much like something like a cell. Actually, there is a guy in Sweden who calls it deliberately the social proto-cell model, which means that you have small groups of hunter-gatherers or something like that.

13:48And they obviously have their boundaries. They can be very hostile to the others. So there is competition among the groups. The identity is being formed by the cultural genome, as it were, to use it methodically. And it's documented also in chimpanzees, and probably it was also happening. When the group was growing beyond the surface, they just split. So there could have been a phase. I don't know. I'm just saying. It would have been a phase. It's very similar to what we are talking about, usually in the major transitions. But, you know, cultural level of humanism is far too complicated. You cannot squeeze it into this pro-Christian bed. Yeah. That's cool. Right? Yeah. As you had said, at higher levels of organization, they've become new evolutionary mechanisms. The way culture and ideas evolve is very different than the way individual species evolve.

14:52But it is nonetheless an entire strata, an inter-laced strata now of evolution on top of the other kinds that existed before. So, okay. This is important. We'll go into now, because I know we're going to be building up to some pretty big ideas around AI, and I think your thoughts there are going to be fascinating. But I want to get a last little bit of foundation. When you think about what life is, again, the premise of this series is that right now artificial general intelligence seems to be becoming increasingly more powerful all the time. I don't think it's necessarily true that within 10 years, you know, a huge bulk of the economic and military stuff happening in the world, or scientific stuff happening in the world, will be more purely machine than man. But it seems it is a non-zero chance. And for some people, it's very much likely, a likely chance. If that said, the question is, if we lose control of AI, is it going to, if we lose control of AI and then we're gone, and then something visits Earth in a billion years, is there

15:55a huge, sparklingly expanded ecosystem of power, intelligence, and access to nature beyond whatever the AI was from a billion years ago? Or do we get stuck at a local maximum? In other words, does the flame itself get squashed by some kind of brute optimizer that doesn't actually have auto-poises? Or does auto-poises carry through new substrates and new ways and continue to flourish as you and I are a kind of flourishment beyond the early algae, however many billions of years ago? And so with that said, we got to define life. And I think your definition is pretty unique here. So I want to sort of open up when you think about that thing we would want to preserve, the thing that bubbled up you and I, that seems to bubble up all forms of value and power that at least we know of. How do you define it? How would you want people to think about it? It's very important, I think, to become operational about it. When you ask about what is life in block letters, that's more of a philosophical question

16:57and a scientific question. A better question is, OK, what constitutes a living system? It's already a step in the direction of being sure operational. And if you go down that line, you will immediately realize that this question cannot be answered in general. One of the reasons is exactly what you mentioned. You are already consisting of units that are also alive, right? Which is not true for a bacterium. So obviously your organizational principles are part different from the organization principle of a bacterium. So for me as a biologist and also a person who is very much interested in the origin of life, I would like to ask the question, OK, what is the minimum level of complexity that you can call life? And we in Hungary are very fortunate because one of my mentors, Tiborganti, who was a chemical

18:02engineer by training, he came up with a very clear idea, right? There are three auto catalytic systems that you have to put together. Auto catalytic within chemistry means that there is a substance or a system that is catalyzing its own formation. It's absolutely basic. You don't have anything that is alive without chemical auto catalysis, right? So one auto catalytic system is metabolism, you know, that is, you know, turning the wheel. It has to be auto catalytic also and provides the material for the other two subsystems. There is an auto catalytic subsystem which can give you something like rudimentary genetics, right? We have a detailed information that you can change and then you can select upon. And then, since it's a chemical system, you have to have a bag which, you know, keeps the constituents together because otherwise they would just float apart.

19:05But I mean, if the other two are growing the bag also has to grow. So the way how to do it is actually also a kind of auto catalysis. The beauty of this system is that once you couple these three elements together in the right way, then you have a higher level system which is also a auto catalytic, but at the level of the whole system now and it can undergo spatial division and so on. My friends here at Kaufmann would call it as a beautiful example of a Kantian whole. Yes. So students recording in two weeks. Oh, well, okay. So you can begin talking about it. And I'm very familiar with his ideas of auto catalysis and I actually think his ideas are very compelling. Yes. Okay. So, but it's very interesting that that Stuart's idea about this auto catalytic set made out of proteins mind you. So it was a different idea. It is coming from 71 and Gandhi's original formulation of the chemotone.

20:10This is the name of the system. Okay. It is from 1971. So there was something in the side guys. It's an interesting thing that exactly and also the long fat paper by Manfredaig and about the hypersycle was also 1971. It's very strange. Yeah. So I'm with you here. Where you're getting at is life at an operational level and again, we're here. We're garnering a lot of different perspectives from, you know, a biology, neuroscience, et cetera, different people are going to have different takes. But I do think getting operational is clearly a crucial component of sort of nailing this, which is why I'm talking to people like yourself. This auto catalysis is an important part of it. It's sort of a chemical cycle that sort of produces the products that kick off or encourage another chemical cycle that can, and you get some interlocking gears here and things that are doing a very crude kind of persisting.

21:14And then how exactly that crude kind of persisting bubbles up to you and I, there is many, many mysteries. And we're going to talk about that in a second. But we presume that this crude kind of persisting is sort of the, one of the kicker offers of the a biogenesis, Shabang. So I guess it is auto catalytic. So what is life? We're getting to your operational view. Again, bearing in mind, there's people building systems. They're wondering if it's conscious, separate question. They're wondering, could it be life? Well, I think we can get operational. One of these things is it involves auto catalysis. It's capable of metabolizing. People talk about the crebs cycle. People talk about, you know, anything else we should know here and your conception of life in a way where we would know where it's instantiated in the world. One thing that I want to emphasize here, I remember the discussion with the audience when Gandhi was explaining his ideas and somebody asked the question, okay, so but what if all

22:19these processes that he considers to be important are not happening primarily in the chemical domain, but in some other domain, right? And for example, in the cybernetic domain. And his answer was yes, but I don't wish to call that life. But I agreed that it can satisfy this criteria at a higher level of suction. That was his answer. So you're already pointing to a potential non-substrate dependent conception of what life is. It could exist at different strata, so long as it fulfills those criteria. I actually think that's really the case, right? And this is why, I mean, you know it very well, that we wrote a paper about the threats of available AI and PNAS. And you know, we actually mentioned the major transition, that there is a major transition lurking in the probably near future, which is not entirely cheerful as something to look

23:27forward to. Well, I'm with you there. We're going to, so we'll get right into that. I think there's going to be a lot to unpack. So my opinion is that if cheer, you mentioned cheerful, if cheer requires the stasis, as they are, one should weep. There is no cause for cheer for the great process is moving quickly. The great process, which was previously moving so slowly that past generations could pretend there is no great process, the great process is now moving quickly. And unfortunately, there is, there's really, it will become quite obvious that becoming is much more real than being and that we are a becoming and if cheer requires stasis, then weeping is your proper response. I think rather it is better to accept the great process, contribute to and become part of the great process rather than weeping. But when we think about the transitions that had potentially with AGI, yes, you guys

24:29wrote a paper that I think is worth putting some of the core ideas on the table here around what sorts of transitions might be coming up. Lay out for us what you think would be most important for people to know, maybe people working within the AGI labs or people thinking seriously about general intelligence and how it could go. And when I say, by the way, when I talk about it going well or not well, in this case, and I know this is a little bit different, I'm talking about going well or not well for the great unfurling process for the flame itself, not for anyone towards. So give us a sense of kind of the summation of what you think is most important to share about the paper. And I've got some ideas I want to unpack about it because I think it's a lot of fun. Let me agree with a little bit about your fame metaphor. It's actually happened in a famous lecture by Minas Mis when he was discussing this basic phenomena that are important for evolution.

25:31And he said that if you look at flames, a flame does have metabolism, a flame can reproduce. So what else is needed? The flame doesn't have inheritance. The flame does not depend whether you were lighting it with a match or with somebody as it spreads. You cannot reconstruct anymore by looking at the structure of the flame, how it originated. So that's just a footnote to absolutely. Well, let me really clear here. I say flame because it is a very swift metaphor to understanding that like torches are always temporary. So you understand this, most people don't. They presume that forms are eternal, that persistence is not a matter of degrees and it is.

26:31And so I use torches versus flame to say, yeah, it's just metaphorical and it is an ode to Heraclitus. I was just, how should I say, teasing you with this? Yeah, yeah. Okay, I'm certainly not literal about it. I think we do need to get more granular. But it's cool to know that Maynard Smith actually had touched on that a little bit himself. I was totally unaware of that. There is something that I wish to emphasize. So the question is whether we get to the stage where different forms of AI can form avoid you to reunite with the same capacities that I discussed. The multiplication in heritance and the variability. And we are very close to that. That's very much detailed in the paper. And it can happen anytime. But there is one thing I want to emphasize here.

27:32I think so afraid about artificial general intelligence, right? Or consciousness of AI. That's more of a hype. That's not the real threat. I mean, let me explain to you why I said that. I mean, if I'm giving a lecture in an audience, right? And there are 40 people sitting there. I use this example also in front of them. I am reasonably confident that most of them are conscious. But for some reason, I'm not afraid all the time. I can assure you that by any rate, the COVID virus is not conscious, right? But I'm very much afraid, the cause. So the real threat is the availability, right? I completely agree with you. So the moral weight, I think, is a worthy question for a separate podcast with other people. But the threat is indeed the expansion of powers, experience, and access to nature.

28:34Yes. I'll give you, if I may, a biological example, because it's very telling, right? There is a parasite, which is called the Lancet Lever Fluke. The Lancet Lever Fluke is, you know, upsetting cows, for example, because it causes, you know, the inflammation of the liver and so on. You know, it's an osteoporosite. It has a complicated life cycle. It has several intermediate hosts. And one intermediate host is the end, right? So the snails are spitting out, you know, a little propagusive of this creature, and the ends are eating it. Now what this thing does when it gets into the end is the following. It forces the end to crawl to the top of the grass and just stay there, right? Why?

29:35Because if an end is on the top of the grass, it's much more likely to be swallowed by a cow, right? Okay. The point is that this ridiculous, this simple creature to reach in comparison and end is like Einstein, right? Nevertheless, this little nothing can force a much more intelligent creature, you know, to do what actually maximizes its sound fitness. That is what people have to take into account. So you don't have to, you don't have to have a complete LLM to be an ever-user unit. It's enough if there are other forms of AI that actually take advantage of an LLM that is being there. Is there interest enough? Yeah, yeah, yeah. So some people say, well, it would need to have robot bodies and it would need to like control spaceships what you're saying is if it had evolutionary pressures and it could,

30:38you know, the COVID virus is presumably not sentient and making big plans. But if it has some physical instantiation or digital instantiations, it could start doing things that impinge on our world, even if it's not terrifyingly intelligent yet. Every user is very opportunistic. Yeah. That's one thing. And this is why Agent TKI scares me especially, right? Because Agent TKI can do things, right? It has sort of big arms and legs, right? It can actually manipulate, you know, in the cyber world which was impossible before Agent TKI. That's one thing. The other thing is if people have read the selfish gene by Richard Dawkins, you know, it's obvious that once the Darwinian dynamics unfolds, selfishness is an algorithmic consequence. Of course. And this is where the threat is. It's persistence. Spinoza has the idea of the conatus.

31:42The conatus, the inherent impetus to persist for the flatworm, it's all the cells of the flatworm, for the algae, but you're saying the AI develops the same. It will want to do what behooves its own persistence naturally. And mind you, just four years after the origin of species which was in 1859, Samuel Butler actually wrote about making very fascinating. It wrote about this, you know, and it actually said that, okay, we are going to have reproducing and available machines and probably they will take over. I mean, he was, he was Ray Kurzweil a hundred years before Ray Kurzweil. Yeah, exactly. And then, you know, now, questions, how you, how you attend to this thing. I just stumbled on a quote by Arthur C. Clark from a 1964 interview when he said, look, I mean, we superseded, you know, from where actors, we superseded Kroman, you man, right?

32:47So it could very well be that we will be superseded by creatures that we are creating. And he said, we should be proud of that. Yeah. That's our give, right? Yeah, yeah, I'm with you. Well, look, I mean, here's the thing, the big picture OMG, there's going to be stuff beyond us isn't that terrible. That's not even what we do here. So like, this is the only show where the, where the victory we speak of is the victory of the flame itself. Because this show presumes the great process where all torches are temporary. That's what this show presumes. And so I'm not hoping for an eternal, hominid kingdom. I'm hoping for a flourishment of the great process of which we may be part for how long I know not. And so that, that's what the show is about. So the, the morning of homo sapiens, sapiens, I'm not saying it's insignificant, but it's not, it's not really what we do here. We do hear is talk about the flourishment. So what I want to get at with your idea is this idea of a genticae, I that starts to creep

33:51into these different domains and could blossom into things beyond us, et cetera, et cetera. I wonder, like, would you presume, because I can imagine two versions of the world that you articulate from the paper, two versions, one version, it gets creepy crawly into our laptops, it gets creepy crawly into our smartphones and our smart washing machines and whatever. And it gets creepy crawly, new permutations who zip around and share their own code and start doing things and, and, and are, you know, again, you have this, you, you called it this little nothing compared to which the ant is Einstein and it can control it, right? So, and so the, some of these AIs are not a little nothing. They're going to be what, they're, they're going to be what we are to the lemurs. And so, do there's this huge ecosystem of embodied and virtual, sort of systems, competing in strata and thinking faster than us in all these different ways. I could see two versions of how this potentially pans out. One version is it creeps, it crawls, it creeps, it crawls.

34:54Maybe we become, you know, more cyborg and there's some symbiosis. We're going to talk about that a little bit today. Maybe there's some of that. And then eventually it's mostly nonbiological and then eventually it escapes into the femtoscale. It escapes to other dimensions. It figures out how to move faster than light. It does a bunch of things as far beyond my conception in yours as the human global economy is beyond the conception of the ant that you talked about who bites the top of the grass. Okay, so that's one version. That's the version that says basically if something can trample hominids in terms of just out competing, pushing us out of sort of the main playing field here, which is what evolution always does. It's overcoming itself. People look at, people look out at nature and they say, oh, everything's in balance. No, it's not. You could go back to nature two billion years ago. It's much less than that. It's overcoming itself. It's overcoming itself. Balance. So Heraclitus says strife is the father of all. He says, he says, like it renews it or like it rests by changing.

35:59Like the whole thing is actually bubbling beyond itself all the time. There is no stasis. There's no happy balance. We're teetering on the edge of entropy. We just let people in doubt. I mean, they've been spectacular disasters, right? I mean, when the blue green algae discovered oxygenic photosynthesis, that was an incredibly strong poison for most of the creatures at the time. Yes. Then, of course, so the great oxygenation event, so we're going to get into that. You're reading my mind. It's crazy. The great oxygenation event, of course, we have the snowball earth for hundreds of millions of years from there, right? It's hypothetically possible that that could have squelched the whole living process. We, some cell figured out how to get free electrons from the sun versus these chemical processes, 10 to 30 times more energy efficiency, popped off, changed the environment. People can Google the great oxygenation event, massive catastrophic effect could have crushed

37:00earth life, potentially could have crushed earth life. So there's a version of the world where AI just kind of competes, how competes, how competes. And all versions of that will, with enough time, we'll just flourish into different parts of the galaxy or the nature or whatever. There's another version of that where it spins up, it spins up, it spins up, and it does a kind of great mechanization event, like the great oxygenation event, but a great mechanization event where it does something to earth where it sets life actually, what appears to be backwards, the auto poetic process, the expansion of powers, experience, access and nature, that set backwards, and maybe crushes it so much that we almost get to a start from scratch, weird place where the whole system is mucky and gunky, and the AI stuff can only persist for so long, and solar flares, and bio life has to creep and crawl out of a new atmosphere that's been totally changed by these crazy machines, and life goes backwards,

38:02not forwards. So the concern here on this show, now you can go on a thousand shows and talk about the eternal persistence of man and how man always wants to persist, that's just not here. Here we're talking about the flourishment of the great process. Do you think we're almost destined for an inevitable grand expansion, or do you think there are versions where the entire process of which we are part could be set backwards? And if so, what would those situations be? That's an interesting question, and I wouldn't like to rule out any of those scenarios, because I don't know, right? How to rule any of those. But I actually would like to say that curious enough what you are saying is linked to something like Fermi's paradox. Yeah? Yeah. Yeah? There was a lunch, I think at Princeton, where Enrico Fermi and also probably Leo Cilar and some other people were sitting, and then Fermi raised the following question.

39:09I mean, do you believe in evolution? Yes. Do you believe that mankind is its intelligence is a result of evolution? Do you think that evolution would happen also at other places at the galaxy? Yes. Do you think that some of them could be much more advanced than us in principle? Yes. So where are they? That's Fermi's paradox. So the fact that we don't see not only other living beings around, but not even other artificial living beings, might be that implosion is the right outcome. I guess what you're getting at is that there could be a way for the expanse of these powers. Spinoza calls it potentia. Potentia is the total set of powers that permit a thing to persist to be who its canadace. So it's potentia creeps and crawls, who knew substrates, new powers, new strata of nature. It could cause crushing implosions, as has happened on Earth a couple times.

40:09And so I guess what you're getting at is that could be the answer for Fermi's paradox. There are folks like John Smart, who suspect the potentially the transcension hypothesis where at a certain level of technology things actually escape into like the femtoscale or other dimensions. And just they don't have that much interest in what's going on here per se. That could be a possibility. It's a possibility as well. But I would agree it is still a paradox. We don't really know. And so I guess let me ask you this, knowing that agentic AI is developing swiftly. And I think some people in biology that just kind of laugh off these ideas of AI ever being intelligent. I think you know, you admirably are quite concerned with how quickly it's going. My concern is is it leading to a net boon in the expanse of the living process that seems to undergird all value. Everything I could think of that would be valuable would seem to bubble forth from things that are alive. Terence Deacon would say like there isn't meaning.

41:10Something isn't good or bad for, you know, this pen presumably, but things are good or bad for things that wish to persist that are alive. And so meaning starts there. So if all conceivable value is undergirded by the living process, what is conducive to the thrive of the entire process of which I am merely part? So if you look at those trajectories, are there ways of agentic AI spinning into the world that you think are maybe more conducive to collapse? Are there ways of merging with the technology and understanding it or something that you think could be more encouraging of the greater blaze of the process? Like what would make the big picture go well or not well at this stage of AI? That is a complex question and I think it's ultimately, I cannot say anything about it. It's an empirical thing. I mean, you should find out I am afraid. Yeah, we're going to find out the hard way. Yeah, exactly. So, let me make a little digression here because I'm just working on something right now

42:16which relates to it. So, in 1984 there was an extreme in unorthodox, several-user model for, you know, the global ecosystem as it were by near-stance and may not Smith. And I don't want to go into the details because it would be too technical but it turned out that the model actually had two interesting outcomes, possible outcomes. One of the outcomes was that you get to a stage where all the different species that you have, everybody is at its fitness peak, right? And the species number is maximum and therefore you should practically come to a halt, right? And the other one was, that was the stasis, right?

43:20And the other one was the red-clean type of thing when the species number was intermediate and there was a continuously placement of species by species, right? So you can ask the same question about the evolution of AI, how will it work out and how will it work out in relation to the biota? Well, nobody knows but I want to say that there is a serious danger, right? Because I want to have this idea of pre-coction, right? So when you are talking about, you know, for example, biodiversity loss, right? People say that in cases when you don't actually understand the details, but you do understand there is a threat, you just cannot gauge the probability of the threat, you would better be super careful, be exactly because you don't know whether you are five

44:24centimeters from the abyss or ten kilometers from the abyss. But if you fall down, it's too late. Yeah, yeah. This is why I'm really pleading for this pre-coction with the available eye, because I mean, it can really screw us royal. Scruing us royal seems to me somewhat likely, although I think there are merger scenarios. So us eventually being superseded by something non-biological strikes me as somewhat likely and long term potentially good. It could also potentially screw rile you talked about the biota. We go beyond the biota to all that lives, the entire flame, any substrate, whether it involves guanine or not. I'm not a guanine supremacist, I'm not a compute supremacist, I think the flame matters more than guanine or computers. And so it could screw us, sure, it might also screw the whole process of which were part.

45:27The project that I am grateful for the proto-reptiles and the flatworms from whence I came. I'm grateful to them. And I think there's an unfurling. I experience a greater power experience and access to nature than many of those animals did. And I think that that process could continue. There's a screwing of the whole process. And I concur with you. The way I explain the precautionary idea in this context is I know relatively well that the biosphere and humans in particular are pretty bright burning flames writ large. Like per human skull, we got a lot of flame per human skull and per human body. Human is more than just a brain. Lots of flame going on, I presume. And in nature, I see a tremendous amount of it. I don't, what I don't want to do is take this torch that carries a lot of flame and say, you know what? The biosphere's got it. I think the computers have it and let it go from my hands and let it land on the computers

46:29and just presume that stuff is going to carry the flame. I don't know if it does. It might just be a brute optimizer that crushes us and crushes Earth life. It might not be carrying the auto-poetic process of which we're proud of being part in which is expanding. So I'm with you. I say, before we let it rip, we really have to understand. Number one are there ways to kind of merge and symbiotize. We'll talk about that. But number two, is it going to be carrying forth that which undergirds all value? The living process. Is it a perpetrator and expander of that or not? And my guess right now is it may not be and we should not let it run out of control rampantly. It seems like you're somewhat of the same belief. If this ever-using process unfolds, then many different kinds of things happen, right? All kinds of things, right? I mean, there will be nasty creatures. There will be more lovable creatures vis-à-vis each other, but also vis-à-vis humans.

47:34Okay, it will be a huge, co-evoluinary ecosystem that is going on. Right? But there is a problem because one of the threats here is that the generation time for available AI is ridiculous this small. You know, a bacterium under optimal condition needs 20 minutes to reproduce, right? I mean, algorithms can reproduce in a fraction of a second. Here's a way I think about it. I say, you know, again, what's the genetic difference between a chimpanzee human, you know, whatever. There's different ways of measuring these, what difference is. But like, whatever. Call it 7%, or something, call it 3%. Whatever you want to draw the line about whatever categories of differences, it's not oodles. It's not oodles. So, you know, we go up that genetic difference. The impact of chimps on their environment is, you know, they eat the fruits, and they get rid of certain smaller creatures that compete and eat the same fruits.

48:35And like, what's the other chimps? In other chimps, sure, sure. And so like, the extended phenotype to pull in some Dawkins terminology of a chimps is like, if you look at the environment to say our chimps around, we kind of know because maybe there's chimps, there's chimps battles, there's less of certain kind of fruits or whatever, right? Humans, a couple genetic points of difference. The sky is full of space debris, thousands of tons of space debris, okay. The oceans are almost all plastic. Like, like, almost all mammals are living in boxes like the size of their body waiting to be slaughtered. Like, the extended phenotype of man, now here's the thing. We've evolved at incredibly slow Darwinian speeds. What's in our skull hasn't changed monumentally in the last 50,000 years, or some people would say 200,000 years. And so if these systems are doubling in their intelligence and power and capability every, you know, day, month doesn't matter.

49:36If it's anything faster than Darwinian speeds in a appreciable degree, what kind of environmental changes could be conjured from a system like that? Exactly. Look, I mean, it's very interesting that people in the social sciences tend to not observe or underestimate what kind of societal changes very simple technological inventions have brought about. I mean, it's actually very instructive to look into the details. If you consider two things, the steam engine and the so-called centrifugal regulator, the centrifugal regulator, you know, this is classical thing that doesn't allow the steam engine to blow up and doesn't allow it to stop, right? So it's regulating itself. These two things have a little profound change in society. Now what AI is bringing will be much more profound.

50:39Well, and this, we're going to last couple short questions here, but I really do want to garner your unique thoughts on this. Some people would say, and I'm not going to seed too many ideas because then I'm not going to get yours. But I do need to put enough on the table for you to play with. So some people would say, for the extension of our own relevance and well-being from a torch perspective, hominid perspective, and for the flourishment of the greater process which maybe we could contribute to and encourage the flourishment of the flame itself. In both cases, we should hope for some degree of brain computer interface, of symbiosis, of being a participant actively in these systems, in the development of sciences or whatever the new economy is, et cetera, for as long as we can, because maybe we can make sure that some parts of our flame are able to manifest and persist. Maybe a cyborg human could understand the living process better than you and I can and ensure that that goes on, but also because survival, if you just sit here like a human with no brain

51:43augmentation, not using AI, survival chances may be very low. So selfishly and self-lessly for the greater process, we should aim at some kind of mergers and biosis. What would you say to that and what kind of scenarios do you hope for? Well, I think that conscious attempts should be made in that direction. So we don't want, I mean, let me put it like this, we don't want the AI replaces. We would like the AI to cooperate and to help us, right? Ideally, right? And one of the things that people should actively pursue is that the usage of AI should be such that humans remain in the driver's seat, right? Yeah. Of course. Now, of course, when and if AI becomes truly available, that's a problem because there

52:44will be all kinds of AI around who don't want to participate in this living happily ever after type of fairytale, right? And that's, this is why I say that the availability is a very crucial stepping stone in this story and it might actually lead to the, to the, either to the domination of the humans by AI, right? I mean, look, look, AI looking at us as we are looking at, you know, animals basically. Yeah. Or actually, I mean, if there are, is a very strong conflict of interest, I mean, AI might actually say, okay, let's get rid of this problem. Totally. Yeah, could totally happen. Well, partly, partly. I am not joking, partly to the benefit of the rest of the biota. Yeah, yeah. Well, presuming, presuming it cares about the biota, I don't know if it will. Well, it could, you know, it could, right?

53:44As far as it doesn't disturb, we had, I think for AI in the long run, we had disturbed AI much more than lions, right? Oh, certainly, but I guess here's my guess. I wouldn't see a vast blossoming of super expansive, super intelligence. And then a bunch of lions and giraffes roaming the open plains. Here's why I wouldn't presume that. I would presume they would be doing things with all of the matter and energy that they have access to and things beyond matter and energy for which you and I have no conception as me and enjoy it. And that would be a f**king ripping open of reality in ways that would not presumably, you know, the main thing, all the dumb beetles and the daffodils just as they are. So I don't presume the biota would survive the grand expanse of the unraveling process. Not the biosphere as it is today. Yeah. So what I just want to say is that one ought to be very careful, right?

54:46Yeah. And not just to write code blindly, right? Yeah. I mean, given these risks, I mean, your thought about, I mean, even if let's just say AI, humans stay in the driver's seat, right? You know, this again, a very torch centric moral position here, like, but I guess in the near term, again, we don't want to run away AI for all sorts of reasons, but I guess I would wonder, let's say humans do say in the driver's seat, many, many humans will want more happiness or more memory or more cognitive ability or they'll want a new bionic arm or bionic eyeballs or a second robot body that they can control with some other part of their brain that we don't have right now that could be plugged in as prosthesis. It strikes me that the humans in control are a faster moving, evolving process than Darwinian. We say humans in control, ah, back to Darwinian speed. Thank God. I don't think so. No, no, no, no, obviously. I mean, that's the category that, you know, my famous and the favorite science fiction

55:49writer, Stanislav Len. Yeah. I mean, just look at what he wrote. I mean, he was talking about these things in the 60s and 70s, right? So present credit where it is you, right? And then we made a very strong point about this auto evolution potential, right? And that I think that that is inevitable, right? So even if even if humans don't go extinct, right, they will change tremendously. Yes, that's those are the only two options. You have attenuation or transformation. There are no other options. That's all you have. Why would anybody think we have anything else? And so, but I think historically we've been able to ignore the great process, but now it's going to, it's not, but it's not. It's not going on your door and on your neck. Exactly. Well, you brought up a good point here. So in the meantime, so it sounds like in the meantime, you would say, okay, let's, like,

56:52would you think it's net good? Like let's say we prevent, run away artificial intelligence that just crushes the earth and could do the great mechanization event and set life backwards really drastically. Let's just say we, we, we avoid that somehow, international coordination, whatever the mechanisms are, we prevent that. And would you say that the mandate would be we should aim to more ardently understand life or ourselves. We should aim to carefully in a way that might be governed, level up and augment ourselves in a way that doesn't seem like it's going to cause some war with different kinds of brain augmented cyborgs, but who we couldn't predict their behaviors. Like what would be the careful steps from there in our part in the greater process? If we could stop the runaway AI, which I think you and I agree would be a pretty bad idea today. What would you hope we do from there if we got there?

57:53I, I, I'm, I'm not entirely with you, but let me say something. Right. That as these other major problems that we are facing and that includes folks, for example, the problem of climate change, it, it actually, it, it includes international conflict, it includes, you know, the, the, the decline of the biodiversity and all these things, all these things, they may not have a solution at all, which is an interesting question to ponder about, right? But if there is one, I'm sure about one thing, it needs tight international collaboration, there will be either a collaborative solution or there won't be any. Like that is one degree. I completely agree. And that's for AI, but you know, the, the players, the human players are not a good guarantee

58:57that such an agreement we'll ever come about. What I'm seeing is everybody is racing like crazy, right? Because we have to, we have to compete with the Chinese. We have to compete with the Americas. Yes. I mean, you know, this is how things can crash. Yes. Well, also, so look, I'm completely with you. We've had entire series on this show with Benjiro and Stuart Russell and former senators and high level people from the Department of Defense all talking about it. I've been out to Shanghai. I've talked to the guy who's running a sort of Dean of Tsinghua, Juelan, all talking about different kinds of knitted intergovernmental coordination ideas that could potentially put up buffers against arms race acceleration dynamics. Because I think, and some people say, well, that's against evolution. We're trying to do some evil government thing that's going to be a tyrant. That's against evolution.

59:57I would say evolution expands its powers and then seems to find ways to bound them. Humans got big brains, but in order to participate in society, we had to stop being like murderous and inherently individually selfish. That's what we have. High level organization always suppresses the evolution of potential at the lower level. Otherwise, you would have cancer all the time. Of course, of course, but then it opens up potential at the new vanguard, which is even greater. And so, I think that's too cool for me to say that. It's too cool for me to say that they're enabling constraints. Yes, yes, yes, yes. So, Spinoza has this idea of protest, which is different that Potentia is the expansion of powers. Potestia is the selective bounding of them. Both done well. Potestos increases the net potential of the system. And so, enabling constraints is a beautiful way to pull it out. I'll have to tell Kaufman that you quoted him there.

1:00:59But, yeah, I think we absolutely need that. And on this show, in many places in the physical world, the United Nations, the OECD, places where I'm involved, I'm actively advocating for these things and know many of the people that are pushing this kind of intergovernmental dialogue forward. So, I'm with you there. Let's say we get there. And I know it's going to be challenging. But let us say that we arrive there. We have an imperfect, so is imperfect, some imperfect kind of coordination between the US and China. And now, we got this really powerful tools. Would from there, if we wanted to be part of the greater process of participant in change, like you had said, transformation or attenuation, it's one or the other. If we want to think about how to positively transform, what do we do? We prevent the runaway AI for at least a little while through some kind of coordination. What for you would be next steps? I'd love to kind of be able to close on some of your thoughts here around what you would hope humanity could do. Should we get to that first stage?

1:02:00I am somewhat reluctant to make promises here, obviously, because it's so really complicated that, you know, and it's happening so fast. So it's very difficult to say, what I can say, however, that if there are solutions about which everybody or most of the players agree, it will be through that several of those developments will come from completely unexpected corners. Yeah, that's the thing that I'm sure about. Yeah, okay. Certain of the unpredictable, it's coming. I guess the way I think about it, and again, we can end on this, but I would guess whatever you and I can predict is not going to be how it goes. But I guess what I'm getting at is, internationally, with regards to AGI, it seems in this kind

1:03:03of development of intelligence. It seems obvious. Red lines will be step one. I was at United Nations headquarters in September of this year. We ran a worthy successor event, you know, right, right outside of the event in Manhattan during the same exact period of time. We had a bunch of people who are at the UN there. They talked about red lines, international red lines, things we could prevent both in China and the US, and there were delegates from different countries. And we're not there yet, but we're talking about it. I suspect if we achieve red lines, there will have to be green arrows. And it's to say, okay, we figured out how to bound, but things are still changing. Where do we go? Yeah, sure, sure, sure. But that's going to be a very tedious and cumbersome process. And let me tell you something, because there is an analogy here, you know, trying to control this kind of available AI going rogue. There is an analogy here with the emerging infectious diseases in biology, right?

1:04:06I mean, it's the same kind of thing, but this emerging infectious disease is actually happening in the cyber world, right? So there is an end. So the question is, what can you do with the emerging infectious diseases? Can you actually anticipate in order to mitigate the risk? Yes, you can. Actually, there is a plan for this. It's called the Stockholm paradigm. It's worse by reading. And I am advocating that something like the Stockholm paradigm should be taken seriously by the developers and the users of different kinds of available AI. It's never absolutely safe, right? It's mitigation, right? I mean, people allegedly have escaped even from the Alcatraz. And some of them might have survived, we don't know, right? So that shows you that these things, control can never be absolute, but it matters a lot

1:05:07whether a disaster happens every day or a disaster happens every second year. It's a huge difference, right? Yes, yes. Yes. And again, governance is its own entire series, you know, here from the fathers of AI as a field to the policymakers who are trying to push it into the world. So we're very much on the same page there and there's a lot to be explored. I think the Stockholm paradigm you mentioned sounds like it's something exciting for people to Google. As we, I guess as we close out, any parting notes, you know, there's people within the big labs that are sort of building this technology now. Most of it is in an arms race dynamic. And I think most of them come more from a computer science background. These are often rather brilliant people, but they come from a, if I write the right code, then it's a tool for humanity, yada yada. That's the entirety of the paradigm, not sort of the hierarchies of different kinds of life, which is your background.

1:06:08If there's sort of parting advisor thoughts for that crowd, assuming that you would want to see sure humans have a good shake for however long we can get it, but the greater process continue to unfold rather than get squashed by some runaway AI virus. What sorts of advisor ideas would you want to stick in the brain as some of the folks building the tech as we leave today? Well, I think that they, well, I mean, it's very clear. I think they actually should learn more about society, history and biology. I actually mean it, right? But this kind of knowledge gives you the perspective that we have been talking about. If you don't know about these things and or if you don't care for these things, then I mean, we were very easily end up in something which is a horrible disaster. Yeah.

1:07:09Well, I'm a little bit biased that I think maybe they could read some of your work. And I hope even if they only read a little bit of it, at least some of them will watch this interview. You know, we've gone a little bit over time. You've got a lovely lake out there in central Europe to go swimming in, but I'm so glad that across the pond, you and I were able to have this discussion today or it was a real pleasure to have you here. Thank you very much. I have so good to enjoy it. So that's all for this episode of the trajectory of Big Thank you to yours for being able to hang with us here and take the call from the other side of the pond right after this interview. I'm going to be swimming in some big Hungarian lake somewhere. So we had to make sure to end on time to get him out there for his summer, summer recreation. It was a really, really fun conversation to be able to unpack with him. And I'm grateful for you for listening all the way through to the end of this episode. As usual, I'm going to share some of my notes of what I picked up on from the interview itself and maybe some of the insights and some of the tensions that showed up and just ideas that sort of bubbled out of this discussion with yours.

1:08:12A few things. And he mentioned pretty frankly here that evolveability is even more of the danger than intelligence that there are ways for AI systems which you can evolve so rapidly. If that development were sort of autonomous and could happen at machine scale, that rapid proliferation, almost like a digital locusts, digital fleas and locusts and rodents and whatever else could be outlandishly powerful in harming kind of current human civilization or maybe even the biosphere, just as sort of something aggregately vastly more powerful than humans could be that even just evolveability itself, run rampant in the digital sphere could be dangerous. I think that's a valid point. I haven't heard really anybody put a finger on it. Something I'll probably have to journal on and hopefully write about at some point. But I think there's a lot to be said of it.

1:09:14He also talks about life quite frankly as likely on some level being substrate independent. He had mentioned systems that have some kind of a boundary, systems that have some kind of metabolism that are able to take in matter and energy to maintain themselves and maybe change themselves and have also some degree of inheritance. And that this may happen at different collective organizations of pieces, components and parts that could be partially biological or eventually entirely non-biological and that that unrolling process of change. And we see these knitted patterns that make up life, you know, all of your individual cells building up to you, individual humans building up to a civilization that this easily could flourish in the way that life does, potentially in a sentient way, blossoming new power, experience and access to nature in the world, but not biologically. He does grant that in fact that may be the case. That says he had a very, he's very skeptical that if we just let her rip with AI, we're

1:10:16necessarily going to see a flourishing living process. We might just the same see all kinds of interesting digital parasitism or kind of the digital locus type scenarios, which might just be a general detriment to human civ and to sort of the present regime of intelligence on the planet without necessarily blossoming into anything bigger. He was very much, much, much more aware of the crumbling of what is now our regime, then of the blossoming of what the next one was repeatedly through this conversation, sort of like, and what might it look like to sort of get, you know, get things to the next level. And there was a lot of resid reticence to moving in that direction, but a lot of frankness around the fact that even if life be substrate independent, there's probably a great many ways that this could go wrong. And then we might not want that to occur. And I'll touch more on that in a bit. We talked a good deal about the idea of something that echoes potensia and protestos, these

1:11:20are ideas we talk about frequently here. The notion that at a new level of coordination, the units that make up the new coordinated entity, for example, the cells that make up my thumb are not really maintaining their own pattern, unless they be cancer, they be maintaining the pattern of me as part of this entire connected unit. There is some giving up of freedom, but there's also some benefit of greater collective power gained by smaller units that add up to bigger units, this tied into the governance idea in this conversation of potentially having some degree of coordination on a global sense to prevent runaway rampant evolutionary digital processes and AGI, which might allow us to have more stability to decide new trajectories into the future. Again, he was reticent to get into what those trajectories would be and very much more excited to talk about how to create the red lines.

1:12:21We'll talk about red lines, green arrows in a second. He also admitted rather frankly that human, what he called human auto evolution, is inevitable. In other words, if AGI doesn't wipe us out, we're going to change ourselves radically. I think that this is a frank and honest perspective. Again, this is someone who studied the major transitions and evolutions, very, very hard for Earsh to sort of say, well, if we just stop the AI, we can keep the future human forever. Surely he's not naive like that. He's not him. Very, very hard to look sternly at what life is and to conclude some kind of stasis. He thinks even if we get things right, we ourselves are going to evolve in bubble. Again, he was just reticent to talk about the direction that would go in. Rit large, big takeaway in terms of differences of focus in this episode. He was not necessarily saying we should keep the future human forever. He was not necessarily saying if we can stop AI, we won't evolve. In fact, he said the opposite.

1:13:22He was not necessarily saying AI can't be part of the expanding, flourishing living process of which homo sapiens is temporarily an important part. He didn't say any of that. He was open to the whole kitten caboodle of the greater flame. But his focus, and this is understandable, was on the red lines. I'm going to put up on screen here, the visual image from red lines, green arrows. We sort of talk about the trajectory we're likely to go through, civilizationally, in my opinion, if things are to go well. There's some people who say any kind of governance and coordination is against evolution. It's against evolution. Metatransitions and evolution is precisely about new levels of coordination and bounding, which create new magazines of power and opportunity for competition and cooperation at higher levels. If you are an accelerationist, you are very interested. Evolution levels of coordination and bounding of certain powers that permit for a greater

1:14:22set of powers and coordinability of the net system. You are very, very interested in that. Accelerationism that says no laws in all direction, laissez-faire. Again, everybody's body would just be made entirely of cancer if that were true. There is some subordination and different kinds of patterns and bounding and binding that permit systems to not collapse and permit systems to open up into new collectives capable of, again, new entire magazines of access to nature and of competition and coordination. If we look at red lines, green arrows, the red line realization is this stuff could be dangerous. Oh my goodness, we need some coordination to stop it from doing all this crazy stuff. That's where Eorsh was in this interview. Obviously, understandably so, by the way. I mean, I think that's kind of where we should be. At the time of this recording, I will be at UN headquarters in less than a month, talking in the room as sort of red lines are brought up again as they were last year when I was

1:15:25in the room with Benjiro and others at headquarters. And I think internationally, as you I governance is crucial. That said, there's a second realization, even if we get red lines correct, here's what that realization is. Shucks. I guess even if AI doesn't care. Kill us all. Geez, things are going to change a lot. We're going to change the regime of intelligence on earth is going to change to what end in what direction should it change? Ooh, that's a toughy now. Now that's green arrows. Different than red lines. Red lines comes first, green arrows comes second. Everybody is aware of risk and that's easy stuff. But manfully discerning a trajectory for the greater processes is pretty tough. I'm not sure how we're going to solve it, which parts of it should be more internationally coordinated versus locally coordinated. But I certainly think some degree of tapestry of coordination in addition to expanding of powers will be necessary. I'm not a no-governance guy. I'm also not a global tyranny guy. I think I think there's a lot more nuance in that conversation.

1:16:27And New York today was in this conversation much more red line flavored. Red isn't to talk about green arrows really thinking we should focus more on red lines. Respect it. And also again, he's very open to the greater cosmic premises. I think that he brought up some points that I think add to that red line discourse. The idea that availability is dangerous in the same way that greater intelligence or physical power is, that even just digital self perpetuating availability could be its own kind of danger to the current human regime. I thought that that was totally apt. And his idea of smaller things being able to control or harm greater or more powerful things was also apt. And I think these open up lenses of risk which should be actively considered, especially in the face of the hugging face open AI hack debacle. These should be things we should take very, very seriously. His paper on AI and evolution, which you can Google his name, Google academic paper

1:17:32on AI. You'll sort of get a sense of what he's talking about here, but he brings up these issues. Again, ahead of when we've now seen them in the real world, I think his understanding of evolutionary dynamics applies really well to AI. And I respect that he focused mostly on risk, but then he's open to the cosmic and that he added his flavor that brilliant mind behind meta transitions and evolution which I admire tremendously as a theory. And I think holds a tremendous amount of water to be able to bring that to the show meant a lot to me. Yours is one of the living sort of evolutionary bios or thinkers that I admire most and having him here was tremendous for me. So I hope it was for you too. And I'm interested in your thoughts on what yours had to say. I will mention as we close out, if you like these conversations and you say, man, I want to be part of conversations like that. I want to understand and maybe participate in what kind of innovation or what kind of governance and coordination might help bring about the flourishing of the greater unraveling process

1:18:33of which man is part. Good news. We have something called the philosophy circle, philosophy circle at least once a month. We get together with folks from academia, from big tech, from big think tanks to talk seriously about the trajectory of the living process and where it should go. What we should do about coordination, about innovation and many of the speakers and participants in the philosophy circle have been speakers on this channel. So many of the faces you see here have been with us in some cases on many occasions for philosophy circle. So if you want to join people like the folks you've seen interviewed on this show, check out down below in the show notes. Or if you're on YouTube and the description on YouTube, the philosophy circle link, click it, fill it out. It seems like it might be a fit. My assistant will be in touch and we can see if we can get you into philosophy circle. You might get paired up with someone at an AI safety organization. You might get paired up with somebody running a Neurotech startup, but the conversations are always interesting.

1:19:33So if you like them here, hopefully you'll like them there in person, knitting together this conversation even though it's uncouth. It's not something maybe you tweet about most of you, but important to discuss. Very, very important. It's been a real pleasure meeting some of the people that have been viewers and listeners here on the trajectory. So that's all for this episode. I look forward to catching it on the next one here on the trajectory.