Sovereign AI Stacks: The New Strategic National Resource

0:00I think it's astonishing to have a technology where the very people building it are, you know, the ones saying, look, we don't know how this works. If we succeed in our goal of making it extremely powerful, we think we might not be able to control it. I tend to think that even if we had no more progress, yeah, I would already be impactful at the level of something like the internet. The posture we need to be taking right now is setting ourselves up to navigate a huge amount of turbulence and to make decisions under uncertainty. I think the time for AGI being a useful term, a useful concept is behind us. Hi, everyone. This is the TechSurge deep tech podcast presented by Celesta Capital. Each episode, we spotlight issues and voices at the intersection of emerging technologies, company building, and venture investment. I am Sriram Viswanathan, founding managing partner at Celesta Capital. If you enjoy TechSurge, now is the perfect time to hit the like and subscribe button, and while you're at it, you can leave us a review on your favorite podcast platform.

1:03If you're just discovering us, visit techsurgepodcast.com to sign up for our newsletter and check out the archive of some very interesting past episodes. AGI is becoming very powerful. In fact, it's becoming powerful much before we fully understand how to control it or even how to live with it in a safe, secure manner. In fact, in our previous episode, we talked to Dr. Ruman Childry about AI and safety. In this episode, we're going to focus on governance. As countries raise to control everything from chips to compute to data and models, AI is becoming a new strategic national resource. That raises big questions. Who decides how fast these systems are built, how much uncertainty is really acceptable, and how AI should be deployed when even the people closest to the technology admit there is still so much to learn? Helen Toner has had a rare vantage point into these questions.

2:05For more than a decade, she has worked at the intersection of AI policy, national security and U.S.-China competition as a leader of Georgetown's Center for Security and Emerging Technology or CSEPT. She also previously served on the Board of Open AI, participating in many of the company's early years, both the groundbreaking technology evolution phase and the turbulent phase during Sam Altman's ouster from the board and the company. In this conversation, we explore the concepts of AGI, artificial general intelligence, sovereign AI, whether frontier technology can ever truly be mission driven and why some of the hardest questions in AI are no longer just technical. Helen, it's a great pleasure for me to have you on this podcast. Thank you so much for joining us. Thanks for your arm. It's great to be here. Let's talk about the blinding-y, obvious thing that most people would want to hear about, and I'm mindful of the fact that you may not be able to get into the details of the drama

3:09that you were very much hard of and arguably at the center stage of it all, but I really want to understand more about what led up to Open AI itself. Just frame this thing for us. You were involved as a board member in Open AI and I'm less interested in knowing exactly what transpired that led to the events that happened when you were there related to Sam Aldman briefly getting out of the company and coming back, but a more interesting understanding, what were the things that you learned in that period that you think are real concerns even today, not just in Open AI, but in other companies and other places that are working on frontier AI work, that we should all be mindful of and be concerned about. What is that issue that still concerns you? There's a lot here, as always with AI, a lot to be concerned about and a lot to be excited about, maybe to zoom out and give some context.

4:12I joined the board of Open AI in 2021, but I had actually been familiar with the company since it was founded in 2015 because that was right around the time when I was getting professionally involved in AI and started working on AI policy and AI national security issues. Something that's really easy to lose track of now is how unusual it was at the time that they, in 2015, set their goal as AI, artificial general intelligence. At the time, that was really not sort of a term that you bandied about in polite company or in serious machine learning company. That was part of why the companies took out, stuck out to me at that point because that was such an unusual thing to focus on and it was something that I was already interested in back then. Coming to your question about what I've learned, what we've learned over that time, I think there's a lot that really isn't specific to any one company, but is about this growing industry where now it's not at all taboo to talk about artificial general intelligence

5:12or even super intelligence. On the contrary, we have some of the best-funded biggest companies in the world, pouring enormous amounts of capital, pouring enormous amounts of effort from incredibly smart people. Some of the very smartest people in the world are working at these companies. We still don't really have that great of an understanding of what this technology is or exactly how it works or exactly what kind of future it's going to lead to. To me, I think we've definitely made some progress over the past 10 years or so in understanding how AI works and how it can be beneficial for society broadly, but I think there's still a lot of open questions and I think for people coming in from the outside, it can sometimes feel confusing, but then I think people tell themselves, I'm sure that the experts or the CEOs have it under control and I think it's really important to say that there are just a lot of unknowns and a lot of open questions and so I think it's appropriate to be somewhat confused, to be somewhat wary and to want to make sure that this goes as well as it can. It's just lingering on that a little bit more, so OpenAI at that time was really a non-profit. What was it that attracted you to OpenAI when there were lots of other places that you

6:16could have spent your time on in all of these key topics? What was it that attracted? Was it personality, so was it that the mission, was it the open non-profit nature of it? What was it that was attracted? The mission and the technology, so like I said, I've been familiar with the company or I should say the organization since 2015 when it was purely a non-profit. By the time I joined the board in 2021, it was already a non-profit managing a capped profit entity, so no longer the peer-non-profit, but all throughout that time they were doing really interesting technical work. In 2019, I watched their Dota 5v5 where they were training teams of AI's to play professional level. Dota and have five different agents coordinating on how to have a good strategy, work on robotic hands. They were doing back then was really interesting. I didn't track GPT-1 at the time, that didn't notice it at the time, but GPT-2 was a huge deal. I was very familiar with their work, very impressed by their work and very impressed by the mission. I think the mission to ensure that AGI benefits all humanity is a really noble one.

7:16So when the chance came my way to help contribute to that mission, I thought there was no way I could say no. If you were asked to play the game again, maybe with the difference of a card, maybe different entities or companies, what is it that this experiences taught you of a governance and safety and institutional responsibility and corporate governance essentially? Because when you got in, as you said, opening eye by that time was already managing a full profit part of the business underneath it, even though it was a non-profit. You were part of the non-profit governance construct, correct? That's right. The organization at the time, the non-profit board was at the top of the structure, there was no other board. So if you were to do it again, what have you taken away from this that you would do differently and would you do something like this again and what would you be concerned about? I mean, I would absolutely do it again, I think, the chance to contribute to that mission and the responsibility to be an outside voice as well in some of these conversations and decisions that can be very focused within Silicon Valley, I do think it is, was a big responsibility that I had the chance to take up.

8:17What would I do differently? I mean, it's hard to go back and look at the many different decisions that were made and without getting into stuff that's sort of hard to talk about publicly still. I think a big lesson has to be that that non-profit legal structure just wasn't up to the weight that was put on it in terms of the amount of commercial incentives that were writing on decisions for the company, pressure from investors. A lesson for me, perhaps, was how few people understood the legal structure that OpenEI had designed for itself, including even employees, but certainly most onlookers, I think, in their mind, saw it as a kind of typical Silicon Valley VC back start-up. And the way that boards work there is they're really there to support the CEO. That's their whole job, essentially. And that was not the way that OpenEI was set up legally. It was not legal responsibilities that board members had, but I think given what a big role in our perceptions and outside pressure plays, the fact that a lot of people didn't actually understand what we were there to do was more of a problem than I had expected

9:18it to be. So this would be a couple of things. I'd love to sort of get you to comment on, are we entering a phase as an industry, especially in technology with, you know, there's some profound disruptive technologies that are coming into the horizon. You know, AI is obviously, they have lose to the AI itself is going to get very complex as we all would agree. And in a semiconductor is in the same place in social media, you would agree with this sort of a framing, which is the complexity of the technology that very few people in the industry are starting to really, you know, grapple with, you know, the nuances of what this technology could be and the dual use nature of it. And then very powerful entrepreneurs that seem to have substantial degree of control, such as from a voting standpoint, but from an influence standpoint on the, on the company and the industry, when you could sort of think about, you know, you can't fire some of these CEOs, Mark Zuckerberg, practically as a pretty significant shield on this degree

10:19of control over meta, you could say the same thing possibly about Jensen, you could say the same thing about some of the founder-led large companies. And then on top of it, you have huge incentives that the board economically stands to gain from and by just towing the line. I mean, are we entering a phase where governance is going, this is the new reality that we can't really change corporate governance, especially in the context of some of these new technologies and new structures and new idols, if you will, are iconic, you know, entrepreneurs? Is this the new reality that we're going to be in? I mean, I think it's always a bit of a dubious proposition to expect small groups on boards to be able to sort of steer things for the better. Obviously, you know, in the situation I was in, that was our job, and we did our job, or we did our best to do our job. But I think I would actually go in a somewhat different direction, which is that I think we're entering a phase where more and more of the broader public is, you know, familiar with AI, thinking about AI and interacting with AI.

11:21I don't know that the tech industry is grappling with how much concern the public has or large chunks of the public have about what is happening with AI and what it's going to mean for them. I don't want to overstate this. There's definitely a lot of people who use AI products and love AI products. And when you look at polling, it's not like, you know, 80% of people hate AI, but it is a significant, it is, you know, at least 50% have significant reservations about what this technology is doing for them. And I feel like when I talk to people in Silicon Valley or in the Bay more generally, I don't know that they're reckoning with what could happen if people get more concerned or if AI is playing a larger role in their lives. And they think of it as a force for bad, rather than a force for good. So I worry that we're heading towards a time where there may be more backlash against the technology and that the industry itself is not doing what it could to prepare for that and to head that off. What Helen is describing here is a shift in the center of gravity. For years, AI was mainly a technical discussion, model performance, research progress, and

12:23just the evolution of the compute required for AI. Now it is becoming a systems question. What happens when a technology moves from the abstract into daily life, education, healthcare, finance, national security, and sectors like that? At that point, the circle of relevant experience must widen. There's still matter, but so do checks and balances and public opinion. The debate is no longer about pushing AI to do more. It is also about who shapes how it is used. It was an interesting point because we can talk about CSAT in this context because your career, your plans across engineering, policy research, governance, and all of these things in the context of frontier AI, how is your perspective changed now that you have this experience of having been in an organization that was arguably the pioneer in the field, which has now become the most powerful company in the field? What is your perspective changed with respect to AI capabilities itself?

13:27How would you frame that as here are the issues that the industry is dealing with? Is that a difference between the sorts of issues that you're focusing on in your research which seems to be different from the issues that frontier companies like OpenAI or Anthropic are dealing with? Is that a synergy? Yeah, I think there's some synergy. I think what I've seen over the past 10 years is this technology going from being a twinkle in some researchers' eye to being real and existing in the world. And by this technology, I essentially mean general purpose, AI. So 10 years, even 20, 30 years ago, we had narrow AI systems that could do a specific thing. So you have USPS using AI to read zip codes. That's very old technology. That's a type of AI, or you could say chess computers are a type of AI, but very narrow, very specific. And for a long time, the idea of AI that could do multiple things was science fiction. But now it's not. I don't know if you want to get into conversation about what is AGI. Do we have AGI?

14:28How should we think about that? I'm happy to go there. But we certainly have general purpose AI that can do many different things. And I think we're now in a time where it isn't just a technical question of how do we get the AI to do more and more. It's really increasingly, it's a societal question. It's a question for our education system, our health system, our finance system, the parts of the world that I tend to work with, the question for our military, our intelligence community, our diplomats. What is happening with AI? How is that going to affect our lives? How do we try and ensure that it goes better rather than worse? So I think a big thing that has shifted is, as a technology has advanced, it goes from being something where the primary questions are questions of science and engineering, to something where the questions are much, much broader and need to draw on a much wider swath of people. I was at the TED talk that you were at in Vancouver last year, and I think you framed it very well. And one of the things that stood out for me was that there's a bright line that people used to have between general purpose AI versus very narrow AI. To talk about whether that line needs to get re-looked at, and how would you reframe

15:31that separation between narrow versus generally AI? There's a bit of an obsession in some circles, AI safety circles, maybe AI circles more broadly, in talking about how far away we are from AGI, artificial general intelligence. And some people will say, maybe we're only one to three years away, others will say, that's ridiculous. We're obviously 20 plus years away from that. It's always very hard to pin down what exactly people mean by that, or if you do pin them down, people mean very, very different things. So the term came from this time when the AI that existed, the AI that we could actually build was all very narrow. So each individual AI system was trained to do one very specific thing, and then it was used for that specific thing, whether it was recognizing zip codes or detecting credit card fraud or classifying spam or whatever it might be. And so the term of AGI, artificial general intelligence, the point was to distinguish from that and say, okay, we don't mean these AI systems. That can just do one thing. We mean something much broader. And I think it was a useful term back then, I think, to try and gesture direction of something more general. But once you try to zoom in on what does that actually mean, some people will say something

16:34like, well, AI that can do anything a human can do. But then more recently, people have started restricting it to, well, we want to just talk about cognitive tasks. So robotics is hard, so we'll just say AI that can do all the cognitive tasks a human can do. But then other people will say, oh, but it also has to be as data efficient as a human. So it doesn't count if it needs hundreds of millions or billions of data points to learn a new task, because that's not how humans work. So there's all these different versions of AGI, which make me now feel like it's actually just not, I've been wanting to write about this. We'll see if I have time to actually put this up on my sub stack. But I wanted to write about, I think the time for AGI being a useful term, a useful concept is behind us. And I think at this point, there's lots of things you might mean when you say AGI, but you should just say whichever specific version you mean. So some people want to talk about AI that can act as like a, the term some people use is a drop-in remote worker, meaning it could do anything that a human could do at their computer. And so companies can just use it as a drop-in replacement for any of their, you know, knowledge workers, essentially.

17:35That's a useful concept. We should totally talk about that. But that's very different from another thing people want to talk about with AGI, which is self-improving AI. So AI that can fully automate the AI R&D pipeline. It's not, you know, obviously the same thing as a drop-in remote worker. And so I think if you mean one, you should say one, and if you mean the other, you should say the other. And there's lots of other things people tend to mean by, by AGI as well. So I'm on a quiet personal crusade that may become a louder personal crusade if I get this post up to say, look, AGI was useful, you know, 10 years ago, 20 years ago, I don't think it's a useful term anymore, we should just be, we should just be more specific. So in this context, a lot of people sort of compare, obviously, intelligence as sort of this strategic resource. And what we'll talk about, you know, what it means to sovereign AI and different governance and all of that. But more specifically, AI as a strategic resource, is the comparison, you know, this is sort of the pivotal moment for the, for the, for the economies, for the world, for consumers, all of that akin to what electricity was or perhaps even the wheel, you know, how

18:40we should think about that? Are we making the right comparisons? And is it as profound? And if it is, because then, you know, the whole series of analogies that you can draw from how all of us handle those key innovations is the analogy appropriate? I'm a believer in there being lots of different analogies for AI that can be appropriate, depending on what you're trying to say. One thing I think you're maybe getting at is this idea from, actually from Nate Silver originally, which I quite like called the technological Richter scale, where the idea is you have, you know, from one to 10, different technologies can be, you know, less impactful or more impactful on each level, like on the Richter scale is 10 times as impactful as the one before it. I forget exactly how he categorizes things, but it's like, I think, you know, the internet, for example, is maybe an eight, whereas the microwave is like a, you know, a six or something like that. And it likes maybe closer to a tap. Yeah, I think electricity is an eight or nine higher than the internet, I think. So yeah, so I mean, I tend to think that for one thing, I think something, a lot of people in the space agree on is because technology takes a while to diffuse and it usually takes

19:43a while for us to figure out how to get the most out of, you know, any new technology. If all AI innovation stopped today, just was frozen in its tracks for some reason, I think there would still be, you know, 10 or 20 or 30 years worth of kind of figuring out how to get all the juice out of the AI systems that we already have that would be ahead of us. And so I tend to think that even if we had no more progress, AI would already be impactful at the level of something like the internet. So already very, very impactful technology would really change how we live. But it seems totally plausible to me that it'll get, you know, one step on the Richter scale above that, which is maybe something more like the steam engine or sort of the whole industrial revolution. And then there are people who think that it will get all the way to a 10, which I think on the original Nate Silver scale is maybe like the wheel or fire, they're just like truly radically transformative for civilization. That seems possible to me. It doesn't seem clear to me that that's, that it'll get all that whole way. So I mean, that's, that's a set of analogies, I think, for thinking about the scale of the impact, which is maybe different than, you know, what kind of technology is this or, you know, how should we respond to it? The reason why I was asking about this procreate is with this analogy to the Nate Silver scale

20:44in the context of fire and wheel and all of that. To me, it doesn't seem like the negative consequences of some of those technologies is comparable to some of the impact that you could end up with in AI, but I could sort of think of some, you know, do-aloo technology like nuclear, which may be more appropriate, which we can talk about, but the negative consequences of that scale, you know, would you say that AI perhaps represents a much bigger potential negative consequences if we get this thing wrong across, you know, different countries and different sovereigns and all of that? Yeah. I think that's right. I think if we're looking for analogs for how badly this could go, then I think potentially nuclear weapons are a better analogy, and then there are poor analogy in other ways, and there's much been written on reasons that people don't find the, you know, AI to make a new comparison apt, but yes, I do think that there is really serious downside here. I mean, I think it's astonishing to have a technology where the very people building it are, you know, the one saying, look, we don't know how this works. If we succeed in our goal of making it extremely powerful, we think we might not be able to

21:47control it. We think it might, you know, take over civilization and or kill all of humanity. Like, that's a pretty crazy thing to hear the people building the technology say, but that's where we're at with AI. I was watching Dario and and Davis's conversation at Davos, and I was a little puzzled because the way they sort of laid it out was like, you know, they all, you know, claim that, you know, this could have pretty detrimental effort, effect on the whole world, and humanity could just be kind of trouble. And then they also say that, you know, we all need to slow down. And then the obvious question is, okay, why are you not spilling it down? And do you see that as there's a discordance in what they say it could end up being and what they could do to avoid it? And it doesn't seem like the major companies are really, you know, trying to slow it down. I mean, would you agree with that? I agree with that. I think it, I mean, I think credit words do. I think it's good that they are making those statements and potentially laying the ground work to have more in-depth conversations about whether we would want to slow down at some point. I do think it's just a tough environment, a tough strategic environment to try and make

22:50a choice like that in where you know that you have competitors who will keep racing ahead no matter what. But I do think it makes sense to be at least starting to talk about, okay, well, in principle, if we could, you know, get agreement on slowing down, it's good to know that at least two companies would potentially be on board with that. I think it's genuinely hard when you have as much competitive pressure as there is right now. Let's talk about that. I think in the context of sovereign and AI as a strategic resource, how would you say the competition between companies like Anthropic or Open AI or Gemini or whatever, versus how would you rank that relative to competition between sovereigns, you know, U.S. and China? Do I want to get into AI as a strategic resource for the major economies and what is more pressing in your mind? The competition with some of these frontier labs doing what they are doing and they are not obviously coordinating it within the U.S. should they be in the context of competition with China? I don't know that I would say so. I think the U.S. has historically benefited hugely from the open competitive markets that

23:55we have. I think so far that seems to still be working in AI. It's obviously starting to be a pretty tight competition, but I do think that the U.S. companies still have the edge and then it's also of course important to say that, you know, competition for the sake of competition or racing for the sake of racing is not the point here. You know, we need to look at what is the particular outcome or strategic effect that we care about. I think if we're trying to look at what should the U.S. government do differently or what should the companies do differently or what should, you know, users do differently. Those are all kind of separate questions with, with separate answers. Sovereign AI is one of those phrases that can sound straightforward when you first hear it until you try to define it. For some, it means control over data. For others, it means access to chips, cloud infrastructure and models. For others still, it means reducing dependence on American or Chinese technology providers. But very few nations can realistically build every layer of AI by themselves.

24:58So the practical question is not whether every country needs full AI independence. It is deciding where control actually matters most. So when we say sovereign AI and just lay out, you know, what are the components of that? There's obviously compute. There's data itself, the models, the infrastructure, the governance, you know, safety guardrails, whatever. So unpack that for us. What is sovereign AI and what does that stack look like? So to be totally honest, I don't always know what people mean when they say sovereign AI. I think the version of it that makes sense to me is to be thinking in the content, is that countries are thinking in the aftermath of how the internet has developed and how big tech has developed and spread and how dominant American companies have been that company. Sorry, countries, maybe especially countries in Europe, but also elsewhere, are looking at their internet infrastructure and saying, wow, we really were just very dependent on the US, very hard to untangle there. I heard recently that Amazon is trying to set up a totally isolated Amazon EU division

26:02so they can try and tell the EU that they're not connected to the US, which I just think is amusing, where you know, only EU citizens are allowed to work on it and it's totally walled off. So that's sort of the backdrop. And then the idea of sovereign AI arises as, okay, if AI is the next technological revolution, how much control can countries have over their own AI infrastructure of different kinds, whether that is the data centers or whether it's the models or whether it's the applications. I don't think I have seen a great breakdown of how countries should think about this. So I don't think it's straightforward to sovereign AI, obviously sounds great, but if you're actually genuinely trying to be fully, you know, not reliant on the US or in China, that's just going to be an incredibly expensive undertaking and depending on the country is also going to demand a level of expertise and skills that just may not exist if you're trying to kind of do all of your AI in a sovereign way. So then the question is, okay, are there particular applications, particularly use cases, particular setups that need to be more independent than others? And how do you go about that? I think that that needs to be a much more in-depth discussion.

27:04So are we talking about, for example, AI that is used in the military and being really careful about where that AI is sourced from? Even there, you know, I would think that it makes sense for, well, I would have thought until a month ago that it makes sense for US allies to be comfortable relying on US models in their systems. If those alliances break down, maybe that's not true. A useful comparison I heard was to operating systems where I really don't think it would have been a good call for most countries of the world to say, well, we want to have a sovereign operating system. We don't want to use Windows. We don't want to use Mac OS. We're going to build our own. I don't think that would have been practical. I don't think it would have been a good use of resources. And so the thing that countries need to untangle further is where is AI like that? And where is there actually a strong enough reason to want control and independence that you do actually want to build things from scratch yourself? Let's get into that. So if you take space, for instance, or telecommunications, there are areas within telecommunications where there's tremendous benefit for collaboration. And there's the common areas where trade and other forms of communication is really beneficial

28:05and results from collaboration versus places where you want to keep it separate, cybersecurity and so on and so forth. In AI, especially in the context of sovereign AI, notwithstanding the fact that we can't all agree on what the definition of that is, but let's say for a moment, it's the models, the infrastructure and it's the semiconductors that are powering all of these things. What are the areas where perhaps China and the US can collaborate and what are the areas that's going to be really hard to collaborate and you'll have to really depend on some of the policy guardrails that will give you a degree of independence while at the same time you have an opportunity to collaborate. What are the places where we can collaborate? I mean, I think if we're looking at US and China, the ecosystems are fairly separated at this point. So the semiconductor ecosystems are quite separate, obviously, with lots of expert controls and the US primarily using chips manufactured in Taiwan, whereas China is still has access

29:07to a good number of those chips, but it's increasingly Chinese and indigenously manufactured chips. But the cloud layer, there's some more interconnection, so it's allowed for Chinese companies and users to use US cloud services, so that's probably the most interconnected layer right now. And then at the model level, again, it's pretty separated, I guess open models are available all around the world, but if you're looking at the closed platforms, there tends to be not too much use of Chinese platforms in the US and the US platforms are generally banned in China. So I think if specifically looking at that relationship, there is quite a lot of separation. I think looking at almost any other pair of countries, you might see something different. It's interesting. I'd be curious to hear what you think of when you hear Sovereign AI, because when I hear Sovereign AI, I hear a desire to have full control over the AI that you're using, which is a little bit, I guess, different than looking at a cooperation or collaboration at different layers of the stack. Yeah. I think in my own sort of rudimentary way of thinking about it, I think it all depends

30:09on which country you're talking about, because your mileage has been very. If getting access to the leading edge GPUs is one objective, we have to really ask the next question, which is what for what purpose, and if you're going to build your own AI infrastructure for your own domestic applications and possibly for military and other nefarious within the context of national priorities, but with those objectives, then you could say the Sovereign AI will have a different set of objectives versus saying countries like UAE or India that is really trying to attract large investments into the region. The national security concerns may not be as much, but they're really looking at possibly having a Sovereign AI cloud be set up, whether it's G42 or the folks in India are trying to set up a large gigawatt capacity data center, and they can care less potentially if that is being run by an American enterprise, but in that process, they've actually brought

31:11foreign direct investment into the country, and maybe the local workforce is getting exposed to the new technology and getting skills development and all of that. So these are different objectives, so the real issue is as someone who's sort of thinking about the policy issues, I have to imagine that you're sort of grappling with these two things is completely different. The question is, what are we more concerned about in the first bucket? Is it potential access to the weights or the training data that we want to be more protective of, and if that's the case, who cares about gents and selling billions of advanced GPUs to China, we're only protecting the models and the training data. That may be okay, but in the context of UAE, it may be something completely different where you might say, okay, well, it doesn't matter, as long as the actual infrastructure is controlled by an US enterprise, so we're just serving the local market with our infrastructure

32:15so the concerns may be different, do you think about it like that, or is that a difference between these two approaches? I think I would maybe separate this out into a few different issues, so I think for me, the term sovereign AI is really about control and about control at the level of the nation-state, and does a government feel that it is dependent on other countries? I think it's going to be very hard for countries to be totally independent of other countries when we're talking about the semiconductor layer, the cloud layer, I think the idea of every country in the world having their own cloud providers using chips built in domestically is, I'm sorry, that's not how the world works and how the world, not how the world should work, but I think it does make sense for countries to be thinking about, okay, especially for sensitive applications, do they have the ability to test and assure the AI systems that they're using? They have the ability to run them in a way that couldn't be shut off by potentially adversarial external parties. I think it does make sense to look carefully at how independent can they be, I just don't

33:17think that the goal of sort of full independence makes sense. I think those questions are pretty separate from questions around US trying to competition, which maybe seems like are also on your mind. So I think that's a whole other conversation about what are US interests as China is advancing this technology and what best advances the US national interest. So happy again to that if that's if that's of interest as well. The phrase AI race can be misleading because it suggests a single finish line. What Helen is arguing is that there are several competitions unfolding at once, model capability, compute infrastructure, semiconductor development, data centers, economic diffusion and military adoption. The United States may lead in some layers, China may close the gap in others. On the most sensitive questions, public understanding is always incomplete. Strategy looks very different depending on whether you care about frontier model performance or national security or real world deployment and scaling.

34:19So when people ask who is winning, the honest answer may be winning at what? So let's go into that. Where do you see China currently in its development of the technology itself and what is their dependence on stuff that we make here in the US? Yeah, they've been making very good progress. They have been prioritizing AI as a strategic technology since at least 2017 when they put out their AI generation plan. Many story actually they put out that plan in response to a document that the Obama administration released at the very end of his second term, the Obama administration put out a sort of AI plan or AI strategy or something, then China put out their document because they're like, well, if the US has a document, we need one. And then the Trump administration saw China's document and they're like, wait, China has a document and we don't have one. So now the US has to have one. So there was kind of a back and forth sparring there. But since then they have been investing heavily, I think they were taken a bit off guard. The government was by the success of language models. That was not really a priority for them. And in general, I think they're a little bit torn between in China broadly, they're really

35:23trying to shift their tech ecosystem in your deep tech investor. They want to shift it away from sort of the commercial internet platforms and apps and more into deep tech. So industrial modernization and serious hard tech that they think will give them more strategic advantage. But at the same time, I think they're seeing the surge of advancements in language models and also wanting to compete there. So when they see something like a deep seek, the Chinese government obviously pays attention and wants to kind of promote and support that. I think in general, it doesn't make sense to think of one race or competition between the US and China. There's multiple in AI. So there's a competition over the frontier as the very best models. There I would give the US the edge. There's a competition over a kind of military adoption or a sort of hard power adoption where I think it's hard to say who has the edge, you know, I would bet on the US. But that's much more about doctrine and procurement and all this sort of messy bureaucratic stuff as opposed to just purely technology. There's also a transparency issue because you probably don't know what is it that the

36:25Chinese are working on in some of those applications. Is that a first statement, do you think? For sure. I mean, on both sides, there's a lot of classified information. Actually, CSAT, my organization, we are recently put out a big paper based on several thousand contracting documents from the Chinese military looking at AI contracts that they are for things they are trying to buy. We put out part one of that last year and have part two coming out shortly. So stay tuned. So we can actually see certainly some information. They're willing to make available on the open internet about what the Chinese military is buying. There's other competitions as well. There's the broad economic diffusion competition. There's sort of the underlying infrastructure competition. So it's always hard. I get asked a lot, you know, who's winning the US or China and it's impossible to give a single answer, because there's not one single competition. If you were to just sort of unpack that and say, if indeed the race is to get to AGI first for a moment, let's just assume that that's the race. And what do you mean by AGI there? And also, yeah, exactly. And what AGI, if you said it's a more generalized version of what we're all attempting to do,

37:25whether it's a lot of human in the loop in any of the applications that are being sort of rendered and there's a self sort of learning capability that's built in and the advancement of the technology itself is not dependent on human intervention, let's say, you know, I don't know if that's the right way to think about it, but you're talking about the recursive self improvement fully automated research. That's the only automated one to look at. It's just assume that that's as one goal or one sort of version of how we define AGI to be. And within that, let's say, you know, the chips, the models, the, you know, the actual, you know, inference and training capabilities, where would you say China is relative to the US, not whether they're winning or not, but, you know, how far behind are they, you know, what is your, your sort of forecast? Let's say in another year, two years from now, well, what do you expect them to be? Yeah, I would say they're roughly at parity with the US, but the US certainly still has the edge. And I think where it goes over the next few years is going to depend a lot on compute

38:26and the role of computational power where the US has significant advantage due to the scale of build out that American industry is undergoing, where China is much more limited in its ability to access chips. It will depend on how much does the, how much does AI need large amounts of compute to keep advancing, how much compute does China end up getting their, their hands on? They are now able to build to buy Nvidia's previous generation of chips, which will be a big boost for them. They H 200s. That'll be very helpful to them. How many of those did end up buying? You know, there, there's a lot of stuff that's TBD here. In general, I think it's a good bet that they will at least keep fast following the US. They seem to be making very good use of distillation approaches and things like that to not fall too far behind. And also they have some, you know, don't want to underplay it. They have some very successful, very smart companies and researchers. But yeah, I certainly the US has an edge at this point of depends how you count. I think some people would say, you know, three to six months, I would tend to say more like six to 12, but it really depends on as you look at the kind of benchmarks results that they're putting out. So the, you know, the test results for how capable different models are, those can be hard

39:28to interpret. And it's not clear that Chinese models always feel as good as their benchmark scores suggest they should be. Yeah. So it's tough to put exact numbers on it. I would personally say something like six to 12 months, but other smart and reasonable people would say shorter. So if you were to compare this to let's say in the telecom area, Huawei was, you know, it was nowhere 15, 20 years ago. And of course, in the Nokia, Ericsson, Qualcomm, where the three, you know, the triumorate of telecom industry in terms of semiconductors and underlying systems that actually powered all of that infrastructure. And if you dig that and say, okay, five, seven, ten years after that sort of supremacy that these companies had, you know, arguably, you know, Huawei caught up, right? And today there's probably hardly any difference. These in the context of emerging market telecommunication networks, you know, India or in Africa, people would be just as comfortable buying Huawei systems versus something from Ericsson or Nokia or Qualcomm.

40:30So the specific question is, do you see China making enough advances where they don't have to be dependent on an Nvidia GPU? And if so, what is the timeframe look like for them to get that sort of a capability? Yeah, I mean, there's different questions in there. So I think we're already at the point where China is, Chinese models are competitive with US models for plenty of use cases. So Alibaba's Quinn series of models is by far the most downloaded series on, on hugging face in terms of open source models. And so if you, if you're talking about the chips, then I haven't seen updated assessments in the past couple of years. I think, you know, five or so years ago, the best assessments that I was familiar with said that outside China, staff had about a 10-year advantage over China, so that would say maybe if they've been advancing really well, maybe in five years, they can be competitive. That still sounds aggressive to me. It definitely depends on which directions the chip industry goes, so which technologies end up being successful. And so if China gets lucky in a version of, you know, next generation chip technology

41:32ends up taking off that they happen to have invested in, then maybe they get on that sooner. I think that we're, yeah, we're talking five years plus, maybe 10 years plus, but it does depend on, you know, how few different things go. Yeah. So where do you, where do you put some of the other countries like, you know, UAE and Saudi and their efforts in this area? And what are the, are the issues around, you know, AI and how they are, you know, applying some of the technology? Do they rise up to the same degree of concern from that security standpoint in the US? Are our policy recommendations be different for those countries versus what we might do with China? And if so, why? Yeah. I mean, definitely from the perspective of the US national security community over the past eight or so years, China has been seen as sort of the primary competitor to focus on. President Trump is actively changing what US priorities are in his national security strategy and national defense strategy. So maybe there's, it's a little less clear that China is in a class of its own as a competitor

42:32rather than having more of a sort of focus on the Americas. But either way, I mean, I think Gulf countries are not high on the list of concerns from a US national security perspective. The reason to be concerned about them from a national security perspective, I guess two reasons. One is around their ties to China. So, you know, if we're concerned about China, does that also mean we should be concerned about selling chips to a UAE, get or, or a Saudi, given the kinds of partnerships that they have? And then also, I think much further down the list, certainly from this perspective of, you know, your typical defense official, for example, it does, I think, make sense to also be concerned about, you know, what will those countries use AI for what kind of government governance systems they have because they are quite autocratic, autocratic, monarchy-based systems. So, I think that's another reason, but in practice, I'm not sure how heavily that's factoring into decision making right now. And so these sovereign efforts, do you see that as fragmenting the innovation ecosystem or do you think that this enables greater innovation, you know, of course, competition is great, you know, famously, I think, the whole war and the space is raised that in

43:36the start of the 60s, you could argue that that really enable a lot of the innovation that happened in the space sector. So, is this good for the innovation ecosystem or is it not so good for the ecosystem? I would tend to think it's probably not, I don't think innovation is the reason to go for sovereign AI. So I would tend to think that if any, it's either neutral or detrimental to innovation, I think the motivation doesn't come from wanting to innovate faster, it tends to want to come from having more independence or more security. So let's talk about the current administration and the priorities that the administration has to do. Do you think that the U.S. administration currently has the right priorities on AI in terms of regulation, in terms of national security priorities and trade, do you think that we have the right sort of issues at the table? I think the set of issues is right, I'm not sure that I agree with the positions that the administration has staked out, though also different administration officials will stake out different positions at different times. So I would agree more with some than with others. A couple elements I'll pick out if I think about sort of what is the overall approach.

44:39One is a focus on exporting and strengthening the American AI stack, is the language that they'll use. I think it's a totally reasonable goal and an interesting framing to be thinking about not just purely about AI models, but also thinking about the underlying infrastructure, the cloud services, the data centers, the chips, and also what goes on top of the model, so the applications, the services. I think that makes sense, but I think some of the approaches that they've taken to exporting that stack, I'm not sure that I would agree with. So for instance, I think the sales of H200s from Nvidia to China, the state of motivation there has been to, again, to kind of export this American stack or strengthen the American stack. I think that neglects. It's true that if you sell more American chips to China, that strengthens the U.S. stack at that layer, but I think it neglects the ways that having access to much more capable chips, strengthens all the other layers of the Chinese stack, so it strengthens their cloud providers, strengthens their models, strengthens their applications, that doesn't necessarily seem to have been kind of in the calculus. Another big element of the administration's approach to AI, I think has been this idea

45:39of federal preemption, which sounds kind of nerdy, but basically it just means the federal government making rules for AI, and then that overriding state government rules. I think in principle, this could make a ton of sense. I think there's a lot of aspects of AI where you really would want one federal standard, and you would want that to be clear. I think everyone wants to avoid a state-based patchwork of different rules that are hard to comply with. But in practice, so far, the proposed approach seems to have been that the federal standard should be close to no rules, and then that should override the state's ability to create any rules on AI, and then there's a lot of reasons to be concerned about that, especially given how fast-moving this is as a technology. I think it's the perfect time for a kind of laboratory of democracy. Approach for different jurisdictions are attempting different things, and kind of learning as they go. So that's another one where I see the motivation, and I just think that the implementation is not the way that I would go about it. In that context, there's a whole bunch of people that are the doomers that think that this is all going to end very badly, and we should really slow it down.

46:39And so where do you fall in that camp? Do you think that we should slow it down? I'm not suggesting that you are a doomer, but do you think we should slow it down? And if so, what would be the associated policy recommendation you might make to the administration? I don't think we should be in this phase of installing some breaks rather than slamming on the breaks. So I think it is meaningful that Dario and Demis both said at Davos that they would want to slow down if they could, and I think it's meaningful that they don't think they can. And so that, to me, would be the problem to solve here is what could it look like to give ourselves the ability to decide to slow down at some point, even if that point is not now, and there are a lot of, it's very easy to imagine ways that sort of attempting to slow down could go badly, you know, most straightforwardly, if you slow down when the risks are actually not that high, then you're for going a bunch of the benefits that AI could produce, whether in, you know, healthcare, economic growth, or education, or otherwise. I think that we should feel the gravity of the fact that these top CEOs are saying, wow, I wish this wasn't going so fast. Like that, that's a, again, that's an unusual thing to hear from people developing new

47:43technologies. And I think we should take it seriously and try to think about how we could actually install some breaks. Well, the question is, who applies the breaks? Is it self-imposed, or is it government, you know, Dictat? And it sounded like, you know, all of them were sort of suggesting that they need some external force guiding them into, you know, showing what those breaks are and giving them, you know, an instruction on, you know, how often to tap on those breaks. You seem to, I mean, where do you fall in that? Do you think it can be something that a consortium of companies coming together, or these labs coming together, and, you know, being self-imposed versus saying, you know, regulate us and tell us, you know, how we should apply these breaks, which is perhaps more effective in your mind. I think you don't want to go fully in one direction or the other. I think the approach that has started to be built out, and, you know, when I think about if you were trying to install breaks, what would that mean? Is this approach of what Seniors gets called if-then commitments, meaning if we build a certain type of AI, or if we, you know, get a certain type of test results with the

48:45AI that we're building, then we will have to do something whether that's slowed down or installed different safeguards or things like that. So all of the major companies have been building out these frameworks of just, I think, opening the eyes as it's preparedness framework, Google has a frontier safety framework, and thropic has a responsible scaling policy. These are all the same thing, different words for the same thing, which are basically saying, okay, we're trying to create very, very powerful systems. But kinds of behavior performance might we see that would give us pause, and that would make us need to, you know, reach a certain level of safeguards or confidence in our safeguards before we proceed further. And so they've started to build out, you know, what are test suites you could have for cyber offense capabilities, meaning like hacking capabilities, autonomous hacking capabilities or assisting human hackers. How do you test for, you know, how do you think about what level of capability there is concerning? And how do you test for whether your system has that level? One thing for assisting people with bioweapon development, same thing for kind of autonomous capabilities, or as well looking at that, the AI R&D piece of can AI systems advance their

49:47own, advance the next generation of AI, do you get into that self-improving loop? All of the companies are in the process of trying to set up those thresholds and think through at what point, yeah, what levels might you see that are worrying and what do you do if you, if you see those levels? And there's the beginnings of industry, sort of intra, intra industry cooperation on that. So there's an organization called the Frontier Model Forum, that is, you know, trying to coordinate some of that work. There's indications of, but there's early signs of industry to government coordination on that as well with things like the UK AI Security Institute as, you know, a really center of excellence on thinking through some of those, what kinds of things would you want to test for and how would you test for them, which is very complicated, you know, technical but also multidisciplinary work. So I think you want to have some kind of, you know, thoughtful, interwoven framework like that, not, it's purely up to, you know, the individual CEOs or, you know, the government is coming in with some heavy regulatory hammer. I think it is a really complicated question and so you need to have a correspondingly, you know, thoughtful and nuanced approach to it. What Helen is describing here is the early architecture of AI governance.

50:52Not one regulator, not one dramatic moment when someone pulls an emergency break, but a set of thresholds, evaluations and escalation points meant to make frontier model development more legible and governable. That is the logic she's describing behind preparedness frameworks and the responsible scaling policies. If a model crosses a threshold in cyber offense, biological misuse or autonomous capability, the expectation is that security and oversight should rise with it. The heart part then becomes who defines those thresholds, who verifies the breaking of those thresholds and what happens when commercial pressure collides with caution. Let's play a thought experiment here. So just looking ahead, you know, two scenarios. One in which, you know, we've understood it, we've got, you know, enough people collaborating. Of course, everything is, you know, and like in self-interest and from a competitive standpoint between U.S. and China and all of that, but that understanding is a positive scenario

51:54where, you know, things get aligned, guardrails get established, the, you know, the breaks are identified and the cadence in which the fapping of the breaks, you know, is agreed upon on one side, which is a positive scenario. But the negative scenario when this thing doesn't happen, you know, what do you think would be the result of those two things and, you know, you know, how do you see this evolving? Let's say in a five year period, what could happen? In both these scenarios. Five year period, I tend to think that probably things will look pretty similar to how they look today. I'm not in the camp of people who think things are going to be radically transformed between now and the end of the decade. For instance, I think that they might, but that's not necessarily what I expect. Yeah, I mean, I think if we're looking at more like a 20 or a 50 year time period of things going really well or going really badly, I mean, you think in both cases, probably the answer is it'll look unrecognizable and there's no, you know, it'll be hard to have predicted exactly how things will go. But I think in, you know, in the positive scenario, there's a lot of basic stuff we can say. Hopefully we have a lot better treatments for a lot better, you know, a lot more diseases.

52:56Hopefully we have really amazing AI driven education that allows anyone to learn anything they want. Hopefully we're able to solve really basic problems that still exist in the world, extreme poverty, access to water, things like that. The question of how AI, I think, you know, there's a lot of conversation around AI, sort of safety and sort of specific discrete risks. There's not that much around AI's impacts on democracy and how do we have, if the world is, if the economy in the future is very heavily machined driven, what does that mean for kind of the role of the people and our government and how that all works. But, you know, if it's a scenario where things are going well, then that's somehow panned out. And we have, you know, government work, we have some system of operating that empowers individuals and helps them leave flourishing lives. I mean, it all sounds kind of vacuous because it's very, you know, stories about utopias never sound nearly as fun as stories about dystopias. But I do think that, you know, there are really good futures on the table. If things don't go well, you know, the worst case scenario is everyone dies. I think outcomes where literally every human is killed aren't, you know, it's a pretty specific place to go to. A version of bad scenarios that seems more plausible to me or where I see where the incentives,

54:01how the incentives lead there is essentially a world that is increasingly industrialized. So maybe more and more of the earth's surface, more and more of our energy use, more and more of our capital expenditures are going towards, you know, AI-driven things like producing energy, mining silicon, building, building data centers, and there's, humans are less empowered. Maybe a lot of more people are living at or below subsistence level. So maybe there's, there's not really many jobs. Maybe people can't necessarily afford, you know, a lot of the kind of things that we're used to in our cushy everyday lives. I tend to think that the natural condition for humanity through much, much of history, almost, through almost all of history, almost all of humanity lived at subsistence level, meaning that every day you're worried about where your next meal was going to come from. And so it just seems very, very plausible to me that we head back to a world like that. And that is also going to be a world with more violence, with more disease, with much more in happiness and much less flourishing. Again, that's, that's kind of an abstract answer, but I think there's a lot of different, in terms of trying to get to the concrete specifics, I think it'll be some version that

55:02we don't anticipate. But I do think there is a really enormous range of potential features that are, that are still possible, both good and bad. So let's, let's start from back to where we started with this whole open AI situation. That experience and with your subsequent experience and sort of, you know, dealing with all of these policy matters, sort of at a global level, specifically under, you know, with, with CSAT, it has your bias change from where it would have been when you were at open AI versus now, you know, what's the 51% 49 of these two scenarios in your mind, which is more plausible and which is less plausible? I don't know, I think the posture we need to be taking right now is setting ourselves up to navigate a huge amount of turbulence and to make decisions under uncertainty. And because I think the level of disagreement between very, very smart experts on these topics and the ways that AI is a technology keeps surprising us, mean that we can't just try and lock ourselves into one path or lock ourselves into one set of decisions, we're going to have to keep adjusting course. And I think that's what we need to set ourselves up for now.

56:04Let's, let's have a few rapid fire questions, right? So, you know, just short, short answers. What is the most misunderstood idea about AI that you think that we're all dealing with today? Misunderstood by whom? Well, misunderstood by all of us, you know, you and me and, you know, our children and people that are using AI and are not as sophisticated in understanding the nuances of the pros and cons of it. What are the endless understanding about AI today? I mean, general public, maybe one would be that AI is not search engine. It's not a database. So if you ask it something, it's, it's not infallibly, you know, pulling something from a record. It's, it's predicting tokens. Yeah. So, it's as if you're an active with people. All right. Great. Underrated region or player in AI? Probably still Canada. Canada's, you know, been a real source of strength for a while, but it doesn't really pop up in many of these discussions these days, but they have some pretty powerhouse researchers there. Yeah. And, and if you had a magic wand and if you had, you know, one governance reform that

57:10you would rush to implement tomorrow, what would it be? Transparency, more reporting, more required disclosure from the top AI companies about what they're building. And, and, and the trend that you're most optimistic about within AI? There's lots of them. I'm really interested to watch the, the data coming in on AI for education and healthcare in poor countries. So in countries where the existing education systems and the existing healthcare systems are just awful, it seems like there's a lot of potential value there. Great. Well, Alan, I can keep talking to you about a lot of these things for a much, much longer time. You don't have that much time. I would love to sort of link some of the other resources from C set on our podcast so people can get to understand some of the topics that you think we should, we should really, you know, get our heads around. So thank you so much for being in this podcast and I really appreciate your time. Thanks so much. It was great talking with you. AI governance is not a problem we solve once. It is an ongoing act of judgment under uncertainty.

58:10Helen is an honest pragmatist. She doesn't offer us a simple utopian or for that matter, a dystopian forecast, but something more useful, a way to think clearly when the stakes are high and the answers are not just incomplete, but sometimes even unacceptable. After laying the discussion of open AI, sovereign AI, China and AGI itself is a deeper, more significant question. How do we adapt when a century or a once in a lifetime technology is moving faster than we can actually comprehend? That may be the real ultimate human challenge of the AI era. Thank you for tuning in to the tech surge podcast from Celestei Capital. If you enjoyed this episode, feel free to share it, subscribe or leave us a review on your favorite podcast platform. We'll be back every two weeks with more insights and discussions of all things, eat tech. Bye for now.