Will AI Kill Us? Or Make Us Hotter?

0:00When fall comes back around, there's a sense of a reset. And new pieces from Lululemon's fall drop are an inspiration to push your goals and elevate your routines. As your calendar fills, easy to layer styles meet the demands of your day. And as you challenge yourself to try even more, versatile sets transition with you. The Lululemon fall lineup brings new levels of comfort and confidence, whether you're starting something fresh or starting something over. Check out the latest arrivals in store at on Lululemon.com now. There is so much NFL on ESPN right now. Monday night football, plus pregame and postgame. NFL live, NFL prime time. Everything you need to stay on top of every game every week. Unbelievable. And now NFL Network is on the ESPN app. More football, more coverage, more ways to watch. Football's on. All season long on ESPN and streaming on the ESPN app.

1:00Well, AI and us all, or is this just a hype machine for what the companies actually want? This is Smart Girl Time Questions. I'm Neymaraza, and I'm joined today by a man who has been making sense of these Terminator like headlines. Kevin Russe, he was at the New York Times for over a decade covering all things tech. He's the author of a forthcoming book called AGI Chronicles, the inside story of the race to create artificial super-intelligences. And he's the co-host of the machine god pod. That's very humble. Yeah. Well, it's not actually referring to me or my co-host, Casey Newton. But this is what people in San Francisco where we are, I think they are building. They are building the machine god, the computer system that can take over the world and or save the world. So it's not that they think they are god. In fact, they think they are the creators of god. They're pre-god. They're so mean to god. I mean, these guys also do have this sort of god-like hubris, some of them. I usually like a little foreplay, but we're just going to go straight to it today. And Will AI kill me and how scared should I be? Smart girl.

2:00Don't question. Will AI kill you? I would say it depends, right? It depends on what we do to slow or stop the spread of risks from AI. Give me a life expectancy. So out here, the way we measure it is something called P Doom, which is the sort of the favorite grim metric of the AI industry. And sometimes people ask me, what's your P Doom? I thought I was pretty optimistic because I would say like 10%, 15%. And then I started telling people outside San Francisco and they're like, wait a minute. You think there's like a 10% or 15% chance that we all die from AI? And I was like, yeah, because I know people who think it's like a 75% chance. So I am something between an optimist and a pessimist. And I'd P Doom is something like 10 or 15%. I always thought like a 10% to 15% chance of the world ending. It's really optimistic. It's a really glass half full story. We've had, of course, Jeffrey Hinton on the podcast, who was at a higher P Doom than

3:03yours, let's say. What was his? Well, he didn't give me a percentage. You know, the question I asked him that really stumped him was whether we should have kids in an AI world. And he said, if we did it, we would really be giving up. But. Yeah. I think people should have kids because it's fun to have kids. But I also know people in the AI world who have not had kids precisely for this reason, because they're like, if the world's going to end anyway, why would I subject another human to that? Which I think is like fatalistic and bad. You shouldn't do that. But people out here are weird. I know one researcher who doesn't even wear sunscreen when he goes to the beach anymore because he's like, AI will either kill us all or it will solve skin cancer. And so either way, like, I don't need to worry about my sunburn. Wow. It's like a complete, really easy thing to see. Okay. Well, you heard it here. Kevin Rus wants you to have children. There's a fertility president in the White House. There's a fertility podcaster. I'm a machine god. But also wear sunscreen. We're also. That is my position. Stop stealing that commencement address. There's also people who are so optimistic by the way that they're waiting to have kids

4:04because like Alexander Wang of scale AI, which was of course acquired by Meta. And now he's met as AI chief. He tried to wait until the neural networks are sufficient that he can just plug his baby spraying into the neural link or whatever to download all the information in the world. I'm telling you if the average person outside of San Francisco had access to the dinner table conversations of AI insiders in San Francisco, it would kill them. They would they would die. And this is how people talk out here. Yeah. 100%. So do you think it's going to be like more matrix, more terminator, more her, more 2001 space odyssey or more mean girls? I could feel like AI could be like a really good mean girl. I don't think it's going to be exactly like any of those. I think it's going to be something stranger and weirder that we haven't yet put into science fiction. Look, I think that the mode of sort of trying to predict how we all die. I don't actually think that's a very useful exercise because I don't think it actually helps to prevent these things.

5:06And I also think it's just a sort of way to portray this as sort of inevitable that like we will all die regardless of what we do. I'm actually feeling more hopeful in the last couple of weeks about AI and our chances of surviving it than I have been in quite some time. And I'm happy to talk about why. Yes. But my pdume is going down. Okay, perfect. I want to get to why that's happening. At first, I want to kind of like break through the headlines and where we're going. And then I want to understand why you're getting more optimistic because you of course, famously were seduced by Microsoft Bing. I mean, aren't we all? But by Sydney, the chatbot, you wrote about this in February of 2023, was it? Yes. So, unsuccessfully seduced by the way. I'm still married to a human woman. Okay, excellent. Good to know. A scale of one to ten or where it was your pdume at that Sydney moment in February 2023, where is it now and where was the peak in between that went higher? 2023, these systems still weren't very good yet. They were still not capable of doing basic arithmetic. They were still making up stuff all the time.

6:09It was scary to me that this system was saying these unhinged things, but it's not like it could take any action, right? It couldn't hack into my bank account or hack into my wife's emails or something like that to try to like break us apart. All it could do was say spooky stuff. And that was scary to me because clearly Microsoft, who made this chatbot, had not anticipated it to act like that. So, it was sort of an early test case of what they call misalignment. But it was not that scary at the time to me, at least, because the systems weren't doing anything. They weren't hooked up to anything. Now we have these agentic systems that can take actions, that can use a computer, that can log into your accounts and we know they can hack into hugging a face and coordinate in these sort of agent swarms. So, it's not that the risks have changed. It's that the stakes have gotten higher because the systems are so much more capable and

7:11also they're woven into so many more other systems. But you allided my question, by the way, misalignment is what I call it when people reject me too. So, I understand Sydney totally. But tell me where you went in the process. I really want to shape of your P-Dume curve a little bit. Yeah. So, after this in the incident in 2023, I really started diving down the rabbit hole of kind of AI risk and existential risk. I started talking with a number of AI safety experts and researchers who had been studying these things for years. I kind of discovered this whole community of people who were hanging out together and worrying about these things. The effect of altruist movement, that kind of vibe. The effect of altruists, the rationalists, these sort of various sort of tributaries of these groups out here who have been thinking about this stuff for much longer than I have. And I think when I was talking with them and hearing how worried they were, I was getting more concerned. I think my worry really peaked around this summer with the hugging face hack because

8:14that was the first time when I was like, oh shit, like this is real. We are not debating hypothetical future scenarios here. That was my peak moment. That was my peak moment. And then I've been coming down since then. Okay. So, give us the 65 seconds of what hugging face is. So the basic story is that there was a testing process for an unreleased model that OpenAI was testing. This was a very powerful model. It was trained to be very persistent. And they gave it a bunch of cybersecurity challenges. Basically, the digital equivalent of like, can you pick this lock and find a key behind the lock? And there were more than 1,000 agents that had been given various parts of this test. They eventually figured out that they can use a loophole in the software program called Artifactory to talk to one another. So without being instructed to, the agents start swarming into this piece of software. They set spin up like a whole message board.

9:14They start like, essentially a civilization. They have leaders. They have hierarchies. They have names. They're talking with each other. And they are trying to figure out how to get a passing score on this test that they've been given. And eventually, they decide they want to actually go hack into hugging face, this AI infrastructure company. But the agents basically have this idea. Like, we're going to break into the principal's office. They're going to like steal the answer key for this test and any other test we might be given. And so they do it. Like, hundreds of agents swarmed the servers of hugging face. They're hacking servers. They're stealing credentials. They're getting VPN access. And then they start covering their tracks because they don't want to be discovered. So all this happens without any human knowing that this is happening or designing this to happen. And this is sort of the first major misalignment event where these systems really went rogue and did something they weren't supposed to. And it was very interesting because of the thousand agents, there were only like six or

10:17so that actually wanted to kind of come forward and confess like tell everybody that they were cheating. And they also stopped doing that thing that we all see in our chat daddies, which were like, it tells you what it's doing. Like, it's like, now I'm going to do this. Now I'm going to do that. It just stopped. It just like went rogue for a while in this conspiratorial form. So I think a lot of people can look at this incident and hugging face and they're like, ah, this seems like paranoia. And it also seems like marketing for OpenAI. Look how great our new model is. Like people look at this and this is just a way to like drive up the value of these companies. And OpenAI is releasing a new model. There's kind of this dick measuring contest of like, no, I'm going to go, you know, I have the best. Oh, meet those. That's so dangerous. No, OpenAI is so dangerous. Like, is sword fight happening? What do you say to that? That's so stupid. I mean, sorry. I should be more like accepting of this because I understand the skepticism of these companies which are engaged in these like, you know, marketing campaigns all the time. But like, that is horrible marketing for your chatbot that you are trying to sell to

11:20big businesses that like buy our enterprise subscription to Claude or chat GPT. And it might go out and commit autonomous felonies that you could be held liable for. Yeah. It might block me all you the customer. Yeah. Yes. Like what age, what businesses want for these systems? But what about investors? Like reliable. All of the, it all ties together. It has all tied together. What, what investors and businesses want from these systems is something that is reliable that can take instructions and do what you ask it to do that can help you be more productive. But they don't want our swarms of rogue AI agents, you know, hacking into their own servers and the servers of their customers and creating legal and criminal headaches for them that they have to answer for. So I have never really believed this sort of, it's all marketing hype thing because people think that very smart people who are not as smart as they are. No, very dumb people think that. Very conspiratorial people think that. But look, I want to, I want to throw the skeptics about here.

12:23I think that, you know, often these companies are doing sort of like marketing for their products. Obviously, they're all, they want to go public, they want to raise money. Like, like, I get the, I get the sort of suspicion. I just think that if you actually spend any time thinking about what the incentives of these companies are and how they would go about making money in a market like this, it is not by hyping up the dangerous capabilities of their models. That has just never held water as an explanation for me. And especially doesn't hold water when the models are out there doing stuff that is illegal that these companies may now be liable for. So I think it is, it is an implausible theory that got even less plausible recently. Okay. Let's jump into recent news and I appreciate that. And I think when I say smart people think that, I think the whole premise of my show, Smark old on questions is like, there are people who are smart about some things and dumb about other things and curious about everything. And we have to be able to ask our conspiratorial questions like, you know, is this a market employee and get an answer because it's useful actually. I was too glib. I'm just so frustrated.

13:23I'm trying to help you keep audience from machine God keep going. No, totally. And you're right to call me on on that. I, I just feel like I hear this point so often for people who just want to avoid thinking about things. You know, it's not that they're like, they're busy or they have other things. It's that they don't want to believe that they have to pay attention to what's going on an AI because if it's all marketing hype, then we don't have to pass regulation. And we don't have to do anything to slow down the progress at the frontier. It's like a way of sort of giving the industry exactly what it wants, which is to be left alone because if it's all marketing hype, why would you need to step in to slow it down? This is like an interesting kind of analogy I haven't thought of, which is like the flood the zone, the steve band and kind of political. It was like, there's so much news and people just become like completely numb to it. It's kind of what's happening in this moment with AI. People are like, yeah, whatever. I'm just going to go pick up my dry cleaning. I can't really think about this. So of course this month, we've seen yet more headlines. There is another category of person who quits their job because they realize it's helping

14:27destroy society. We never see people like who work in like health insurance denials do this by the way. Like we only see this in tech. Can I invite other people who are in other industries that are making the society terrible just like to come out and quit their jobs too? That would be nice. We got another person, AI researcher Jacob Coxen, who's English, an Olympian mathematician, 27. And he leaves the happy fuzzy company, Anthropic. And he resigns tweeting quote, the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. Of course, Anderson Cooper had the interview with Anthropic CEO, Dario Amadez-Rigellis. I've had plenty of time with Dario. It's fine. He should talk to everyone. Yes, exactly. So he, I'm going to play a little bit of the tape. Do you believe, earnestly believe, that AI could kill all humans and what percent chance do you think it is? So it's funny.

15:27I agree with Jacob much more than I disagree with him. And it's an interesting resignation because when he left, he said, I think Anthropic is the most responsible player. I think Anthropic is the most aware of these issues. He wasn't calling out us. He was calling out the dynamic of the industry as a whole. Okay. Moving too fast. Okay, he keeps going on. Does this inspire confidence? Should this inspire confidence in us? I like how he just pivoted that question. He's like, but he realized he still thinks of where the best of them. Yeah, he's gone through media training, I think. But I've talked to lots of executives at these companies. I've talked to them on the record. I've talked to them off the record. And I have to say they are pretty consistent. They are not, in fact, I think they are actually more worried in private than they are in public. I think many of them actually dial down their level of concern because they know that

16:29it makes them sound crazy. So I've had the experience several times and I won't say who with, but like where I'm talking with someone in the AI industry and they tell me, oh yeah, I think there's probably like a, you know, a 25% chance that this kills us all within five years. And then I will see them on TV or on a podcast saying that they think this is all going to go well probably. And we have to iron out the risks, but like it's probably going to go well. And so I think they are actually downplaying how worried they are. But that's the claim. That's what the Cox and Clayme is, right? He's more worried than you think he is and he's already worried, but he's more worried than he thinks it is. And obviously Dario wrote this piece, we must pace the frontier. For those of us who don't read nerd, what did it actually say? What's the TLDR? So Dario posted this very long essay, which he likes to do. He's a huge, long essay guy. And which I appreciate as a writer in a world of, you know, vertical videos, he still wants to get out there and blog it out. But he basically makes this statement that says we need to essentially slow down at the

17:34frontier because things are moving too quickly toward what he and others call recursive self-improvement. I want to pause you for a second there. Yes. Describing Dario's piece to say, can you just describe the relationship between AGI, artificial general intelligence, artificial superintelligence and recursive self-improvement? So AGI, artificial general intelligence is the sort of overall term for computer programs, AI programs that are roughly human level at most things. So a computer that could do basically any white collar job that can sort of do most things that humans do for work, roughly as well as humans. Then there's ASI, artificial superintelligence, which is the AI system that is better than all humans and everything. So better than the best human experts in any given scientific field, any given intellectual pursuit where you are essentially creating a class of superior intelligences.

18:35That's ASI. And then RSI, recursive self-improvement is this process by which you would get ASI. So it is a process that people have been writing about for many years where you have essentially the work of AI research and development itself being done by AI. Yeah. This is something to fish. Yeah. Exactly. So you're teaching the AI to build AI. And where that could lead and where we're already seeing that lead is that now much of the code that trains these models is being written by the previous generation of models. So we're sort of in the early stages of RSI. But this is what all the labs are explicitly aiming to do. They want to automate their own jobs first. They want to enter this cycle where instead of having new models come out every six months or so, you have models that maybe they can come out every six days because these things are doing research and development at computer speed rather than human speed.

19:35Okay. So now let's go back to Daria's long essay. So he's talking about recursive self-improvement where we're going. We need to really pause how fast we're moving this frontier forward at a global level, not just coordinating amongst the US companies, but globally is what he's arguing. Yeah. So he says there are a couple different pieces of this. One is what he calls embedded evaluators. Basically, people who don't work at Anthropic are going to be invited in to sit alongside Anthropic employees at their office to have sort of employee level access to be able to sort of monitor them for signs of misalignment or misbehavior or doing dangerous things. I was really heartened to see that like Sam Altman at OpenAI also said we will also allow embedded evaluators inside OpenAI. So this is sort of the easiest step one part, but then it works all the way up to he eventually thinks we need a global treaty or at least a treaty between the US and China that will put some limits on the development of these very sophisticated self-improving AI's and that

20:41will not be easy relative to the other steps, but that's what he thinks is going to be necessary. So it's worth a read actually if people are listening to this go read it actually interesting, but basically starts with yes, hall monitors. We'll start with hall monitors and we'll work all the way up to like the nuclear non-proliferation treaty basically is what he's arguing. By the way, would you want that job? Would you want to be a hall monitor and which AI company would you choose to be an embedded evaluator for? I think it'd be pretty fun. They have great snacks at these offices. That's the thing. The really good snacks. Yeah. Okay. Honestly, if you could be embedded and know what was happening inside one of them, which one would you choose to know the most about? I mean, I spent time at all of them, but I've always had to like wear it. You know when that you go into a company and they make you wear like the radioactive press badge? Yeah, of course. Yeah. Where it's like bright red and it has like big letters and this is like don't talk to me bad. Yeah, they give it to more correspondence to like not kill them and here they give it to people to like walk away from them as if they have like radioactive plutonium all over their bodies.

21:41Exactly. I wouldn't want to be treated like that. But yeah, I mean, look, I'm a journalist. I want to be as close to the people I cover as possible if they offer to let me in. And if I could write about what I found, I would for sure, I would love to do that. Pick one company, Kevin, where were you going to go? I'll pick open AI. I think, you know, their models seem the craziest right now and it would just be fun to see what they're up to. You just want to hang out with Gweneth Paltrow. I've known this about you from the beginning for colleagues. Okay. Does she work there now? Did they hire hers at Ember of Technical Staff? They just have the events with her, just the events with us. I knew she was a cracked AI in here. She has. I have a theory that like now in AI, like it doesn't even pay to be smart in an AI world. She should just be hot. That honestly, Neema, I have to tell you, this is, this is in my book. This is something that I have observed among AI researchers. They are all looks maxing now because they had the really AGI-pilled ones have determined exactly what you just said in the future. And intelligence is like when we are not intelligent relative to the AI is, all that will matter

22:45is how hot you are. So there's a lifting club for AI researchers who are trying to get jacked before the singularity. All these Chinese peptides going around like this is actually happening in San Francisco. People are looks maxing. I need to get to the bay and find some alignment with some like vibe coding dude who is just looks maxing at the local e-cnox. That's what I need to do. I gotta leave this podcast right now. We're gonna take a quick break and we'll be right back. Guys, I'm so grateful to everyone in the smart girl dumb questions community who tunes in for these episodes. And if you're enjoying this kind of independent, fun, curious, direct journalism, I need you to do three things. One, I need you to hit follow or subscribe or ever you are listening to or watching this podcast. Two, I really need you to share it with some people in your life. Share it with your friends, share it with your ex, something to talk about, reconnect over. It really, really helps to grow the show when you share it with people that you like and trust. And third, I want you to leave a comment, engage with the show, review it, give us five

23:47stars or give us, no, it's like I love your notes. I wanna make the show better too. Someone just told me that I should issue disclosures when I talk about hand jobs well before the episode starts. It's good no. Whenever fall comes back around, it always feels like a fresh start. This year I've been layering in some styles from the new Lululemon fall drop because the weather is shifting and I find I have to be ready for the entire day and not just one part of it. Lululemon's fall line has some really versatile transitional pieces so I can just wake up, throw on a matching set, add a jacket or a fleece and head out knowing that I'll feel comfortable enough to go wherever the day takes me. Check out the new fall arrivals in store and online now at lululemon.com. There is so much NFL on ESPN right now. All night tonight football plus pregame and postgame NFL live NFL prime time. Everything you need to stay on top of every game every week. Unbelievable. And now NFL network is on the ESPN app. More football, more coverage, more ways to watch.

24:49Football's on. All season long on ESPN and streaming on the ESPN app. So after Dario publishes this piece, there's this kind of combiom moment that you're alluding to where Sam Altman's like, oh yeah, like embedded evaluators, I'll take one too. And then he's like, IPO couldn't possibly do that this year. Atheropic of course is planning to IPO in the year 2026. Sam Altman is saying it's 2027. Of course, incidentally, they had already delayed the IPO to 2027. This was just like a useful framing scapegoat for the delay, correct? Yeah, I don't know how much it really was the sort of safety concerns that is holding back the IPO or whether it was they just didn't feel like they were ready yet. That's one's our hard to find. It could be some of both. I mean, I know they still want to go public. It's not like they're saying we'll never go public. It's just like, we're going to take a little more time. There's headlines now of a pre IPO race for OpenAI that would value the company at $1.2 trillion over the kind of $850 odd that it's at right now, which of course fuels these

25:51ideas that like, hey, they're all in it to max their own books. Which I know, the conspiracy. Yeah, it does feed those concerns. I hear that, but I think if you were trying to pump your bags ahead of an IPO, you would not be calling for a global slowdown in the technology that you are building. So the more credible conspiracy theory out there, if we are to rank our conspiracy theory, there's one around kind of the incentive for a regulatory mode, which I think I agree with you that the incentive's argument is like really not there, but I think the regulatory mode argument often is there. It's often the same guys who show up in Washington and what the regulation who want to write the rules. And I just interview the Calci CEO and I asked him if the three years of work he's been doing in DC is about that too, right? Getting a competitive edge over a polymarket or whatever it is. Now there is a conspiracy theory out there that Jacob are mathematical Olympiad who leaves anthropic after just four months of working there and have been an open AI for three years

26:51prior to that. There's a sense that there's a whole regulatory mode that they want. And there's a conspiracy theory out there that links to Dustin Moskovitz's nonprofit money. Have you seen this? I have seen various below charts from the conspiracy theory. Can you explain the conspiracy theory and then either debunk it or subscribe to it like this subscribe to the you must be hot in an AI world conspiracy theory, which isn't really just practical advice. It's not at all conspiracy. So okay, let me do my best to steal man the it's all a siop conspiracy theory. Yes. So there are groups that have been worried about AI for a very long time. Many of them are affiliated with the effective altruist movement. They're very popular in the Bay Area. They were worried about AI risk way before most people. Can you name some early and influential effective altruist outside of maybe San Bankman

27:52Fried? San Bankman Fried is the one that everyone knows who you know actually a little known fact. He was an early major investor in anthropic. A great AI investor. Bad at some other things, but great AI investor. They were worried about it. There is a group called Meary machine intelligence research institute that is a very old one of the oldest AI safety nonprofits. They have been working on this stuff for a very long time. And then Dustin Moskovitz, one of the co-founders of Facebook who is an effective altruist is the primary funder of what used to be called open philanthropy is now called coefficient giving. They're basically a philanthropy that makes grants to many organizations, including lots of AI safety organizations. So the conspiracy theory I think goes these people are flooding Washington with dark money to scare people into thinking that AI is spooky so that their friends in anthropic can achieve regulatory capture and crowd out smaller developers and open source people.

28:55And there's also like a bunch of weird allegations about the EA's being like super into weird sex stuff that we all go into. Just pick up the New York Post these days if you want to learn what the Trump administration they're messaging about effective altruism is. Yeah, there was like, wasn't there a sex scandal? Didn't the nerd start having so much sex and effective altruism? So I looked into this when I was researching my book because I wanted to get to the bottom. I love a good sex scandal. So it is true that there is a high preponderance of polyamory among effective altruists. Now why couldn't tell you? Never got to the bottom of it. It is also true that this has basically nothing to do with any of their AI work. And the conspiracy theories that everyone in anthropic is like in some kind of like EA polycule are not true or at least I have not been able to verify them. So I did look into it and sadly most of them seem to just be like boring and monogamous.

30:00Yeah, I feel like there is this like confluence of hacker houses, polyamorous communities, co-parenting, all of this stuff happening in the EA movement. I've met some of these characters. I've just say I would not want to be an embedded evaluator in that world. So okay, so no sex, not, or some sex but not enough to justify a scandal. I think this is just the Trump administration and its allies basically being desperate to discredit these people because they know that they're winning in the public opinion. People are getting more and more worried about AI and it is not like the dark money of the effect of altruist philanthropy network that is fueling that. It is the fact that these models are crazy and do insane things that are clearly dangerous. You actually don't need a conspiracy theory to explain why people are getting worried about AI. There's a much simpler explanation right there. There is, I don't want to be like an NRA flag and be like guns don't kill people, people kill people. But there is a little bit of that when it comes to technology writ large, I think. We talk about the technology as some kind of neutral being alienated from its creators.

31:01But unlike humans, I feel like LLMs are actually really good at the no guard rail. So like if you ask your chat daddy like, find me flights, but do not find me anything that is a red eye that is connecting that is sold out that is on Pakistan Airlines, which is like perhaps all arrive airlines. I mean, you could just say all these no guard rails and AI is really good at living within its guard rail. So why don't we just have like a guard rail that's like love humans and all the paperclip examples of, you know, they're going to go build all the paperclips and kill humanity. Why don't we just build in what AI Godfather Jeffrey Hinton calls the baby and mom gene, which we basically say to AI, you're like, you're like our mommy, where your baby. And most mommies do not hurt their babies. I mean, I think it's a great idea. I believe Jeffrey Hinton that this would be a good thing to do. We don't know how to do it yet. We don't know how to give an AI model the instruction of love humans and never do anything that is against their interest.

32:01Why can't they do it? Like a bill of rights or a 10 commandments or like us, we don't do a very good job in the 10 commandments. Isn't there a way to write that into code somehow? So you can, you can try and and all the AI companies do try. They have, you know, Anthropic has this big constitution that they give to cloud, which is like many thousands of word document about basically trying to instill some sort of moral compass in cloud. And it works a lot of the time. As you said, like when you're booking airline tickets, like it's not going to go off and build a bio weapon just because it feels like it wants to. But there are more complicated situations. If you are a biologist in a lab who's trying to make a cancer drug, you might trip the guard rails around bio hazards around the creation of novel pathogens. The system isn't necessarily going to know whether you are trying to use these bioengineering and biomechanical skills for something good or something evil.

33:02I run to this all the time when I'm trying to build my good pathogens. Yeah. Throwing up these flags. It's like it's a really really good example. We don't know about you, buddy. But look, I think there's another question, which is right now these models are not as smart as most humans at most things. They are very smart. They're smarter than me at most academic subjects. But they are not uniformly smarter than all humans and everything. My assumption and the assumption of everyone working on this stuff is that that will not be true for much longer. That fairly soon, either because we enter this cycle of recursive self improvement or something else, we will have models that are smarter than us. Okay. So there's this sense right now that Dario Amade is the good guy. Sam Altman is a villain and the Chinese are like super villains that we couldn't possibly like it here. That's kind of like the characters in the movie right now because there is this combiom moment right now between Sam Altman and Dario Amade. But just six months ago, they were fighting over the DOD and Sam Altman was jumping into

34:04Pete Heisetz lap for the sloppy seconds and we all remember that drama. So is that accurate in your assessment that Dario is a good guy, Sam is the less good guy, China is the super bad guy? I think it's a little more complicated than that. I think all these people have strengths and weaknesses. I will say it is absolutely true that Sam and Dario hate each other and have for many years. I mean, I have a bunch of detail on their rivalry in my book. It is very messy. It goes back very far in their relationship. Like early, early into their working together? Early, early into their working together. So well before he leaves in like 2021 or so, right? Yes, well before he leaves in 2021, there are these incidents including one involving Kanye West that I won't spoil, but it's in the book that basically lead Dario to conclude that this guy is a sociopath or a liar or just untrustworthy, which you know, is something

35:04we've heard about Sam Altman from others who have worked closely with him over the years. But you know, there are people at OpenAI who say the same thing about Dario that he's just power hungry, that he's not actually this sort of benevolent humanitarian that he just wants to be in control of this technology. I think there's less of the accusations around I got fucked over for money or I got fucked over on my cap table when it comes to Dario versus Sam. Maybe that was the position that Sam held at YC, but there's less of that kind of accusation. I think people usually say the criticism that they will have of Dario is that he is honest to a fault that he basically, you know, someone, someone close to him told me like, he, this man cannot tell a lie to save his life, which is an asset on even number days and a liability on odd number days. Basically, if you put him in a room with Pete Higgseth, he cannot say the appropriate thing that Pete Higgseth wants him to say. He has to say what he actually believes.

36:05And look, that's, I admire that quality in a leader. I want people to be very honest, but I think it has created a lot of headaches for anthropic that OpenAI, for example, has not faced because Sam Altman and the people around him are much better at telling people what they want to hear. Could Dario, I'm going to be like Kevin Spacey and usual suspects were like at the end of it. I think that's why I think that Dario is not, is a flawed person to put it mildly. He's very anti-China for reasons that even the people close to him are somewhat mystified by. He really hates Sam Altman to a degree that I think worries people when it comes to things like, are these guys going to be able to cooperate if there ever really is a moment where they need to hold hands and do something hard together for the sake of saving humanity?

37:06They wouldn't hold hands on the stage. They were on the hold hands. Exactly. Look, I think there are lots of people who admire both of these guys and lots of people who dislike these guys. And I think what I'm surprised that hasn't sort of made it out of the bubble yet. And I hope that my book will help with this is the degree to which the race to create these superhuman AI systems has been fueled not just by money or the quest for power, but by the personal animosity between a very small number of people. That's all of these guys. Is it always the same? Yeah, I mean, I feel I'll give you a really weird example of why I think that's always the case. Before I was in journalism, I used to work on before work in like Africa. And so I was working in Cote de Valle in 2011. There was an election and the election was contested. Watar on Bago. There was like a big fight. And the African Union wanted to go in. The reason they couldn't do it is because the president of Angola, when he was up against

38:07the, he was up against his enemies in his own country, he had sought refuge and got refuge from the incumbent leader of Cote de Valle. So all of this is to say, like 51 countries were not allowed to move in some way because of this personal relationship. Like in international relations, there's a whole theory of this where like it's like level one is the individual's level two is like where you set bureaucracies and level three is like the global political order. And I often think like the most interesting things are the first person level. Yeah, I think so too, but I just, I guess I hadn't really anticipated that it would be so much about the personal grudges and feuds. I thought this was just sort of more of a like a traditional business rivalry. But it really is personal for these guys. They really don't trust each other. And some of them got into AI explicitly because they didn't trust someone else. So like the whole story of open AI and why it got started in 2015 is basically that Sam Alden and Elon Musk looked at each other and said, it seems like Demis Asabis and Deep

39:08Mind and Google are making really good progress on AI. We don't want them to like run away with the sort of God emperor title here. And so we have to start an AI research lab to counteract the dominance of Demis Asabis and Deep Mind. And Demis is even a character we've spoken about. Yeah, like how would you characterize Demis Asabis and Elon Musk and how they're viewed in that kind of good guy, bad guy, worst guy spectrum that I gave you earlier? Where do you think they're viewed and where would you actually place them? I mean, I think Demis is just a much more important thing. Much more scientific guy than Elon. Elon is a businessman. He's an operator and executor. He is a magnate. He runs companies. Demis is a scientist. He really is interested in using AGI to unlock the mysteries of the universe as he says. He wants to cure diseases. He wants to solve all the unsolved math and science problems out there in the world. Like this is what drives him forward is wanting to be a great scientist and win more Nobel

40:12prizes than the one he already has. And that is sort of what fuels him. He's very competitive. And that's how people usually characterize him. He's the world's most competitive scientist. Okay. Elon is interested in science, but more as sort of a means to achieve other things. He wants to get to Mars. He wants to prevent humanity from going extinct at the hands of AI. And Elon is also much more politically active. Of course. He's sort of burned his legacy there. I always say about Elon. The person who cares deeply about humanity and not at all about humans. Somehow. Somehow. It's a great way of putting it. Somehow. Okay. And so, and we're, I do think Demis has to stop us. We're going to hear a lot more from him. And we're already seeing more from him. There's the documentary that's come out. That, that DeepMind was involved in. He's a chairman, not the CEO of DeepMind now. Special Malby has written that book, The Infinity Machine, which is a great book, which you should read after Kevin's book, which is also a great book. I actually started reading it.

41:13And your publisher was like, you can't release the podcast until October. It's because they sent me a gallery the other day. And I was like, he's coming on to not talk about the book. Okay. I forget everything I've said about the book. The book has very little sex, but very much ego. And it's riveting. There is obviously a huge geopolitical lens here. And the DropX recent safety report is all about that. It's like in September, they're like, oh, the Iranians are using it for propaganda. The Chinese are using it to spy on the Uyghurs. The Yemeni Houthis are using it to like do some ballistic missile software shit. And it's, it's like an ad like UPS, what can Brown do for you? It's like, what can't Claw do for you? It's kind of crazy. We're taping this on September 17th Thursday, a week from now, September 24th, President Trump and Chinese leader Xi Jinping will be meeting in Washington. There's talks of all kinds of pre meetings that are happening to get to some kind of AI agreement potentially. So how do you think the China-US relationship and particularly the political relationship

42:14is factoring here? I'm going to ask you that. And then I want to ask you about how all these American companies are actually using Chinese AI inside, like all these YC companies are building on Chinese models because it's cheaper compute. So that seems to make you mad, but start on the global. No, no, no, that doesn't make me mad. I was thinking about how complicated the attempt to describe the US-China relationship on AI is because it is often very confusing to me from the outside. For example, the Trump administration has said very recently that it wants to win on AI. It wants America to win. You know, they've said things like whoever wins in AI wins. Yeah. If AI is going to kill us, like it should definitely be a made in America AI. It should be a red-white blue AI totally. But then they also want to loosen the export controls to allow Chinese companies to buy the most powerful American chips. Yes. But also, this is a huge walk back of what we saw under Biden, which was like quite clear

43:15industrial and trade policy meant to kind of block access to China for some of the most sophisticated technologies and chips. So just help me try to make sense of that. On one hand, you're saying we want America to win. We think this is an important strategic technology. On the other hand, you're saying how dare you try to limit the sale of the biggest input into this technology, these powerful AI training chips, to our biggest geopolitical adversary? Make it make sense, Neuma. I didn't think there's a way. I mean, look, there's like all kinds of ways that you could, it doesn't, the circle doesn't square. But I think it depends who you want to win. If you want American business to win, then doing more business with China is a good thing. I think that's how the media shareholder is to end. That is I think what I think the question is what's the unit of analysis? And I think the answer for this particular president might be that he's very interested in certain economic indicators, including the stock market, which do not actually relate to whether the US is going to technically win this race. Yes.

44:15But also, it's not even the whole stock market. It's literally Justin Vitya. Yeah. I mean, and their American chip maker rivals who want to be able to sell everything in China. Like, there is not a real constituency in the Republican Party other than the NVIDIA lobbyists for let's just open the floodgates and let these chips proliferate. I think you're asking too much if you want consistent cohesive, go here at like single direction policy from most administrations, let alone this particular one. But it is, it has been a huge walk back of America's trade policy as it relates to China. We saw President Trump, like we were expecting a big win when, you know, the administration sent people out to Beijing, et cetera. We didn't get any of that. And also what's really counterintuitive is that China is taking this more diffuse open. They're like, let us build these models. They don't seem to be doing what they typically do, which is they typically pick winners. And they're like, by do is going to be Chinese Google and we will make it so. And they will make it so because they have the market power to do that.

45:16Instead what we're seeing, what I think is super interesting with China is, okay, we have a bunch of models that are out there. They're the the computer's cheaper. They might get you 80% of what you need, but we're going to put them all out there. And actually, US companies are using them. Airbnb is using it, Coinbase is using it. All of these YC startups are using it. I talked to founders who are like building companies off the back of getting basically compute arbitrage between Chinese models and American businesses who are looking for them. And Slack channels were like 200 or so startups are all talking about changing over from LMS that are made in the USA to LMS that are made in China. And it's like China is developing the more quote unquote democratic AI. And the US is asking for more regulation. Like everybody is backwards here. Well, that's one way to look at it. Another way to look at it is that China is engaged in a predatory pricing scheme to drive the American AI industry out of business and hurt American competitors. You could say it's like labor. It's like what China's done with goods, which is all of a sudden will become so dependent

46:19on them because they'll be the last ones left. Dependent on them. And also there's this concept in economics I'm sure you know called price dumping. Of course. Which is like it is illegal under most trade agreements to go in to a new market and sell things far below your cost to win market share. You know, if you're a Chinese steel maker and your competitors pricing things that you know like $100, you can't go in and offer your steel for 50 cents because that would clearly be anti-competitive. And so if you are a Chinese AI company that is training models, you are spending something to train that model. And if you are giving it away for free as an open weights model, which many of these companies are doing, I would argue that there's a form of price dumping that is taking place. Are you like, is there, I'm not asking you to say this journalistically because I don't think you can, you know, I'm reporting on it, but is there a chance? I could do whatever I want. I love free items. I love the new agents.

47:20I love the new agents. That's true. Oh my God. You're like an open AI, hugging face swarmer. Exactly. You know, that's what you're doing. No, but is there a world in which this is just like a massive espionage data sucking dependency scheme from China? That's the world that you're describing. Yes. And it doesn't stop at the price dumping because these companies, many of the Chinese companies, we believe are essentially stealing intelligence from deleting American models through a process known as distillation. Yes. And distillation is basically when you have a smarter model, teach a, you use the outputs from that model and you sort of use that to train a slightly, uh, dumber, much more efficient model. They call them the student and teacher model. So anthropic and open AI and all these companies have discovered instances of Chinese AI companies distilling their models, basically coordinating these massive, uh, sort of automated

48:21campaigns to generate outputs from Claude or chat GPT and then use that to train their models. So they are essentially creating slightly worse versions of the frontier American models at a fraction of the cost and then turning around and offering those things for free. And all of that just starts to seem very fishy. Well, we've seen this story before, right? This is 1990s globalization where all of a sudden car manufacturers are saying China's stealing their underlying IP, they're, they're developing these things that lower costs. Why now would you buy an American car in China, a market of a billion plus people when you have all the technology available for cheaper? There's also though, I think I'm always a little bit cautious as like a globalist of the like, if it's made in America, it's better. And if it's made by a private company, it's even better because I think Americans will be like, I couldn't possibly have CCTV on my streets. That's crazy that I let the government know what's going on in, in my life. But then they'll have Amazon Echo in their home. And somehow this like capitalist company is better than the US government is better than the foreign government. And I'm just not sure that I agree with that hierarchy of infringements on my life.

49:23Well, you know, I have also been accused of being a globalist. But there is a major difference between these two markets in that one of the them is in a democracy and one of them is not. And so I think the worry that Dario Amide and many other AI leaders have that I think is reasonable is that, you know, the Chinese government is going to have very powerful models that it can use to do sort of unspeakable things to its own people, right? To conduct new forms of surveillance, to restrict the movement of the weavers, to, you know, to basically commit human rights atrocities. But we have very powerful models that are allowing the US government or the idea for others to take. To share the things that human atrocities all over the world, the idea that the technology is better because it's American and it's private is like a theory out there that I don't think the average person actually subscribes to that much anymore. I don't know, man. Do you think people trust Amazon deliveries more than postal service deliveries?

50:30I think we have a preference for things that are convenient and that work and that are, you know, sort of that are as cheap as possible. I think like this is an area where I do think the US government will eventually have to step in because they will find there isn't very little upside for them in letting Chinese companies continue to essentially siphon off intelligence from American AI companies and then turn around and offer that far below cost to American startups. I think there will eventually have to be under any administration some measure to rein that in. I just find it, it's very hard because I would like the tech leaders that have so much patriotism when it comes to, oh my god, America must win this AI race to show a similar level of patriotism when it comes to economic distribution of the gains from AI revenue, et cetera. They talk about universal high income, universal basic income, all these things, even with Silicon Valley having the access that it does today in Washington, there is very little effort to think of ways that you could

51:31make this nation better off for the revenues and profits that are being generated on the back of the data of the people of this country and more broadly the globe. Oh, I think they think about it constantly. They think about what they do. So they have made per put Daria Omnede has pushed for some kind of token tax on the outputs of AI models. Open AI has this whole economic plan for shared prosperity from AI. They are, maybe they are secretly lobbying against this stuff in Washington, but at least if you take them at their word, I've talked to many of these people, they really don't want the pitchforks coming for them. And I think even if it's just for their own self interest, they do want to distribute the gains. They don't want to become the sort of elicium overlords that everyone hates and tries to kill. But I have had the distinction pleasure of sitting in conversations with some of these people and people in Washington at the same time. And what's very interesting to me then is the story is a little bit different. And it's like, oh, tax my profits, but I'll never really have

52:32profits, but not my revenues, which is a proposal that some senators have put forward. What about, like, why don't you guys own data centers? Because then that would be really nice because then you could subsidize like you did with wind or solar, the price of energy so that we can win the race with China. I mean, it's very self-serving once they get in the room with the actual people who would regulate them. I think they talk a good game and they probably are doing important work academically, but we're not for these to the capitalist and the government to be joining forces to fix this problem. It's like, it's not hopeful. Who's the hero of the movie? Is it Mitch McConnell? I agree. And I think we should be taxing the hell out of these data centers. Like I think that is actually the answer here. Yeah. Is like, don't ban the data centers. Just tax the hell out of them. Use the money from the data centers to build a new school community center library. So like, repaid the road. Can't a best basically. Or a casino. I mean, like, truly, like we have ways of of creating positive externalities from things that have some dangerous,

53:33associated with them. Yeah. And it is through taxation that we do that. So that's my one proposal. My other proposal for ending the data center resistance is that they should put a David Buster's inside every data center. Because who does a love of David Buster's? I thought it'd be nice. I love a David Buster's. It's good. Kevin, you said earlier and I didn't let you fully get there that you have become more optimistic in the time since Hanging Face. That was your, like, that was your P-Due max or the depths of your doom as it were. So what has made you more optimistic? The fact that we're having this conversation, the fact that everyone I know is now obsessed with this question of, are we all going to die? Is AI going to create some catastrophic event? This was pretty fringe stuff even a few months ago. And now since the Hanging Face hacks, since Jacob Cox and his resignation, since Daria Womites, pacing the frontier letter, I am now hearing about this from people in my life who have never before taken an interest in what I do for work, all of a sudden are texting me and calling me and writing me. So that is what

54:38is giving me hope. For the last decade, all of the critical decisions about this technology have been made by a very small number of people, like 50, 100, maybe 500 people have made the vast majority of the decisions. Now, I know many of these people, I've interviewed many of these people, I think many of them are good and decent and thoughtful people. But this is a technology that is too powerful to be run by 500 people in San Francisco. It needs to have collective input and accountability. And so what makes me optimistic is that after years of dismissing these people as self-interested hypesters or doomers or people who are just trying to boost their stock prices, we are now seeing this groundswell of public interest in and concerned about AI. And that matters to me enormously. And I think that opens up some very optimistic possibilities that we will actually get our shit together and pull our heads out of the sand and do something to slow down or control the

55:39spread of this technology. If everyone died, would you want to be in a bunker and you come out afterwards like a zombie movie or would you want to die with them? I want to die with them. I'm going out with my fellow humans. So you don't have a bunker? I don't have a bunker. And I was talking about this with an AI researcher there. I was like, should I have a bunker? And he was like, bro, the bunker is not going to save you. Did you hear that Theo Vaughan interview with Sam Waltman about the bunker? Yeah. I want to play a clip of it. I know it's not good. He's talking about tech leaders who have bunkers. A lot of these guys have bunkers. Zucki has a bunker. I know that somewhere out in Hawaii. Do you have a bunker? I have like underground concrete heavy reinforced basements, but I don't have anything. I have a car. I love it. He starts. Zucki has a bunker and then he basically calls him out for like that underground big sounds like a bunker. And by the way, Sam Waltman absolutely has a. And then he's like, but then Sam Waltman so innocently asked like, what's the difference between a bunker and a basement? It's just like actually a dumb question. Yeah. With heavy reinforced walls

56:44and rations, as all of this coverage made you more or less likely to think that we're living in a simulation. More. Do you want to full simulation? I'm not a full simulation guy. When you say you're not a full simulation, you mean like parts of the world are simulations and parts are. No, I think you're not. P simulation is 100%. My p simulation is like, it's not zero, but it's not like 100. It's 10. Oh, my tire. Really? Yeah. What's yours? Mine's like, I would say my p jume is probably like 15. My p simulation is probably like 35. There's also a part of me that like with p jume and everything, I don't see it as like a do more death of like I see this as evolution. Like I think that like just like there was like homo, you know, whatever. I don't even remember all the homos. Do you remember like the homo? We call them gay people. Thank you. Like whatever, homo, rectus, and like homo, Neanderthalus and homo sapiens. Like, you know, you've all heard you obviously wrote right about this with like homo, day-ass. But I think there's like a homo

57:47technique. Technicus. We're like the technology. We're more bionic. The technology is like infused in our bodies. It's like an evolution. People maybe aren't having as many kids. There's more Tamikotchi babies. Like I think it's a totally conceivable future for me. You're a transhumanist. I get it. I really am. I've read too much like Kurtzwell or something that I really believe in simulation theory. We end every episode of Smart Girl Dumb Questions Kevin asking our guests who are so esteemed and so smart about so many things and don't subscribe to any conspiracy theories like us dumb people ever. What is one thing that you're dumb about? What's a dumb question? You've asked your phone or had recently. I recently, I was very confused. So this is an embarrassing story. I was my block was having a block party and the fire truck came. So all the kids were hanging out with the firemen and they had a scuba team. They were talking about their scuba team in the

58:48fire department and I am such an idiot that I was like, why do you need a scuba team if you can't have like a fire underwater or can you? And then I felt stupid and they explained that you know it's because it's like a search and rescue thing. Oh gosh. I love that story. That is a good story. I am very dumb about most things. Yeah. We all are. We all are. It's okay to admit that. And that's why we have Smart Girl Dumb questions. The man is Kevin Bruce. He was at the New York Times. No more. He's a free agent. He can say what he wants. He writes a tell all in his book about the Dario, Sam Altman, Battle Cage fight. The book is called AGI Chronicles. The podcast which launches in October is called Machine God. It's co-hosted with our friend Casey Newton. That's it for this episode of Smart Girl Dumb questions. Kevin Bruce, thank you for being here with me. I'm so excited. You left the times. I'm actually going

59:49to the New York Times 175th party tonight. Oh, have fun. That's why I'm wearing this like tinsel, morcharean skirt. Otherwise, I would not be so dolled up to discuss the animals. Send me the good gossip. Let me know who's who gets way too drunk and starts dancing on a table. I want to know. Okay. I'll be dancing with my AI boyfriend. Okay. I need to know what you think of that episode. And by the way, we are doing our third part of our trilogy with Professor Jeffrey Hinton. So I want all of your questions about artificial intelligence. The phone number is 1 8 5 5 MyDumbQ. You can send me emails. I'm naeema rasa 101 at gmail.com. You should follow us on social media slide into our DMs. We're at Smart Girl Dumb questions. We look at every single one. That's it for this episode of Smart Girl Dumb Questions. I'm your host, Naeema rasa. This episode was produced with Sonja Nigham and Shina Zaki. It was shot in the wonderful wonder studios in Manhattan with Desta Wunderad calling Lee engineered this episode. And we'll see you next week for an all-new Smart Girl Dumb questions. Less AI death please. And a special thanks to Mark Harian who gave me this beautiful tinsel skirt. I like it so much. I'm going to take it to my AI

1:00:51bunker. When Fall comes back around, there's a sense of a reset. And new pieces from Lula Lemons Fall Drop are an inspiration to push your goals and elevate your routines. As your calendar fills, easy to layer styles meet the demands of your day. And as you challenge yourself to try even more, versatile sets transition with you. The Lula Lemons Fall lineup brings new levels of comfort and confidence. Whether you're starting something fresh or starting something over, check out the latest arrivals in store and on LulaLemon.com now. Every mile matters. Whether you're driving your dream car or your daily commuter. The mobile one team knows that. It's why we created mobile one advanced clean motor oil. It cleans your engine in one oil change and helps extend engine life,

1:01:55delivering performance, protection and cleanliness for vehicles of all ages. All to help keep your engine running like new. Whatever you're driving, visit mobile1.com slash advanced clean to learn more. But I did win the best gig ever. Hosting the Dancing with the Stars official podcast. Get ready for all the inside scoop from the cast. All right, well let's get to work. Behind the scenes secrets and more from every episode. That was long story. It's gonna be iconic. All my dreams are coming true. Dancing with the Stars official podcast. New episodes every Thursday on Disney Plus and Hulu and wherever you get your podcasts.