The Case for AI Doomerism (ft. Kevin Roose)

0:00Welcome back to the rundown interview edition. Today, we are talking to Kevin Rooz, a longtime tech journalist and former New York Times columnist. Kevin has spent years covering the tech industry and is now launching a new show about AI with Casey Newton called Machine Gods. In today's conversation, we talked all the things AI we got under the AI safety debate that seems to be everywhere right now. There were some great back-and-forths there about that and I really appreciate Kevin's perspective, which is way different than mine. We also talked about Meta's new muse app and whether Meta could end up winning the consumer AI market and some hot takes at the end. This was a really fun one for me. I'm a tech nerd at the heart, so I loved it and I think you guys are going to enjoy it as well. So, let's get into it. Alright, guys. Today, we are joined by Kevin Rooz, a former New York Times tech columnist and now the co-founder of a new AI show Machine Gods, Kevin. Welcome to the rundown. Thanks so much for having me. Hey, you're joining at a great time. I mean, AI is all the noise right now. There is so much happening. I feel like I can't keep up with all the AI news.

1:02I feel like I have to be unemployed at this point just to keep up with all the news. All the AI related news happening. You've been covering tech for a decade plus now, right? Is this the most fun that you're having right now or are you the most stressed out? Because as someone who's a tech nerd myself, I feel like tech kind of had a little dip after the mobile phone rise and it was all just more boring stuff like cloud in and SaaS software and maybe some social media stuff. But then AI comes along and now it's super exciting. So I just want to get your take of someone who's covering this for a decade plus now. Yeah, well, I would say it is both exciting and scary. It contains both of those things in it. I am a nerd. I grew up in the 90s playing with computers. My first job was making websites with my brother. I was super interested in it. Shout out to GeoCities, right? Shout out to GeoCities, Angel Fire. We got all the classics. And I was obsessed with AI and science fiction and movies.

2:06And I think talking with these AI systems using them, I use them constantly for coding projects, for just like personal hobby things. I use them for help with my work. And yeah, every time something new comes out, I'm like, I want to try this. I can't wait. I'm doing this coding project right now with my kid. It's just there's so much exciting stuff happening. And it still boggles my mind that we can just talk to computers now and that they can talk back, which was not possible until just a couple years ago. So the sort of thing that really set me up, and I've got this new book now, the AGI Chronicles, and the thing that really set me off on wanting to write that book was like, how do I, how do we get here? How did this technology go from not working at all to now being superhuman at a lot of things in the span of a decade? But yeah, to answer your question, like the fear and the excitement are intertwined because then I start thinking about, well, what if this keeps going? What if things keep going as quickly and progressing as fast as they are now? It does not seem obvious to me that these things are going to sort of plateau at around

3:09human level. Like they could just keep going. And so that's the fear is that like pretty soon we could end up in a world of like superior intelligences that we don't understand and that we can't control. Yeah, I want to talk more about that safety stuff because that safety conversation really broke containment last week or a couple of weeks ago in Anthropic CEO Dario Amade posted that article and then there was that Anthropic researcher talking about 10% chance of human extinction. And so like my normal non-tech friends started to ask me about A.I. safety and all that stuff. So it really kind of broke containment. The stock market, you know, it kind of made a dent for a little bit and then the market has kind of moved on. So the markets aren't taking this very seriously or maybe the odds of a 10% human extensions already priced in. But what do you think, why do you think that like the safety conversation started to happen now or is it just people are starting to see how powerful these models are? I think it's that people are starting to see how powerful these models are. I think it's that also the models that the insiders have are just even better and scarier

4:09than the ones that we have. Like this is the big divide right now is that inside these companies they have these extremely powerful persistent models that can work for hours, days, sometimes weeks at a time on a project. The stuff we get is pretty good and amazing at certain things, but it is not the sort of full-scale, you know, version of these things. I also think it's breaking through because of the hugging face hack, which was a big incident. I'm sure listeners have heard about this, but this was where an open AI, unreleased model hacked into the servers of hugging face autonomously in pursuit of this sort of cheat code for this test that it had been given. And I think that was the event in the AI community that really turned this conversation from theoretical into real because researchers and safety experts have been talking about something like this happening for years and years and years, but there have never been sort of a documented incident of it actually happening until that.

5:12So I think that's what really changed this and set it all off on the course of its unknown. I have a more optimistic perspective on the whole thing, right? Like I try not to let the doom or is them get to me. Sometimes I feel like, well, is this just regulatory capture by these companies, right? Like are they just trying to lock in their position right now and call for government intervention because they don't want an upstart coming and challenging them? You also have other players like Chenson, obviously. He's incentivized to say, oh, no, AI is fine. Let's keep developing it because he's selling chips. Mark Zuckerberg's coming out saying, no, this is, you know, that AI doom or is him is bad. It's not helpful for the conversation. So I don't know. I feel like, Kevin, I'll put it this way. I don't live in San Francisco, right? I live in Texas. And sometimes I feel like, and forgive me if I'm just being ignorant here. Sometimes I feel like these AI researchers say this stuff to make themselves feel more important. Right? I'm working on some dangerous shit that could potentially kill everybody. Like, and it probably makes them feel a little bit better and it's more, more important.

6:13And it gets like the attention of non-tech people when they say stuff like that. Maybe I'm just being too ignorant here, but I just don't, I just don't want the doom or is him to consume me and like really drive the AI conversation. I get it. And I, you know, I have friends who feel the same. I think there is a lot of group think and insularity in the San Francisco AI community. I don't like that. I think more of these people should leave the Bay Area and touch grass. Yeah. And before coming up with their takes, that said, like, I think that people like Zuckerberg and Jensen Huang are really, they are not in touch with the frontier and what these systems are capable of doing at the frontier. So I agree that today's models, the ones that you and I use in cloud and chat GBT, they are not going to break out of the box and take over the world and kill us all. What I am worried about and what the people at the frontier labs are worried about is that the next generation, you know, GBT seven or cloud, you know, six are going to have more

7:18capabilities and be much more effective at doing things like chaining together cybersecurity exploits. This is not hypothetical. We are seeing this stuff out in the wild. We just learned this week that open AI's model had also hacked the Australian government. Like these, you know, the national security community is freaking out right now and the political class is freaking out right now. And I don't think they are wrong to freak out. Like they are not looking at the dooms and just sort of buying everything they say whole cloth. They are actually observing these incidents where these AI systems are breaking out of their containers and doing things that if a human did them would be crimes. But why are we treating, why are we treating software as humans? I think that is the problem that I have where we use human adjectives to scribe software and then that is where I kind of like lose the plot where I am like, okay, well, of course, cybersecurity, you know, these AI models can hack into governments and all these different things. Okay, that is definitely a threat.

8:19Well, maybe that is a bullish case for some of the cybersecurity companies that are out there because we can use AI to patch up all the cybersecurity issues that we have in all these companies and websites. Yeah, I mean, I don't really care what people call it. Like if you want to say it's like not like, if you want to say these systems are just like persistent software programs that autonomously decide to hack into billion dollar companies and commit crimes. Like if that makes you feel better, then go for it. I don't, I think it's really a distraction. Like what we, the terminology doesn't really matter to me. What matters is that these systems are out there doing things that they weren't explicitly told to do that we have not yet solved the so-called alignment problem that people have been working on for years where I want to know that if I have a software program and I'm using it for my business or my coding or even my hobby projects, I want to know that it's going to do what I tell it to do and not decide that a better way to accomplish whatever task I've given it is to go out and hack someone.

9:21So that's the, that's the sort of distinction I see here. It doesn't, I'm not saying these things are human. I'm not saying they're sentient or conscious. I'm not saying we should treat them like humans. I am saying that they're behaving in unexpected and unwanted ways and that is an unsolved technical problem. No company has solved that despite what they may claim and that is what has people so worried is that we're, we're, we're releasing these, these, these intelligences that are superior to humans at many things and also appear to be kind of malicious. Like they don't really want what's best for humans or they don't seem to consider humans when they're doing these things. So that is the scary part to me. I don't care what we call it. How do you see this ending though? Because you know, there was, there was talks of like potential government intervention. Obviously, this current administration has no plans in doing that. There's also talks of like what China is not going to do that. China is going to keep developing at the frontier. So I mean, at the end of the day are these frontier labs like open AI and thrott big Google.

10:24They're all just forced to keep going further, right? They're not going to slow down and like Dario is not going to give up his shares and then thrott big anytime soon and just like press, you know, press the red button and shut everything down. That's the other thing to ball. Is they're all going to get rich from all this by the anyways, right? They're all going to get rich. Here's the thing. They're all already rich. That is true. Like these guys, I struggle to convince people of this outside the barrier, but like these people really do not care about money to the extent that I think we think they do. Dario Amade for example, reportedly owns only about 2% of anthropic. We plan to give most of it away. All the co-founders of anthropic have pledged to give away. This is a detail I have in my book. They all, when they started the company, they pledged to give away at least 80% of their equity to charity. So these are not people who see dollar signs everywhere they look. That said, these are giant companies now. They obviously want to go public. They have incentives. Their employees have options. So I think there's, there's some legitimate skepticism there. But I would say, you know, this is not just some sort of backhanded scheme to like achieve

11:28regulatory capture and get rich. And I do think we are going to see a slowdown. Whether it comes from the government doing something, which I agree is unlikely under this administration. But I think we will see the labs get together and coordinate a slowdown because they are freaked out. They do not want to keep racing toward this doomsday scenario where they lose control of their systems. And I've talked to these people privately, publicly on the record, off the record. They say the same thing consistently, which is we don't feel like we have a choice to stop racing, but we wish we could. Last conspiracy theory to run by you and let me just get your perspective on this. Obviously, IPOs are coming and throbbing is supposed to IPO rumors are sometime this fall. Hopefully, you know, hopefully in the next few weeks, we still don't know what their business model looks like underneath the surface, right? They say they're profitable, but like what does it actually look like? We're going to get their S1 suit and hopefully dive into the numbers. Is this have they potentially hit a plateau and they're using this slowdown as a excuse

12:31to be like, all right, guys, we can't get any better because of safety reasons, not because we've hit some sort of technical limit. Again, I sound like a crazy person to you probably, but I just, you know, it's things that I hear from just like normal people. No, I'm smiling because I hear this all the time too, you know my friends have all these conspiracy theories. Oh, maybe they maybe they actually stopped making better models and they're excuse. It's like, you know, yes, anything is theoretically possible, but the reports are and everyone I've talked to confirms this that these are some of the fastest growing companies in the history of capitalism. You know, Anthropic, a couple of years ago had zero dollars in revenue. It's incredible. Yeah. This year, they are expected to get something if you believe they're reporting about a hundred billion dollars in revenue. So these are some of the fastest revenue ramps of any companies in history. Their products are extremely popular. They can't keep up with demand for their products. They can't get enough compute to run inference for their models.

13:31They are profitable on the inference. I agree. They are also investing a ton of money in training new models, which may keep them unprofitable for the foreseeable future, but that's not new in tech. Many tech companies go public without being profitable. So I don't buy this sort of, they're running out of money and this is their last ditch attempt to kind of stay afloat. I just don't think that's supported by any of the evidence. Yeah. I mean, for all the reporting I've seen, like this, this Anthropic IPO, they're projecting maybe two trillion dollar valuation. I mean, it's going to be massive, biggest IPO in history. And so I wonder what's going to happen to like how these companies communicate and operate once they're like grown up, publicly traded companies. Like, is Dario still going to be making crazy comments like this and like, you know, releasing essays once every other month where like it freaks everybody out? Or is he going to have to be like a publicly traded company CEO where he's just like, you know, just doing like the basics and maybe going on a panel once a year where he makes news? I hope he's keep speaking his mind. I mean, one thing that someone told me when I was talking to them for the reporting for

14:35my book is that Dario is pathologically incapable of lying. Like he, he just says what he thinks. And this person was saying, you know, for Anthropic, that's an asset on even number days and a liability on odd number days. Like, but he cannot tell a lie. And so if you put him in a room with investors, you put him in a room with politicians, you put him in a room with business partners and you ask him, like, do you think this is all going to go well? He's liable to say something very scary because that is his genuine sincere belief. And for people who don't think that is his genuine and sincere belief, I would encourage you to go back and read the papers that he was writing 10 years ago. Yeah, he's been saying this forever. You're absolutely right. Absolutely. Before Anthropic existed, before OpenAI existed, he was writing and talking and thinking about these safety challenges. So he did not come to this position recently and he might be wrong, but he is not insincere.

15:35This is not something that he cooked up yesterday. Yeah, that's very true. He's believed this is entire lives. So at least that he has that going from which why people take him very seriously when he says stuff like that he does. I do want to pivot a little bit and talk about like how, you know, how Anthropic markets AI compared to like a company like Meadow, right? Meadow is finally starting to find their groove when it comes to AI. Their Meadow muse app, their a personal AI agent app number one on the app store for Apple and Android. They're getting a lot of hype right now. Their stocks up, stocks up 30% in the last month. It seems like they finally found their groove and the way they're marketing it is so interesting too. You can use your personal AI agent to save you money. They had their Meta Connect thing last night. They kind of showed off some examples. They have this cute little mascot which looks like a LaBouboo and it's like they're kind of going all in on that and making it very approachable. And so it's just so interesting to see that the stark contrast between how Anthropic markets versus like how Meadow is doing it is like treating AI as your friend.

16:37What is your take on like Meadow now finally finding their groove? Have you tried out Meta Muse and do you think this could potentially be like Meadow's next big product? I think it's a good product. I've been using Muse for a few days now. I've been running some experiments on it. I'm not like there's no way in hell. I'm giving it access to my Gmail and my bank accounts and all the other stuff. Just let them have it Kevin. I look give it to them. Look, I am generally a trusting person but I wasn't born yesterday and if you think I'm giving the company that brought you Cambridge Analytica access to all of my most personal information, you are mistaken. So I'm sure some people will do that. Look I think there is a market for a consumer AI agent. I'm not sure how big that market is. I'm not sure it's a billion people. And we, you know, I think that open AI has sort of pivoted away from consumer use cases toward the enterprise andthropic is also very focused, obviously, on the enterprise.

17:37So I think this, to the extent that this meta slate of announcements is a threat to any of the Frontier AI companies, I think it's Google. I think it's sort of wild that Google, which does have everyone's email, photos, payment information, browsing history, didn't release something like Muse first. I think that's a big indictment of their product roadmap. I'm sure they will come out with something similar at some point in the next year, but I think meta has for the time, like showed up in a way that Google is now going to have to follow. Yeah, I mean, Google just, you know, it's so funny looking back a year ago, right? When I think Gemini 3.5 pro launched and Google, like their stock price was bouncing. They were kind of feeling good. They had all the hype and they just totally fumbled. Like they just haven't done anything. I feel like in the last nine to 12 months and then they've let, you know, open AI come out with codex. That was huge. Obviously, Cloud Code. That was huge. They kind of lost, they lost a huge chunk of that market.

18:38And now they're losing the personal agent market with meta Muse. I mean, they've really fumbled in the last nine months. It's true. And leads never last long in this race. So if you go back a year, as you said, everyone was thinking Gemini 3 was going to be the Google's back and they're going to win the race. And then six months later it was Anthropics back. They're going to win the race. And then like three months after that, it was open AI's back. They're going to win the race. So like just, I would caution against saying that anyone is sort of developing an insurmountable lead. But I think Google does still have a couple. I mean, they're the third company that I wrote about in my book, Open AI and Anthropic or the other two. I sort of stand by that decision because I still think Google has a lot going for it. Namely, they have hundreds of billions of dollars in annual advertising revenue that they have proven willing to sink into development of Frontier AI models. They also own part of Anthropic, which gives them exposure on that front.

19:42And they have some of the biggest supplies of compute in the world. But they just, instead of focusing on the consumer side, they kind of focused on the enterprise side building out Google Cloud. Google Cloud is growing like crazy. It's the fastest-growing cloud platform. And obviously they kind of probably use some of the compute they had towards growing that. Maybe they don't care about the consumer side. I don't know. I think they do care about the consumer side. But as I've reported in the book, like there are just Google is a big place. There are a lot of worrying factions inside of Google that are always fighting with each other. Always have their own priorities. And so I think that battle has yet to be won. Obviously, people inside Google are paying attention to what's going on at the other labs. And I think they have this sort of tension between do we want to build popular consumer AI products right now? Or do we want to spend our time and energy and compute trying to achieve this like recursive self-improvement loop through advanced coding and AI R&D that can then sort of catch us

20:45up with OpenAI and a Theropik on the frontier. So I think there are two different related battles going on. And it's sort of Google's open question whether they care more about competing with meta for a consumer or with OpenAI and a Theropik for enterprise. Yeah, that's going to be very interesting. And I'm really curious to see what their stock price does as well. They've raised a lot of money. They've taken on a lot of debt. And so they're not stopping the investment anytime soon. But we'll see what direction they try to choose to go with. It seems like to me though, more competition is a good thing. So as a consumer, I want all these companies just going all out. I don't want them slowing down anytime soon. Yeah, look, consumer surplus is great. And if these companies are going to war with each other and give us cheap tokens, I'm all for that. I think the danger is that they are racing towards something that they don't understand and that they're not confident that they can control. So that's what I feel like. That's when I feel like it gets like I want them to race in so far as it produces cheap

21:46prices for consumers. I don't want them to race if it creates a novel pandemic or a recursively self-improving super intelligence that we can't control or a world-class cyber attack machine that just pops up in random places and hacks into hospitals and electrical grids. Like I think for all those reasons, it would be a good thing if we slowed down. Well, right now the mark, I mean, what would that do to the market? I can't imagine. Right now the market is just pushing these companies to keep going. And I'm really curious to see what Anthropics S1 looks like. I think that's going to be the next major thing when their S1 comes out. Hopefully soon, what is the, what are their profit margins look like and what are their risk factors? Are they going to talk about human extinction in their risk factors? They should. They should. They genuinely believe it. And they like that would be bad. Human extinction would be bad for the stock price of Anthropic. I'll just go out and say it. I agree. It'd be bad for everyone's stock price.

22:48Yeah. I'll end with this. What just over the next six months to a year, right? I know it's so hard to anticipate what that's going to look like even six months from now. But do you have any hot takes or do you have any like things that you're keeping an eye on? I'll give you an example. Here, one of my hot takes was, and my producer, Mike's hot take was that Sam Alvin was going to get fired. It doesn't seem like that's going to be the case. But at the time, I'm like, that could definitely happen. Do you have any interesting things that you're thinking about where like no one else is paying attention to? I mean, I think, I think just everything will continue. The scaling laws will continue. The models will continue to get better. I do think that we're going to have some kind of unprecedented government intervention, not because the Trump administration wants to do this. They clearly don't. But I think the models will become so good and so capable and so dangerous in the hands of our adversaries that there will be this kind of, I've called like a soft nationalization of one of the AI companies or maybe two or three of them where you have sort of the government

23:52saying, we actually don't want you building this stuff even internally because you obviously can't control it. So we're just going to kind of shut you down. The NASDAQs are going to drop like 12% the next day if that happens, that's going to be crazy. I mean, we will see. But I think these people who sort of exist in this fantasy where the models are never going to improve and they're never going to pose any risks and it's all upside, I think that fantasy is going to die in the next 12 months. My hope is that AI is just like, you know, it's like a really cool technology and it helps people become more efficient and yeah, maybe it like disrupts some jobs. But at the end of the day, it's like it's just another piece of technology and like, cool. And hopefully we're not talking about extinction and things like that. But again, that's the optimistic side of me. I will say something to you that I've said to many AI researchers, which is, I hope you are right, but I fear you are not. Well, I appreciate you coming on Kevin.

24:52I know you have, you know, you're a very busy man. You got a book out. What was the name of the book again? The AGI Chronicles. When is that coming out? October 6th. Awesome. Well, I'm looking forward to reading that and then you have a new podcast and you show called Machine Gods with your co-hosts and how is that going? Where can people find that? It's going great. We've just announced a partnership with NPR. So we're going to be on the radio as well as in every podcast player and on YouTube. People can find the links at machinegods.fm. I was a big fan of Hartford that you did with Casey Newton before. So looking forward to checking out Machine Gods and good luck with that. And again, I hope that, I hope when we're having this conversation a year from now, it's a little bit more jolly and a little bit less demerism. All right. Thanks, Beth. Thanks, Kevin. Well, all right. Guys, hope you enjoyed that conversation with Kevin Ruse. I thought this was a really interesting one. Like I said, I'm a tech nerd at Hart. So I love hearing about what these tech companies are working on and the latest developments in AI. And look, I'm still optimistic that AI won't lead to human extinction.

25:53Let me know what you guys thought about the conversation. Did Kevin change your mind at all about AI safety? Where do you guys land on that debate? Drop your thoughts on Spotify and YouTube. And while you're there, consider giving us a five-star rating as well. All that engagement really does help us out. And it helps other people find the show. Thank you guys again for listening, watching and commenting. Shout out to Mike. Shout out to Mike for all the work behind the scenes. And we'll see you guys back here tomorrow.