Anthropic Whistleblower Says AI Could “Kill Us All”

0:00I'm Aina Garten. I'm my new podcast Happy Hour with Aina. I'm inviting really interesting guests to join me for a drink and a fun conversation at my kitchen table in New York City. I'll be getting personal with chefs, actors, comedians, musicians, and writers I admire. So grab a snack, pull up a chair, and join us. Subscribe to Happy Hour with Aina on YouTube or wherever you get your podcasts. New episodes will be available every Wednesday, starting September 16th. Have you ever wondered why a band as big as Fleetwood Mac would break up?

1:04Well I'm Will Anderson and this week on My Music History Podcast, The Monday Music Club, we dive into that question and all the drama surrounding Stevie Nicks, Lindsey Buckingham, and the rest of Fleetwood Mac. Jump in and listen to this week's episode. All you have to do is search Monday Music Club right now wherever you get your podcasts. If money is evil then that building is hell. Show them some! They're not going to watch the show. Show! Welcome to Profty Markets. I'm Ed Elson. It is September 10th. Let's check in on yesterday's market vitals. The major indices fell as fighting in the Middle East escalated. Brent Crude topped $101 per barrel for the first time since May. The yield on tenured treasuries hit its highest level since 2023. After the treasury announced it is tripling debt bybacks to $6 billion. On Calche the odds of a rate hike next week climbed to 57%.

2:06And finally, MetaShare's rose more than 6%. After the company released its Muse AI agent. OK. What else is happening? A mass extinction event could be upon us. A former anthropic researcher announced his departure from the company on X late Tuesday night. He revealed that employees at both OpenAI and Anthropic believe that AI could quote, kill us all by the end of the decade. He went on to say that at OpenAI many have not quote, deeply internalized the civilizational stakes while at Anthropic the stakes are quote well understood but they are locked in a race. He ended the message calling for an increased amount of safety cooperation between labs urging other researchers to do the same. In response to this message another current anthropic researcher chimed in saying that he is quote correct and that anthropic researchers quote really do earnestly believe that AI could kill all humans. They also put the chances of that happening within the next decade at greater than 10%.

3:11So is AI really going to kill us all? It helped us break it down. We're speaking with Alex Cantruetz author of the big technology newsletter and host of the big technology podcast Alex. Thank you for joining us on the show. This tweet has gone insanely viral. It now has about 100 million views and that's in less than 24 hours. This guy is basically saying that anthropic and OpenAI developing technologies that are going to kill us probably maybe in the next few years. What do you make of it? Do you think he's right? Well, I don't think he's right. I mean, I would say that it obviously has garnered a lot of attention because some of the AI activity we've seen might lead you to that conclusion. We've obviously seen over the past couple of weeks swarms of AI agents doing things like solving millennium prize problems. In OpenAI had thousands of bots that attacked and infiltrated hugging face against its wishes. So we're clearly seeing some crazy activity from bots right now.

4:13But as to the question of will AI kill us all? I strongly on the side that it will not kill us all. Now could it sort of take on financial institutions and hack into companies? Absolutely. But at least for now, AI needs us to exist. Like right now, we're taking so much of the economic activity that humans create and using it to build data centers and power the data centers and feed these bots. So I think offing us would be a terrible decision on their part. And I think they know that. So this idea that they're about to kill us with now or within the decade is wrong. And unfortunately, it really muffles the message of the AI safety advocates because there is a there there. It's just not this take all humanity and turn us into paper clips there that I think a lot of them believe in. Such a whilst able to see that there's a 10% chance that we're all going to die. And also to have that endorsed by someone who's actually a leader at the company right now saying, yeah, we actually a lot of us think this. A lot of us think it's actually even greater than a 10% chance that this stuff is going to

5:13kill us all. Why are they saying that? I mean, and what are we supposed to make of that if they're kind of just putting it out on Twitter that the technology that they are building might destroy humanity. Like is that something that we should even remotely be taking seriously? Well, I'm taking it more seriously than I did in the past. I think previously a lot of this sounded crazy to me. And I think the reason why these people are saying it right now is because many of them were early to this. So before I could form coherent sentences, they saw that if you were to give it more computing, more data and make the algorithms, make the models bigger, that they would improve. And everybody else thought that that was ridiculous, that they would never get to the point where they could actually speak like humans and now they are. And so if you were at this point where early on, you thought that the AI technology was going to improve and keeping it aligned with human values was going to be a challenge. And clearly that's proven to be the case. I mean, the AI's are doing things we don't want them to do.

6:16Then sometimes in your brain, the next logical conclusion is, well, eventually they're going to want to take over the world and kill humanity. So we won't be in their way. But to me, there's a logical hole in that thinking, which is that if AI's want to take over the world and get all the resources that the world has to offer, you have to work hand in hand with humans. You don't take them out of the picture. So maybe there's a way where they co-opt our species and have us continue to do everything we can to build data centers and power them. But we're already doing that. So why kill us? I just don't see where that ends up being the place where they end. A lot of politicians are using this moment to advocate for pausing AI progress. This has been advocated for by several politicians, Bernie Sanders has been one of those vocal advocates saying that we basically need to stop or put a pause on this. And to be fair to him, suddenly this researcher says, yeah, this might kill all of us.

7:20So it's a pretty decent argument when you think about it. What do you think is the next chapter in the AI regulation story? A lot of people have been wanting data centers to be banned, fly it out, AI to be paused. Do you think that that is the next part of the story? It's absolutely going to be a major issue in the 2026 and the 2028 elections. There are going to be politicians that are going to want to say, listen, we can't live with this stuff. So let's pause it. I think for AI, if you're rooting for AI to continue, especially if you're on the business side of things, in good news, a lot of this is localized. So it's municipalities, it states, it's cities. It's not necessarily something that's happening on a federal level and I would be stunned if we saw a pause on a federal level. So there might be some obstacles that come out of this for the AI companies looking to build in certain locations. They should be able to overcome those obstacles. But that being said, I think it would be unwise to underestimate just how big of a political

8:23issue that this has become. And the fact that some of the staunchest allies of AI, I'm talking specifically about their public and party in Texas, they've now become enemies of AI. And if that continues to be the case, this little thing could spiral into something bigger. But my prediction right now is that we won't see AI stop in the United States as some worry, but it definitely is going to become a little bit more difficult to build than it is right now. Alex Contrists is author of the big technology newsletter and host of the big technology podcast. Alex, always appreciate your time. Thank you. Thank you. John Turnis opened his 10 year ago of Apple with the company's biggest change to the iPhone in nearly a decade at Apple's surprise and shine event yesterday. The company unveiled a $2,000 foldable iPhone called the iPhone Duo.

9:24It's about the size of a passport with a display 80% larger than the most recent iPhone. Apple also announced the iPhone 18 and the 18 pro, the AirPods 5 and the new Apple watch series. The event was John Turnis' first product launch since taking over from Tim Cook on September 1st. Apple shares closed roughly flat. So joining us to discuss the beginning of John Turnis' term and the foldable phone. We are speaking with Patrick McGee, award-winning journalist and author of Apple in China. The capture of the world's greatest company. Patrick, welcome back. Good to see you. Let's just start with your reactions to a $2,000 foldable iPhone. The iPhone Duo. I mean, look, I think it's a marvel of industrial engineering and hardware engineering. I mean, it looks fantastic. I've been able to talk to some people who have been playing with it today in Cupertino and they do say it's fantastic. Obviously, the question is, are people going to pay $2,000 for this?

10:24I think the two scenarios are either that it underwelms or that Apple is already sort of baking in that it's not going to have the high sales. And so it's going to meet low expectations, as it were. You know, the thing I think about is when Steve Jobs introduced the iPad in 2010, he was able to credibly market it as a entirely new product category, saying it's more portable than the desktop, but it's bigger than an iPhone. And that felt intuitive and they've sold hundreds of millions of them since. Because the iPhone Duo trying to position itself like that, that's the sense I got from the new CEO, John Ternas, but I'm not really convinced. I mean, I'm someone that has an iPad with a keyboard. And so if I need to get a little bit of work done that's not on the laptop, that's what I'm going to do. If you, if you're in that situation, why would you also have an unfoldable phone? I just don't know what service it serves, even though it's cool. I think the question for Apple is, is this their next generational product? And that's something that a lot of people have been waiting for. It was supposed to be the Vision Pro, or at least that was the way it was kind of marketed.

11:27The Vision Pro, based on my understanding, has been something of a disaster. They don't really report it properly in their ownings. I think because it's not doing well, I mean, what legacy do you think this product will have is this their ticket? Are they betting on this thing being the next era of Apple? I don't think they are. I mean, to be clear, I think the 18 pro, the new version of the best iPhone will do just fine. And people will buy that. I'm probably going to buy that. So I mean, for them not to have market hits isn't to say that they're in any way failing. The closest sort of precursor we have to this is the iPhone Air last year. Now it didn't do well in terms of sales, but it sort of allowed them to really push what the iPhone Pro was doing, right? So they could sort of have this compare and contrast. And really sort of like, if you were someone that complained to the iPhone Pro who just got too big and durable looking and you didn't need that many cameras, it was like, look, here's something that as they would say is later than air.

12:29And even though it didn't sort of work in terms of sales, I do think it worked in terms of the marketing pitch. And you're right, Apple has been so dogged by these reports that it doesn't really innovate anymore that coming out with this product sort of gives a halo to the company, even though I don't think any people are going to buy it. So by saying it's not going to be a big sales hit isn't to say that it's a mistake to have it in the market. What do you make of John Ternas' start? How do you find him as a communicator, as a leader? And what kind of direction do you think he's going to take this company? You know, I'll be shocked if anybody says they feel like they learned a lot about John Ternas, either as a person or as a communicator, let alone as a leader today. And that's because Apple, since COVID, has done away with keynote presentations, live keynote presentations. It was a big deal when Tim Cook had to take over after the passing of Steve Jobs and really do his first keynote. You know, you could tell he was nervous. It's still up there on YouTube if you want to follow. There's nothing like that for John Ternas. Who knows how many takes took place before he sort of nailed it, right?

13:30He wasn't talking in front of a live audience other than sort of waving a little bit at the start. So I get why Apple has these like cinematic experiences. Steve Jobs never would have tolerated that, right? And to be clear, that's not necessarily criticism. I mean, it's not Steve Jobs' company, obviously. And so they're able to pack in more information and they really de-risk things. They don't have, you know, hundreds of reporters around the world all talking about some flop, right? Some error that took place on stage. Like, there's all sorts of reasons why it makes sense to have a cinematic thing. But they've just sort of taken the, like the oomph out of it, right? You sort of want to have the live demo. And it's really interesting that OpenAI does do these live demos, right? It's a little sort of a little quirkier. But it's almost telling you that like a startup can take those kind of risks in a way that a Goliath cannot. He has a lot to live up to. I mean, his predecessors were Steve Jobs and Tim Cook. Two very different CEOs. One, a legendary visionary founder, the other kind of, someone's a boring operator, but who

14:36also created more shareholder value than I think anyone who took over as a CEO ever in history. I mean, what kind of CEO do you think he will be? Do you think he will be more of that visionary product person or more of an operator who just keeps the trains moving? I kind of think neither. I mean, to be clear, Apple is going to keep the trains running on time. I just don't think it's John Turner. So it's going to be making that happen, right? Sabi Khan is the chief operating officer. He's been at the company since the late 1990s. He's an accolade of Tim Cook. He knows what needs to be done and he can get it done. What's going to be more complicated is if they really do position out of China and grow their capacity in India, right? So far, that's more hyped than reality in terms of what needs to be done. And Tim Cook is basically staying on to be negotiating with New Delhi, Beijing, Brussels, and Washington. And so to a certain extent, I don't think we're going to see John Turner's make his mark. Certainly not in the geopolitical scope of things. For several years, either until Tim Cook has sort of just moved on a little bit or has

15:39officially retired from the company. So I know that everyone wants to sort of put the first performance of John Turner's on a pedestal and sort of ask what this means for the future of the company. And I just think, anticlimactically, we're actually not really going to know what Turner's impact is for several years, right? Not a very good sort of media headline answer, but it's still Tim Cook's company and tell something dramatic changes. Like not to go off on a massive tangent here, but Tim Cook fired Scott Forstall, who was kind of like a mini me of Steve Jobs roughly 15 months into his term. That was his like new Sheriff in town, you know, template or approach. And it's going to be tough to see John Turner's make any sort of move like that. In other words, something where Tim Cook wouldn't have done it, but John Turner's would, you know? I don't know that we're ever going to get the board minutes about that sort of decision. And until we have that, how do you really judge whether it's Turner's company or not? Final question. Will you be buying the iPhone? You know, you know, if we spoke yesterday before the, before the product had even come

16:40out, I would have said yes, just as a tech journalist, like I want to, I want to be, you know, be familiar with how it works and everything, right? And now I've seen it. And even though it actually sort of works better and looks kind of cooler, like the continuity feature of how you open it and it feels like as impressive as the hardware. And yet when I'm really thinking about it, you know, I don't want the base model. I need to double the storage. It means I'm buying a $2,200 device. And I just don't think I'm there. I just don't know what purpose it serves in my life. I think it looks really nice, but it looks too big. I'm not sure about the size of it. Putting that on your pocket in your pocket every day. I'm not sure that would be worth it to me. But we'll see. Patrick McGee is journalist and author of Apple in China. Patrick, always appreciate your time. Thank you. After the break, more evidence of AI's concentration risk.

17:42Support for the show comes from granola. There are two types of people on a video call. There are those who are fully engaged in the moment, only to realize they have no idea what exactly there's just agreed to next. Then, there are the note takers who are so fixated on getting everywhere down that they barely participate in the decision making at all. Good news. Granola has solutions for both. Granola is the AI powered note pad that makes your meeting notes useful. It captures what happens in your meetings and turns it into clean, structured notes with decisions and actions pulled out and made easy to find. The best part? Granola can integrate seamlessly with how you already work. So there's no setup, no configuration, no friction. You get to stay present in your meetings and still get clear actionable notes without any extra work. Once you try it on a first meeting, it's hard to go without. You can try it totally free just head to granola.ai slash markets. That's granola.ai slash markets to get your time back. Go to granola.ai slash markets and try it on your next call. Support for the show comes from NetSuite.

18:43Successful business owners know how crucial it is to keep up changing technology. So, when it comes to adapting AI for your business, it couldn't be easier with NetSuite next. Probably no NetSuite. The AI Power Business Management Suite that securely connects all your data, eUnits your financials, inventory, commerce, HR and CRM into a single source of truth and it's trusted by over 43,000 customers. NetSuite next is the next big leap in how business gets done because AI is built into everything you do. It automatically surfaces custom insights throughout your day. AI agents work alongside you to solve problems and handle routine work. And anytime you have a question about anything, just ask like you're talking to a colleague. Whether your company earns millions or even hundreds of millions, it's time for NetSuite next where your business meets AI. For the first time ever, you can try NetSuite next for free. If your revenues are at least in the seven figures, then a NetSuite.ai slash propG. Build for every industry, ready for every boardroom. NetSuite.ai slash propG.

19:44Support for the show comes from AG1. Now that it's fall, everyone's calendar is about to fill back up again with back to school sports and earlier alarms. When things get busy, it's easy to let good habits fall through the cracks. But here's a suggestion for a healthy habit. You might actually enjoy AG1. Just one scoop and eight ounces of cold water in the morning can help you maintain energy, support gut health, and a mute system all day long. And the next gen formula delivers more than 75 ingredients clinically shown to support gut health. Fill common nutrient gaps and improve key nutrient within three months. I use their little sticks when I'm on the road just to kind of shore up what I think I'm missing is when I'm on the road, quite frankly, my nutrition sort of sort of falls down. AG1 can help you keep one thing consistent. High quality nutritional support every single day, no matter where you start your morning. For a limited time, say 20% on your first subscription order of AG1 next gen or AG1 pro at drinkag1.com slash markets. That's drinkag1.com slash markets.

20:45We're back with Profty Markets. We're months away from what could be two of the largest IPOs of all time. Both open AI and anthropic have confidentially filed with the SEC to go public, but still, we know almost nothing about these company's finances until investors can see those filings. Many questions remain about the business of AI. Our next guest though might have some answers. Corporate finance platform ramp updated its AI index yesterday tracking how much American businesses are spending on AI. And the results are quite staggering. According to the report, about 80% of all enterprise revenues at open AI and anthropic come from just 1% of businesses. And those businesses are now pulling back on spending. So here to break down this data, we are speaking with our a carazzy and the lead economist at ramp or a thank you for joining us on the show.

21:47You've just published this index and the report is titled quote cracks in the AI thesis. And this number 80% of enterprise revenues coming from just 1% of the businesses at open AI and anthropic seems to be a big part of that. Could you walk us through that number, what it means and why it's important? In any case, when you're analyzing these kinds of new and nation technologies or analyzing anything with the data, that you want to avoid focusing too much on one specific metric, it's very natural to see some noisy data and you don't want to overly highlight one trend. What's different about this is that we are seeing a number of metrics now move in a negative direction for the AI model companies. And we are seeing a great amount of concentration risk as far as their customer base. So if you are someone who is involved in the AI trade, that might be something that you want to take a look at that I think the market is relatively underpricing. So you're right to point out a lion's share of the AI companies enterprise revenues is coming from a specific segment of customers, 1% of customers driving 80% of that spend.

22:53Bigger problem is that within that, those companies tend to be very correlated with each other. It's by and large high growth tech companies other AI startups as well. Now that's fine as long as AI spend continues to increase. But then our latest finding today is that the top 1% of spenders in terms of per employee, per month spend are also slowing down their spend. So came down about 10% month over month from 8K per employee per month down to 7.2K. Still a lot of spend. But then within that, you start to see a couple other negative trends. People can see that metric and they think, oh, it's because of summer seasonality. And that is somewhat true, but it is not a full explanation. Token volume and usage of these models actually rose in the same period. And there are a couple other metrics that then inform, well, why would spend trend down? One is that open an anthropic are in this price war.

23:54So both introducing frontier models and cutting prices for them, trying to get the best model at the best price, simultaneously introducing highly performant cheaper models at the standard and light tier. And then you see an increasing share of that volume of AI spend volume by businesses going toward the standard and light models as opposed to the sort of frontier models. So businesses deciding they get pretty good ROI performance from Saunet and Terra over Astra, Opus, Fable and GPT 5.6 Soul. And so you have a couple metrics sort of moving in a negative direction. You describe this as quote, a level of concentration risk unseen in any other software category that we track. How does it stack up compared to other software compared to other businesses that you guys are looking at? Well, we looked at some of the other large spend categories for businesses. So software excluding AI, you still see some power law, you know, but you don't see

24:57it to the extent of 1% driving 80% of revenues. You know, you have to get sort of the top 10 top 20% before you start to see something of that magnitude. Similarly, and something like digital advertising, which actually tends to be a little bit more broad and spread out across a number of businesses. So I think it's normal to have this in a new technology to be clear. And it's not necessarily problem unless if those companies in that customer base are highly correlated and simultaneously end up drawing down their spend and that it appears to be where we're headed. Yeah, I mean, this data seems to be evidence of what a lot of people have been concerned about and have suspected about these companies, which is that the economics of the AI labs don't really make sense unless you have a highly subsidized handful of entities that are providing all of the growth and providing all of the revenue. And that seems to be the case on the side of the big tech companies too, where you have

25:57anthropic and open AI, which are contributing to like 70 or 80% of the AI revenues of these big data center companies like Microsoft, like Amazon, et cetera, like Google. And this seems to be evidence of exactly that. What do you make of the economics of AI at this point, given that you are seeing this data, it seems to line up with what a lot of the AI bears have been worried about. What is the takeaway? I'd rather on the side necessarily the AI bears of the AI bulls, right? You can still see a bit of a bullish thesis come through here, which is that open and anthropic will continue to draw more and more spend, maybe at a slower rate than what they've seen so far. And most notably that they will continue to draw a greater share of spend toward their models. Maybe it's going to be their standard and lighter models, which are lower margin, but they will get higher volumes there. And that's an important part of the thesis, right?

26:58Because lately a lot of the concern has been about the threat of Chinese model companies or open source models, eating at the market share of open AI and anthropic. And I don't think that's necessarily going to happen. I do think open and anthropic will be at a very good place to command enterprise market share for AI at least in the United States for the foreseeable future. It's just likely going to be shifting over towards standard and lighter models rather than the frontier models that have generated so much of the attention so far. They will continue, I assume, to rely on a handful of customers generating the revenues for the businesses. And maybe that continues and maybe that is sustainable. I guess that brings me to another question. Clearly a concentration risk exists for these companies. I think the question is how large of a risk that actually is. Do you have any thoughts on how seriously that risk should be taken, how it stacks up against any other kind of business risk that you might see in an immersion technology?

28:02Well, it's a risk that they want to address separate from their financial concerns about what can happen in a spiral where the market starts to turn down on AI. Part of the reason why other AI companies and tech companies are the biggest spenders on AI models is because AI models are most commercially advanced and most productivity enhancing today for technical professions, for software engineers. It's coding agents. AI sort of works especially well and is diffused especially effectively in these highly technical industries. But that's not necessarily the goal of that technology. If you want to follow the goals of the AI model companies as far as their public statements, it's that they want to see proliferation across the US economy into non-technical work into broader areas of white car work, eventually into areas of manufacturing automation. The reason why we haven't seen that uptake looking part, it's because tech is pretty

29:03laggy, takes time for technology to proliferate through a society. But it's also because the model companies themselves have not yet introduced a sufficiently enticing commercially advanced version of AI that works for the general population. So I think they're working on that but time will tell if they're going to be successful. I just can't wait to see the S1 and finally all of our answers or all of our questions will be answered. Until then, we will keep finding out what's going on through you guys and through other third party data providers. Our Corrosion is lead economist at Ramp Ara. Always appreciate your time. Thank you. Thank you. Let's quickly return to the anthropic story, the story of the AI researcher who quit because of his concerns that AI will, as he put it, quote, kill us all by the end of the decade and also the odds of that happening being 10%. I'm just going to give you my initial gut reaction.

30:07My instinct is that this is a massive exaggeration of the problem. I don't think that there is a 10% chance that anthropic or open AI's technology is going to destroy civilization. And the reason I think that isn't because I'm an AI expert but because I've seen this movie before, countless AI researchers have predicted some form of AI apocalypse over the past few years, whether it was some Outman's prediction that AGI, which we have supposedly now achieved, would quote, break capitalism or Stuart Russell's prediction that it would send all wages to zero or Daria Amade's prediction that the chances of civilization catastrophe were quote, between 10 and 25% or Alisa Yadkowski's prediction that the most likely result of AI was that quote, literally everyone on earth will die. Point being, we have heard this before and in most cases, it's statements like this one that end up getting walked back. And yet every time they come out with a new statement, a new prediction for what AI will

31:10do to harm us, we're expected to simply take their word for it. I guess because we're the ignorant ones and they're not. They know what's best. That is my gut reaction as to what is happening here at the same time though. It would be grossly negligent of us to not take this warning seriously to simply write it off because we had a hunch that it was clickbait or doom porn. That would be irresponsible. We would be risking millions of lives. So here's what we have to do. We have to take it seriously. We don't really have a choice. Now some people think that taking it seriously simply means accepting their claims, accepting their predictions is true. That's not what it means. Taking it seriously means thoroughly investigating exactly what they're saying. It means conducting a federal investigation into what they're talking about, bringing these researchers to Washington for public testimony in front of Congress. It means handing anthropic and open AI subpoenas to give us documentation and evidence as

32:12to what they are claiming. After all, it wasn't just an ex-anthropic researcher that made these claims. It was a current employee as well. Anthropics, very own leader of alignment science, endorsed those comments. And they even said that most of the company agrees that their technology might end society. This was a corporate communication from Anthropic. It was material public information that was relevant to investors, relevant to society. And therefore, it needs to be explained. Now if we find that their claims were bullshit or they were exaggerating or it was hyperbole, well then, maybe it's time to stop pressing charges. Because guess what? Now we're talking potentially about securities fraud. Now we're talking about misleading the public in a major way. But in other words, we either take this seriously or we don't. But you can't have it both ways. You can't expect us to take your word that civilization might end, but at the same time,

33:12not expect us to investigate your claims with the full force of the government and hold you accountable if you are wrong. So you know where I stand, I don't believe these statements, but that's not a good enough reason not to interrogate them. It's time for us to get real about what AI's world ending capabilities actually are. We want proof, we want evidence, we want receipts, and I'm sorry vague statements and hand-wavy tweets are not going to cut it anymore. It's time to get answers. This episode was produced by Claire Miller and Alison Weiss and engineered by Benjamin Spencer. Our video editor is Brad Williams. Our research team is Dan Shalan, Chris Nodonehue and Mia Silverio and our social producer is Jake McPherson. Thank you for listening to Profty Markets from Profty Media. If you liked what you heard, give us a follow. I'm Ed L sent tuning tomorrow for a conversation with Katie Malta.