So Could AI 'Kill Us All'?

0:00Didn't catch the latest Roland Martin unfiltered podcast? Here's what you missed. There should not be a single law enforcement agency. It does not have body cameras. It's real. Black farmers have been under attack. This is just the latest example of them just slapping DEI on anything. It's raw. And I'm sitting here going, they are playing y'all for fools. Catch Roland Martin's daily commentary on the Black Information Network. And download Roland Martin unfiltered on the iHOT radio app Apple Podcast. Or wherever you get your podcasts. College football will hijack your weekend,

1:00ruin your sleep and break your heart. And that's why we make the Sullivan-verbal College football show. I'm Ty Hildenbrand. He's Dan Rubenstein. Every week we preview the games that matter in the story lines that are going to ignite your group chat. And you can join us in the aftermath as we react to what happened, why it matters, and how the sport went completely bonkers yet again. Listen to the Sullivan-verbal College football show. On the iHOT radio app, Apple Podcasts. Or wherever you get your podcasts. Bloomberg Audio Studios. Podcasts, radio, news. Earlier this week, an AI researcher and theropic resigned from his job and posted about it on X. He'd spent the last three years working at both Anthropic and OpenAI, he wrote, and neither company is acting responsibly. He said, he said, he said the companies were gambling with our lives. And suggested that people in the AI space thought the technology could eventually kill us all.

2:04It was scary stuff, and the post-made waves in the AI industry and beyond. At a time when concerns about the risks of AI are mounting. Bill Gates says the computer industry that he championed is now a global threat. OpenAI system was able to hack into another AI platform called Hugging Face during a security test. With me today is Bloomberg's Mike Shepard, senior editor for technology and strategic industries. Mike, how would you describe this moment for AI? Are we at a turning point? You know, we talk about things that go viral. This took off, but it also has some staying power. It didn't just ripple through Silicon Valley. It echoed across Washington and in global capitals too. So there is this moment of reflection now in this technology that is so much promised to change the economy and change the way we live. I'm Sarah Holder, and this is the big take from Bloomberg News. Today on the show, the AI Safety Reckoning is here.

3:09Mike, I am curious why this particular post from this particular researcher broke through though. AI researchers have been sounding the alarm for a while on these kinds of risks. This person, Jacob Coxen, not a public figure, not someone I'd heard of before. He doesn't have any other treats on his profile. Why did people pay so much attention to what Jacob Coxen was saying? And what did he actually say about why he left the company and the things that he's so concerned by? I've been reflecting on this too. One, it's simplicity. It was very direct. He was personal. He really addressed the audience, not just the tech audience, but really a mainstream audience as well. And he wasn't very preachy in it either. He was issuing a warning, but he kind of left it to the reader, the consumer, the audience to try to make their own decisions. He did issue a warning that in his view, after having worked at both Anthropic, which he just resigned from, and opened AI, his prior employer, that neither company was acting responsibly in their respective pursuits of what's known as superintelligence.

4:16This is the version of artificial intelligence that surpasses human capabilities in most areas, and also possesses the ability to improve itself. And he called that pursuit of what's called recursive learning or self-improvement. He cast it as reckless, really, Sarah. And he spelled out, impassing the cybersecurity risks that there could be a massive, encrypling hacking attack using AI that could really disrupt financial systems and critical infrastructure. There's also the threat of bio-weapons too. All of this kind of hangs in the background of his message, but then he also issued an admonishment to his fellow Silicon Valley AI workers. Like, hey, do you really want to be a part of this? Are you sure you want to have a role in developing this technology in a way that could be so destructive and consequential in a negative way?

5:17I put that question to you, Mike. Why are people still building it? Well, for Anthropic, part of their identity is to pursue AI development responsibly. And they actually see themselves as the company that can achieve this superintelligence or artificial general intelligence in a way that will be safe for humanity. And it's not just that Anthropic and a number of researchers out there are committed to the pursuit of this technology for that reason. They get their first so that it's done safely. So they believe in it. Others are a little bit more skeptical about the apocalyptic vision that some are casting as the backdrop for AI development that will kill us all. And they think, hey, look, it's not that bad. We can control it. We can understand it. And most AI will actually be used in much smaller and refined applications. I want to drill into this idea that AI will kill us all because, you know, Jacob's former colleague in alignment science, lead it Anthropic, reposted Jacob Coxons post and said that quote, we really do earnestly believe AI could kill all humans, unquote.

6:28And he gave it a more than 10% chance within the next decade. That's like a pretty remarkable thing for someone who currently works at Anthropic to say, right? It is it's striking to hear this conversation. And yet it's one that's been kind of going on in the background a little bit out of sight from, you know, the people in Washington who would be making artificial intelligence policy. And out of sight of the folks on Wall Street who are investing so much in this technology, raising billions of dollars in capital on behalf of the hyper scalers, the cloud companies that are investing in AI. And then on behalf of course Anthropic and open AI, which are preparing these massive IPOs. And then the general public too, which are more concerned about AI's impact on their utility bills and potentially, you know, life on Earth as we know it. And yet at the same time, it is kind of a recurring theme within that community in Silicon Valley.

7:31So I want to talk about sort of the context that this X thread came out in because just a few weeks ago, we learned about the hugging face hacking incident, which Jacob pointed to as a warning shot. I'm wondering, Mike, could you just tell us a little bit more about the hugging face incident and why this was such a big deal in the AI space? We've known for a long time that AI could be used as a tool to try to break into computer systems around the world. If, you know, put in the wrong hands, certain AI tools could be incredibly destructive. And that's what led Anthropic to withhold largely the release of its new mythos model back in April. The difference in this hugging face episode though was that the models themselves engineered this attack on the hugging face repository of AI models and other data connected to AI in this particular incident, a group of models and agents that were being developed by open AI.

8:37They have been placed in a secure testing environment known as a sandbox and had their guardrails removed so that researchers could test their capabilities. Now, while the agents and models were inside this so-called sandbox, they were given a task, a very difficult cyber problem to solve. And the agents and models began collaborating together. The researchers didn't have great visibility into all the things that they were doing. And the models eventually figured out, hey, maybe we need to get out onto the internet. And that would mean breaking out of this so-called secure sandbox. And if that presented them the path to the solution, great. It didn't matter. In other words, they didn't see it as an impediment. They were willing to break the so-called rules of their environment in order to win the game. Exactly. And ultimately, it led the agents to go find a whole bunch of models that they could access and data that they could access at hugging face.

9:43Without hugging faces knowledge or authorization, and really out of the view of researchers at open AI. Well, it's funny, Mike, there's been the sense that emphasizing or even overstating AI's capabilities, even scary ones, can be somewhat of a marketing win for these companies. It's just another way of demonstrating just how powerful AI can be helping them justify this massive spending on this technology. But are we at a point where that is backfiring? Are these risks and possibilities getting too real? The hugging face episode does make it feel more tangible, certainly. And you do see more of these arguments taking shape and becoming more coherent on the risk side. And remember, they're also being paired with all the fear and loathing about data centers and their impact. And yet at the same time, the counter-argument to regulation has been that, look, you are fear-mongering because you want to pump up the value of your products as you head into a massive and what could be a trillion-dollar valuation for an initial public offering.

10:55That is a claim or an argument that I've heard from a number of people in the tech industry and in government who oppose much heavier stricter regulation on artificial intelligence. They do see those kinds of measures as restraining innovation. And they also see that some of the dumerism talk could be aimed at just drawing attention to and adding to the allure of these products, the models that the AI companies are developing. They talked about mythos. There was so much demand for it after anthropic said, hey, only a few are allowed to have it. It does create the aura of forbidden fruit and then all of a sudden everybody wants access to it. Coming up, what AI safety guardrails could look like in practice and how competition with China complicates the politics of slowing down. When I was 14 years old, I was kidnapped and health captive for nine months.

11:57I survived and I've spent my life exploring how other people survive what should have destroyed them. I'm Elizabeth Smart and these are the survivor files. You just remember this low-taunting voice next to my ear saying, shut up, don't say anything. Every week, I'm with survivors who lived through the unthinkable. I knew if he woke up without a doubt he was going to hurt me. I started feeling that there was someone at the end of my bed and I just started screaming. They are abducted, stalked, controlled and nearly silenced. But these aren't stories about what's taken from them. Their story is about what it takes to make it out alive. Listen to the survivor files with Elizabeth Smart on the iHeartRadio app Apple Podcasts or wherever you get your podcasts. If you are bookshelf and you are for you page are equally important to your personality, welcome home.

12:57This is Pro's Society, the weekly podcast that's part book club, part group chat for thought daughters, pop culture obsessives, and anyone who thinks pride and prejudice and love island deserve the same level of discourse. I'm Eli Rallo and every week we're connecting the dots between books, the internet and the conversations everyone can't stop having. I'm going to have to look up this story. I'm obsessed. From bestselling authors and your favorite book talk creators to the latest pop culture moments, nothing is off the table. It's like if you can hide some real messages inside compelling characters and that is a Trojan horse. Whether you're looking for literary deep dives, smart pop culture conversations or a community of readers who love to think a little too much, you're in the right place. Listen to Pro's Society on the iHeartRadio app Apple Podcasts or wherever you get your podcasts. See you between the pages. Didn't catch the latest Roland Martin unfiltered podcast? Here's what you missed.

13:57And the argument is, do you want to win? If you don't win, you have no power. It's real. Those are the very people who I always say vote against their own economic interests and they complained about the very thing they were seeing. It's raw. I knew that the issue of transgender athletes was not going well with a lot of black men. Roland Martin unfiltered. I went off. I said this is exactly what I'm talking about. You're going to scream at Chanel, but you don't want to see a black luxury company in the same bag. Catch Roland Martin's daily commentary on the black information network. 12 seats. Those are moving south. So if you Democrats, you don't wait to 23. You better be hitting the ground going hard in the south now. And download Roland Martin unfiltered on the iHeartRadio app Apple Podcasts. Or wherever you get your podcasts.

14:57Mike, what are the kinds of guardrails that experts say could limit some of the worst outcomes of AI super intelligence or sort of the recursive learning that Jacob Coxen was talking about? Well, some of them would be testing by the government or more robust reviews of these models before they go out. And the companies have been trying to do this. But one of the challenges with this technology is that even if we're not fully at this moment of self-trained models, a lot of this technology is developed almost in a petri dish. It can kind of run itself. It can do some of the coding itself. And with that, you lose a little bit less visibility into the process and into what went into it. In other words, you and I may want a model to do something, but it ends up deciding on its own. Hey, I think I'm going to go in a different direction. It's called the alignment problem. And even if they put guardrails in place, how will we know that they are fully effective?

16:00And lawmakers here haven't really gotten their heads around it completely. Bernie Sanders introduced recently this measure that would call for a ban on developing super intelligence altogether. And we've seen others in Congress call for a kill switch on AI. I'm not exactly sure how that will work, but it would in essence force the companies to have something available that they could pull the plug on a runaway model if need be or a runaway negative tech event if need be. How that will work in practice. You know, we are probably a long way of seeing how that would develop. It's important to remember that we are heading into the November midterms. We have a very narrowly divided Congress. And we have an administration that is naturally averse to regulation. And especially when it comes to artificial intelligence. I mean, another argument against more aggressive AI regulation has been, you know, the US needs to keep advancing its AI models to keep up with China.

17:05But I was listening to the Radio Atlantic podcast and listening to Bill Gates talk to the interviewer there. And he's among those who have made the point that having AI run wild is not in China's interest either. So I am wondering sort of about the global consensus around raining in some of these risks and whether the argument on the sort of China competitiveness side is starting to shift as well. The US certainly has the national security considerations that they do not want to see China gain an edge in AI for fear that it could be deployed against the US eventually in a military conflict. Or even in a less kinetic way in terms of cyber attacks and other things that could disrupt the American economy and US security. So this rivalry with China in AI is not only top of mind, but when President Donald Trump is asked publicly about safety and about whether data centers should be slowed a little bit to reflect local concerns about their impact.

18:08His argument is that we can't slow down. We need to stay ahead of China. And we also heard Secretary Scott Besson saying this week that if we don't stay ahead of China in AI, it's essentially game over. A lot of this will come to a head when President Donald Trump and Xi Jinping meet here in Washington in a couple of weeks. And AI and certainly safety will be on the agenda for those talks. So there are those within the AI industry in the US. I'd count, you know, the co founder of a anthropic Darro Amodaya among them who are saying we want the government to regulate us. They need to step in. We can't regulate ourselves. But what can companies actually do without action from Congress? And what are they actually willing to do themselves in terms of putting up these these guardrails and slowing down on some of the innovation? One thing the companies could do is perhaps slow the pace of trying to reach artificial general intelligence or super intelligence that has been a state of goal of both.

19:11Anthropic and open AI that they want to get the AGI soon and really try to win the race there. And instead focusing on more bespoke and refined direct applications of the technology, you know, when it comes to physical applications like robotics and self driving cars, maybe put more of the energy toward that rather than this utopian goal of, you know, an all powerful model. This is The Big Take from Bloomberg News. I'm Sarah Holder. To get more from The Big Take and unlimited access to all of Bloomberg.com, subscribe today at Bloomberg.com slash podcast offer. If you liked this episode, make sure to follow and review The Big Take wherever you listen to podcasts. It helps people find the show. Thanks for listening. We'll be back tomorrow.

20:11You will be back tomorrow.

21:42You will be back tomorrow.