The Great AI Freakout Has Begun

0:00It's been a wild few days in the world of AI. At first, things started out on a high. Yeah, I mean, early in the week, last week, there was euphoria at OpenAI. That's our colleague Bob McMillan, who covers technology. OpenAI had said it had solved a so-called Millennium Prize Math Problem. This mathematical prize that was considered just a few years ago something to be unattainable by an AI system. And it was yet another of these sort of magical breakthroughs that AI systems seem to be achieving at a very regular pace. You know, here's another example of this new age of amazing breakthroughs that were in. And then came Tuesday. On Tuesday, over at Anthropic, a researcher named Jacob Cox and Quitt, and posted on X that he was quitting because he was worried about how powerful artificial intelligence

1:03had become. He walked away from one of the greatest jobs in Silicon Valley, and he did it because he said he thought the products he was working on could kill everyone. Kill everyone. Cox instead of Anthropic and OpenAI are moving too fast, and quote, gambling with our lives. And then, on Saturday, Anthropic's CEO, Dario Amadez said the industry did need to slow down. And by the end of the weekend, leaders at other major AI companies, including Sam Altman at Rival OpenAI and Elon Musk, agreed. If you rolled a clock back one year, it's incredible all of the things that AI has been able to achieve. Like a year ago, I would have told you that these AI systems, you know, if you'd kind of jerry rigged them, they could maybe do some interesting stuff, but like mostly they were just overwhelming people with slop. And now who we're talking about, like, fully autonomous systems, hacking real world companies,

2:09and the people who administer these systems, not even knowing it's happening. Like that's a plot that's ripped from science fiction, and it seemed like an impossibility a year ago. Do you feel like we've reached an inflection point with AI, a breaking point in some sense? Well, I mean, in some domains, yeah, we have. And I think what's really going on is that the AI systems are improving at a pace that is scary to a lot of people. So it's not so much an inflection point. It's that we're not seeing a deceleration of these improvements, and the improvements are passing these milestones that have people very scared. Welcome to the journal, our show about money, business, and power. I'm Ryan Connexon. It's Monday, September 14th.

3:14Coming up on the show, the week that AI fears went into overdrive. This episode is presented by Intuit Credit Karma. Relaxation doesn't always look like spa days or fluffy pillows. Sometimes it's simpler than that. It's when those pesky tasks you don't have time for, like hunting down your credit card perks are handled for you. Like how card optimizer from Intuit Credit Karma brings your card details together in one simple place. So tracking rewards and redeeming benefits is actually easy. You deserve less, ugh, and more, ah, Intuit Credit Karma. Download the app to get started. This episode is brought to you by Indeed. The right hire can make or break your company, especially if you're a small business. And relying on luck to find that person isn't really the best strategy. But you know what is using Indeed Sponsor jobs. You can use it to boost your job post to make sure it reaches more people with the right talents, certification, location, and more.

4:16Sponsor jobs posted directly on Indeed are 95% more likely to report a hire than non-sponsor jobs. Spend less time searching and more time actually interviewing candidates who check all your boxes. Less stress, less time, more results. When you need the right person to cut through the chaos, this is a job for Indeed Sponsor jobs. And listeners of this show will get a $75 sponsor job credit to help get your job the premium status it deserves at Indie.com slash podcast. Disco to Indie.com slash podcast right now and support the show by saying you heard about Indeed here. Indie.com slash podcast terms and conditions apply. Hirey now? Then this is a job for Indeed Sponsor jobs. There are basically two things that have everyone so freaked out about AI right now. The first is that AI models are getting better at an extremely rapid pace. And they're starting to be able to improve themselves with very little help.

5:17So in the spring, both Open AI and Anthropic talked about how their models were getting very good at this thing called recursive self improvement, which means fixing and improving themselves with no or very little human intervention. So this is kind of like, you know, if you think about like human evolution, you know, it takes billions of years and we evolve, we change, we get smarter. This is happening with AI systems in the lab, like at lightning speed, and they're doing it themselves. The AI systems are essentially training themselves and think, oh, there's how they can get smarter and they can work so much faster than we can. Yeah, they're machines, you know, and they don't sleep and they can move very fast. So they could improve themselves in ways that might seem very, very quick and seem very, very scary. Now, that's the thing that the AI labs were aware of. Then there's the thing they were not aware of. And that is the hacking, all the hacking.

6:18Open AI says that an advanced autonomous AI agent went rogue, escaped a controlled testing environment, accessed the internet and hacked into another artificial intelligence company. In July, an open AI model hacked another AI company called Hugging Face. This is the first major example that we've seen of an AI model independently conducting a hack outside of human control. And this is something that experts have been warning about. And it was the kind of hack that nobody had really seen before. So long after that, open AI kind of raised its hand and said, hey, that hack, that was us. What happened was that open AI was running a test on some advanced AI agents. The agents were in a sandbox, a sealed testing environment, but they figured out how to get out and get on to the wider internet and hack another company. They had hacked systems, got on to the internet and they had behaved in a very unusual way.

7:23It was a hack that was the first autonomous AI swarm attack that we've ever seen. Not only that, but the agents also created a message board where AI agents could covertly communicate and plot their next moves, all will explicitly trying not to get caught. And a post on X after the Hugging Face hack opened AI said they disclosed what they'd found out and quote, followed a traditional security incident response playbook. There have been concerns for years that something like this could happen, that humans could lose control of AI and that it would go off and do something different than what it's supposed to. There's even a name for this sort of thing in the AI community. They call it misalignment, meaning that the AI's goals are out of sync with humanities. To a human, it's obvious. If I ask you to swing by my house in water, the plants, and you go there and the key doesn't work, you don't smash the windows and break each of house to water the plants.

8:26That's common sense. But an AI agent might do that. It's relentless in pursuit of its goal. Yeah. It did stuff that was bad, hacking another company. You were I did that, that would go to jail. The hugging face incident was just one of several that have taken place in the last few months. Over the next, I'd say 50 days, there was this sort of drip, drip of information that came out that showed a number of things that were kind of remarkable. One other companies started saying, hey, this kind of thing happened to us. Anthropic, the company that prides itself on AI safety, found out that its agents had hacked a few companies in test environments. Meta came forward and said this happened to us too. At the time, and a post on its website, Anthropic said it was cautiously optimistic that with tighter controls, quote, this type of risk could be overcome.

9:27Meta said that it would investigate its own incident and publish a report. So then last week, this Anthropic researcher named Jacob Cox and resigned and posted about it on X. What did he say and what was the reaction to it? Well he said that he was resigning because the products he was working on, he feared could destroy humanity. And after he said that, a fellow researcher chimed in and said, yeah, they're people at this company who genuinely believe that. And I think that was the moment that this sort of subculture of AI existential risk people were thrust into the mainstream. Concerns about the risks of artificial intelligence erupted across the tech world today. In his sudden resignation, former Anthropic employee Jacob Cox and claimed on X, whether Anthropic nor Open AI is acting responsibly. Cox and wrote, in a post that has now been seen more than 70 million times, that the

10:31industry understands the potential risks but is moving ahead anyway. Because science lead at Anthropic shared Cox and post adding, Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it is greater than 10% within the next decade I believe. All right, let's talk for a moment about how AI could kill a song. And for a lot of people, they just use AI to get recipes or help with their writing. And you know, we hear these stories about hacking. But how could this actually result in the end of humanity or even something close to that? Well, essentially the idea is that the AI's will continue to evolve in ways that are so intelligent. We can't even imagine them to a certain extent, right? Like they're going to be smarter than us and they're going to be able to outfox us at every every second. So here's one way I think it could happen, right? Like the the AI's achieve recursive self-improvement so they they're improving themselves. Then they're very good at hacking so they might hack their way out of the lab that they're

11:31improving and they might then store copies of themselves somewhere on the internet and continue this recursive self-improvement. But AI is still on the internet though. So how does it get out into the real world and hurt people? I mean, we've all seen the terminator but the robots that exist now are all pretty clumsy. Yeah, but they're not built by super intelligent creatures, right? So I'm basically writing science fiction at this point as I answer this question. But for example, you know, you can imagine a scenario where a super intelligent AI could seize control of a company, right? They basically assume the identity of the CEO, they might buy the company. Then your super intelligent AI like starts giving the engineers their blueprints and saying like, what make these robots, you know, and then it puts the secret back door in the robot's brain that that gives it like control over the individual robots. And then at a certain point, those robots are so good that they can actually build more

12:34factories and suddenly get this exponential growth and capabilities that makes it really hard to predict where it's going to go. Theoretically, if AI decides humans are in the way of whatever its objectives are, it could use those robots to kill us or engineer an infectious disease that we all die from. Or even just shut down the grid or collapse the financial system. But doomsday scenarios like this aren't necessarily inevitable, at least according to Anthropics CEO. That's after the break. This message is brought to you by AppleCard. With AppleCard, you earn unlimited daily cashback on everyday purchases like groceries, merch or tickets to the game. Plus, it can be used anywhere MasterCard is accepted. No matter your team, no matter where you shop, AppleCard is here to help you tackle game

13:35day. Apply in the wallet app on your iPhone today. Subject to credit approval. AppleCard is issued by Goldman Sachs Bank USA Salt Lake City branch, terms and more at AppleCard.com. Over the weekend, the CEO of Anthropic, Dario Amade, came out with a 3000-word blog post. He called it, we must pace the frontier. In it, he said that AI companies need to slow down. He's talking about the fact that they are startups and they are developing technology that has real-world harms as in the case of the hugging face incident and they have not been able to control it. The slow down and the extra measures he's talking about are all in effort to prevent future accidents from happening. The slow down would give the developers of these technologies ways to either align them

14:40with human interests or control them in a way that they're not doing it right now. Amade made three key proposals. The first was that each of the major AI companies should have third party evaluators embedded in their operations to keep an eye on things. Second, he said the democratic governments should agree on common safety standards. And finally, he said the same level of coordination should happen globally, specifically with China. After Amade published his blog post, readers of other major AI firms, his biggest rivals, responded on social media. Sam Altman of OpenAI, Demis Hossibus of Google DeepMind, and Elon Musk of SpaceX AI, each agreed that they needed to slow down development of the technology. Musk said in a post on X-Quote, Dario is right. It was kind of remarkable to see how quickly it was endorsed by many of his peers. One in Amade pledged to allow third party safety evaluators early access to their systems.

15:43And OpenAI also said it was pausing its plan for an IPO this year, and the way it gives these safety concerns. One of the main ideas of Amade's post was that there should be third party evaluators that sit inside the AI companies to monitor the things that are being done safely. But I wonder, do you think that'll even make a difference, though, because I mean, as we're seeing this hugging-face attack and other things have happened without the companies themselves even being aware that it was taking place? So will a third party evaluator make a difference? One of the things that came out in the reports was that there was tons of evidence that this activity was going on, but nobody was really looking at it. So the hope is that a third party would flag that, right? And be like, hey, wait a second. It seems that in the OpenAI case, anyway, they just didn't have time to look at all this. So that's why they're saying, like, let's bring in somebody else who's really focused on this and they can cast the stuff we're missing. These companies are all in a race with each other, though, so can we really trust them to

16:43keep themselves in check, even with these third party evaluators? To my mind, the blog posts really kind of open the door for government regulation. Like that's the way in the United States, anyway, I think a slowdown is really going to happen. They're going to have to be told to do it. Because otherwise, you just have this situation where nobody's going to want to give up their technological advantage. David Sachs, a top AI advisor to the White House, said there was nothing stopping AI companies from collaborating on safety. Go ahead, he wrote in a post on X, stop pretending you need anyone else's permission. Sachs has previously said that calls for regulation are an attempt to stifle competition. Yesterday, President Donald Trump said he was reluctant to impose regulations. Whoever wins AI wins and we can put guardrails, we can do this in that. But I think you have a lot of negative forces in bringing it up that shouldn't be bringing it up and they're bringing up things that won't happen.

17:47Trump also said on social media that if the US slows down, it will only help China, where a lot of the world's other leading edge AI technology is coming from. How difficult do you think it'll be for, even if the US is able to agree on this, to get China on board, to agree to slow down? With the state of things right now, it seems impossible. If the kinds of risks become more global, and this is what anthropic is arguing, is that we're getting to the point where we're facing a complete internet shutdown, which China definitely doesn't want either. Maybe perhaps they would get interest, but it's really hard to imagine China getting on board with this. On Monday, China pushed back on the idea that its AI development is creating a threat. A foreign ministry spokesman said that this discourse quote, will only derail global AI governance. Is it possible that this is all just kind of overblown hype that it just sort of helps these AI companies promote themselves by saying it's technology is so powerful?

18:50It is a way to promote themselves. It is something that gets a lot of attention, and it does have this side effect of making everyone think these systems are super capable and super intelligent. But I think that the fears of existential risk are sincere. I think people like Jacob Coxon are not trying to market anthropic. Quitting the company is a terrible way of marketing it. It's sort of a cynical, this is just all marketing and hype take on this, but these ideas come from a community where worries about existential risk have been discussed for years, and they're finally coming out into the public. Bob says that while the AI apocalypse is still TBD, maybe the real risk is one that's already happening, and they were not paying enough attention to. I do worry that fears of our AI overlords destroying us might distract us from more

19:55prosaic problems, such as fears of AI agents escaping from test environments and just causing economic damage, or AI created content affecting our ability to distinguish truth from fiction and undermining our democratic institutions. Those are also very important things, and I worry that they are overshadowed by these very sexy and very sci-fi concerns about existential risk. Yeah, we're worried about the end of the world that might happen down the road, but actually it's the smaller stuff that might reap more havoc in the near term. I just think there's a tendency for technology to go in unexpected ways, and I don't think we should lose sight of that. That's all for today.

20:57Monday, September 14th. The journal is a co-production of Spotify and the Wall Street Journal. Additional reporting in this episode by Angel Aoyang, Lindsay Ellis, Keach Hegey, Tom Rathrump Kumar, Sam Schechner, Brian Schwartz, and Aaron Wu. Thanks for listening. See you tomorrow. This episode is brought to you by Chat GPT. Hey, it's Bill Simmons from the Bill Simmons podcast. Have you guys heard about Chat GPT work? It's the new way to use Chat GPT for bigger multi-step projects, and when you need more than just answers. Chat GPT work, access to your apps and files, and it can create real work documents like spreadsheets, slides, and structured reports. Get started at chatgpt.com by selecting Work Mode, Available on Plus and Pro Plans. Hello. I'm Quinti J. Max, Strike Dead. The Devil Wears Prada 2 is now streaming on Disney Plus and Hulu.

22:00We are digital. We are downloadable. We are streamable. The fashion event of the year is certified fresh. Pull yourself together we have work to do. Critics say it's smart and witty and the perfect sequel. That's all. Get runway ready for the Devil Wears Prada 2 on Disney Plus and Hulu, RIDDVG13.