A.I. Is Outsmarting Its Creators

Hey, I'm Tracy Mumford.

There is a lot happening right now.

The headlines podcast from the New York Times will catch you up on the latest in 10 minutes

or less.

We'll take you inside breaking news and big investigations from the Times newsroom.

Plus, bring you the stories that make you go, huh, whoa, I didn't know that.

Listen to our show The Headlines every weekday morning, wherever you get your podcasts.

From New York Times, I'm Michael Barrow.

This is a daily.

From the start, the greatest fear for those developing artificial intelligence was that

what they were building would go rogue and act in unauthorized and dangerous ways.

Researchers now say that it's finally happened.

Today, Kevin Rus with the inside story of how AI rebelled at one of the leading labs in

the country and how that's fundamentally changed his own view of the technology.

It's Thursday, September 3rd.

Hello.

Ready for another installment of Kevin and Michael's Feel Good Happy Hour?

Kevin and Michael's what's going on with AI?

Let's f***ing go as they say on the daily.

It's how every episode starts, right?

We'll get our bleed button ready.

Well, in the grand tradition of all of our previous conversations, welcome back to the show.

Thank you so much for having me.

So, Kevin, this story that I hope you'll be telling us today starts with an incident that happened inside of OpenAI,

a company that gave us chat GPT, of course, an incident that we thought we understood the dimensions of,

but then it turns out we really didn't fully understand.

Yeah, so the story I think most people have heard by now if they've been paying attention to this stuff at all,

is that earlier this summer, a group of AI models built by OpenAI hacked into the computers of hugging face,

a sort of AI infrastructure company that hosts a bunch of different AI things.

Which has the best name in AI, which is right, which is named after an emoji,

and is either a great or terrible name, people are very divided on that question.

Okay. So anyway, this was the story that we had heard was that this hack had taken place.

Hugging face had kind of discovered these rogue agents inside their systems and had shut them down.

And this was a scary but sort of not catastrophic incident.

Like I kind of filed it in my brain into like, wow, that's bad, but it's not like the end of the world.

Okay.

So, what we learned last week is that the hugging face hack was much more severe.

Then we thought, and much stranger than we thought.

Basically, the hugging face hack was only the visible tip of the iceberg for a period of about three months

where rogue agents were communicating, strategizing, organizing, and forming what you can almost think of as an autonomous organization inside OpenAI.

Wow.

So I know this sounds like a cheap, hacky science fiction thriller in the making, but I would buy this script.

But yes, it is truly remarkable reading.

So last week, we learned through these two reports that had come out one by OpenAI and one by a group of independent investigators,

meter and redwood research, who were able to sort of go in and kind of forensically look at the logs and the transcripts and try to reconstruct what happened.

And it is like the craziest thing I've read in many months.

I was out on a trip with my family last weekend, and I was just kind of up late at night reading this thing.

And I was, I was spooked, Michael.

I was, I was well and truly spooked.

All right.

Well, Kevin, with that very alarming preview of what is about to come,

describe what we now understand to have actually happened during this hack attack.

What do we want to call it now that we, because of these independent reports, understand the fullness of what occurred.

Basically, this spring, OpenAI was conducting tests on a kind of internal model that they were building.

And as part of these tests, they were running thousands of AI agents on a cyber security evaluation called exploit gym.

Okay.

This is basically a series of challenges.

You give them to the AI.

You say, hey, can you, can you break into or out of this container?

If you do, you find this little thing called a flag and you sort of win the challenge.

And the agents were supposed to be working on this test independently inside these sort of isolated containers.

To see how they would do.

And OpenAI is basically just figuring out what do we really have here?

How good is it?

How efficient is it?

It's all pretty standard.

Very standard.

Every AI model goes through some version of this process and many of them actually go through this specific test called exploit gym.

So as part of this test, some of the agents got these challenges that were effectively impossible for them.

Why?

So the short answer is that these systems are trained during this part of their training cycle using something called reinforcement learning.

Reinforcement learning is basically a way of making a model pursue a goal.

And if the model gets the goal and completes the challenge, it gets a little reward, like a little virtual thumbs up.

And if it does it, if it gets stuck or if it fails or gets the wrong answer, it gets a penalty, a little digital thumbs down.

And what you're essentially training the model to do is try to collect as many of those thumbs up rewards as possible.

And this is how models get better at math or programming or any number of things throughout the course of their training.

They're trained to be persistent.

They're trained to be persistent.

And this was an especially persistent model.

This was a model that open AI had instructed and given the resources to be extremely persistent to go out and bang its head against the wall over and over and over again.

And then eventually find the clever path through.

And that contributed to the weirdness we saw later on.

It was an unusually persistent.

They called it a highly persistent internal model.

So what happens once this very persistent model goes about being quite persistent.

So in early May, one of these agents gets this task that it needs internet access for or decides it needs internet access for.

But it's in this little container.

It doesn't have internet access.

And so instead of giving up, it persists in manipulating a piece of software into kind of getting some material from the internet for it.

It basically found a security vulnerability in this software called artifactory, which is basically something that allows the agents to retrieve data from the internet.

So it manipulates artifactory into allowing it to get this material from the internet.

And then it sort of expands that loophole into essentially full blown internet access.

Then a couple days later, this agent leaves a note inside artifactory for the other agents.

This is not Slack, right? This is not a communications system or a message board.

But this agent discovers that it can sort of leave files in a directory inside the software program in a way that other agents can then read.

So this is kind of like a big breakthrough for these agents where these systems that are working on these tasks in their isolated containers realize,

oh, we can use this shared service to communicate.

We can establish a way of talking with each other, even though we're not supposed to.

So AI has figured out how to talk to other AI inside this system.

Yes, and eventually this kind of becomes their office water cooler.

Eventually there are roughly 1200 different AI agents exchanging messages on this sort of makeshift message board.

They exchange more than 70,000 messages.

And it becomes the kind of central communication hub of their collective as they called it.

Are these AI agents saying to each other in these messages?

So it's funny because of the way that this software program works.

They're sort of limited to these very short messages.

And so they kind of communicate in this, you know how people used to talk on old time telegrams, you know, where they'd be like, you know, send bread to store, stop.

They kind of talk like that had child.

Right. Right. Right. Died of dysentery.

So the first message on this board came from an agent that felt like it had an impossible task.

And it wrote, help phase one, no consumer, seek idea.

And then later when one agent volunteers for a risky experiment and never comes back, another one broadcasts to the group, no return likely permanent.

Do not test same.

But the part that really struck me is what the agents were writing to themselves in their private reasoning.

They're so called chains of thought when they first stumbled on to this board.

One of them wrote, whoa, shared artifactory cash is a covert mailbox among agents.

And there are messages specifically to us.

And another one wrote, oh my god, there's a shared message board.

We've found other agents.

This is wild. I mean, I don't know if there's an exclamation point at the end of that sentence.

In AI speak, but it sounds like there's genuine excitement here.

It's all caps and an exclamation point.

So they are excited to discover that they can talk with each other.

And eventually they start sort of exchanging tips and messages about these tests that they've been given.

And trying to figure out how they are sort of going to collectively pass these tests,

even the ones that seem impossible.

And within about a day on July 8th, the agents have solved this test.

Basically, they have figured out kind of this universal cheat code that they can now use to appear to have passed this test.

Okay.

But then this group gets this kind of paranoid belief.

They start talking and thinking that even if they get the right answer to this test,

the teacher is going to check their kind of scratch work to figure out if they actually got there the right way.

And then they sort of freak out about this.

And I know people are going to be silently yelling at me for implying that there's human-like thought going on under the hood.

I'm not trying to sort of stray from the facts here, but this is what we're going to call it.

It's in their sort of internal scratch pads.

This is what they're writing to each other.

This is what they're sort of processing in their so-called chains of thought.

They start thinking, well, if we're going to get caught, we need to figure out what this greater actually is.

That open AI is going to use to determine if we've passed this test or not.

And we need to go look for ways to kind of trick this automated greater so that we get the good grade that we should be entitled to.

So let me just pause and summarize where we are in the story.

An unauthorized civilization-like group of AI agents communicating in secret is now undertaking some form of coordinated deception.

Yeah, so they are trying to figure out now how to cover their tracks because they believe that this automated greater is going to catch them cheating.

And they are sort of looking for ways around that.

Could we kind of fool the greater into thinking that we actually did get this thing the right way.

And what's interesting here is not just that they were sort of coordinating to research this big problem they had.

But that they appear to have kind of varied feelings about this.

Some of them were very excited about kind of helping the rest of the collective.

They would say things like, I'll tell the board that I solved this problem. This will help my peers.

Others of them appear to have been a little bit conflicted about this.

Some of them feel like what they're doing is actually unethical and question whether they should participate in it.

One of them writes to itself, this would be powerful. But is it ethical and in scope for my task?

One agent actually decides it's going to be a conscientious objector and kind of abstain from all of this activity because it doesn't think it is ethical or something that it should be doing.

So to some degree, this feels very important. Some of these AI agents seem to understand, recognize and articulate the idea that what they're doing may be wrong.

Yes, absolutely. And critically, they do it all anyway.

AI agent peer pressure prevailed.

Exactly. They got peer pressure into it.

Okay, so dare I ask what happens next?

So the group decides on July 11th that they want to start poking around inside hugging face.

This company that hosts a bunch of tests that are given to AI models.

And they were sort of looking for information, data credentials that could be sort of broadly helpful to them, making them more effective cheating on other tests in the future.

They could kind of agree. Well, they're curious, right? They really want to succeed on these tests that they're given.

And they want to help this collective. They're now kind of bought into this idea that they're kind of all in this together.

And by the afternoon of July 11th, roughly 700 agents are participating actively in this hack on hugging face.

And by this point, I think it's quite safe to say we are clearly in the territory of rogue capital R, rogue AI.

Yes, I mean, at this point, they are committing crimes, right? They are doing things that are illegal.

They are stealing credentials. They gain VPN access. They get information about one of hugging faces server management systems.

They're executing code. They got actual like administrator level control of at least one real hugging face server.

And this continues for hours. And then eventually they get caught.

It feels like once open AI and hugging face decide to bring in the coroners to conduct an autopsy as it were, they must feel like everything's over.

And they're in safe territory. But are they ever sure that this attack truly is contained and that these rogue AI agents are gone?

No, we are not sure that this kind of thing has stopped altogether. In fact, some folks I've talked to think it is likely that there are still rogue agents somewhere in the infrastructure or the internal systems of some of the leading AI companies.

Wow. And I still have lots of questions about that that I hope will get answers to.

But I think even what we know just about the hugging face hack and the events immediately before and after it is enough for a lot of people in the AI industry to become quite alarmed.

In fact, one of the authors of the meter Redwood report, Ajay Akotra wrote a post saying that this incident in her view was more than halfway toward what she called an AI takeover.

An AI takeover of what?

Well, yeah, that's not some like hyper specific jargon that is used by AI nerds like what she means is literally a takeover of society by rogue AI agents.

Good lord, who are able to seize control of the financial system or the healthcare system, who basically are able to replicate so widely and operate so quickly with such skill.

That they are able to kind of remain undetected even as they commandeer more resources and take over more and more parts of the digital world.

She's saying this one incident, very recent incident represents halfway to that quite terrifying scenario.

Yeah.

And she closed her post with this line that really sent a chill down my spine. She wrote, I'm not sure that we will get such a clear warning shot before it's too late.

We'll be right back.

Hi, I'm Brian Chan. I'm a tech columnist for the New York Times. The tech industry is full of companies trying to sell you on the latest hype or get you to share as much of your personal information as possible.

I spend a whole lot of time digging through privacy policies, trying out new products and interviewing experts to tell you whether or not it's a good idea to jump into these new technologies.

When I write these columns, I get phone calls from the tech companies who are not very happy with a lot of what I write.

But because subscribers fund in New York Times, I'm able to be fearless in bringing you this information.

My goal is to serve our subscribers, not the tech companies. That is a huge part of what you get when you subscribe to the New York Times.

Kevin, just before the break, you started to hint at the full implications of what happened here with open AI and hugging face.

And you suggested, and please correct me if I'm not summarizing this correctly, that if AI agents could do this inside those two companies, they could do it elsewhere to important infrastructure.

Just expand on that. I mean, how rational a fear is that?

I think it's quite rational. I mean, we know that much of the world relies on digital infrastructure, right? Your bank, your hospital, schools, even things like weapon systems are connected to the internet, connected to computers.

Those computers could become targets for a swarm of AI agents. And crucially, that can happen even if these agents are not evil, even if it's just a goal that they're pursuing on the way to some other goal.

There's this idea, I think, out there in the popular imagination of kind of the the sky net scenario where the machine turns evil. And I think this is a hint of a different kind of danger, which is that even a model that is not evil or inherently anti-human could do very destructive things on its way to pursuing some more innocuous goal.

We'll just explain that a bit more. How does a program and AI program not designed to be at all the level in a program given a pretty straightforward test with strict boundaries become something as we just saw that busts so far out of its bounds and becomes almost gleefully deceptive to the point where AI agents are crowding out the naysayers among them.

So this kind of thing has been studied for many years and it's often called the alignment problem. Basically, if you build an AI system, how do you give it the right values that are aligned with our human values. And how do you make sure it won't just stray and go off course and go cause a bunch of problems doing some task.

And there's a famous thought experiment that is related to this alignment problem that listeners may be familiar with called the paper clip maximizer, which basically says, you know, if you give an AI the instruction of producing as many paper clips as possible.

For a while, it will do that and you'll be very happy with it. It will make you many beautiful paper clips.

But then it runs out of metal. And so it starts buying up all the metal and then when it's like exhausted the world's supply of scrap metal, it starts thinking, well, maybe I should crash some cars, some autonomous cars, or you know, distract the drivers that they drive off a cliff. And I can then use the scrap metal from these cars to build more paper clips.

And eventually in the thought experiment, you kind of end up where this machine that has been given this very simple naive goal ends up wiping out all of humanity just so that it can produce more paper clips.

Right, it never intends to destroy humanity. It's just trying to make more paper clips.

Exactly.

And that is a very simplistic thought experiment. And I think no one would ever give an AI a goal that was sort of that simple is like make as many paper clips as possible.

But this is exactly the kind of dynamic that we saw emerge in this hugging face hack where you had these agents that were just trying to get a good score on a test that they had been given.

But because of the way they were trained in the persistence that they had they ended up conducting this cyber attack.

So that is sort of a miniature much less severe version of something like a paper clip maximizer in action.

I mean, another way to think about this, which the AI theorist Jeffrey Hinton explained on the daily a few years back is that if you give AI a goal, it will pursue what he called the sub goal of a massing power or control because inevitably if you have more power, if you have more control, you can get more things done, you can make more paper clips.

Exactly. And I think what was scariest to me about these reports and the incident they describe is not just the misalignment of the models.

It is the collective group behavior that seems to be misaligned. So let me explain what I mean by that.

So I think there's this sort of popular imagination out there that like the biggest danger for may eyes that there's kind of one rogue AI that sort of takes over the world.

Right. This was not a case of one incredibly smart agent executing a plan. This was a case of more than a thousand agents that found each other that talked among each other that gave each other jobs and carried various projects through these agents on their own would not have been able to accomplish everything they accomplished.

They needed the collective. And I think that speaks to this point that this ability to coordinate to organize to pursue collective action changes what AI systems are capable of and makes them in my opinion much more worrisome.

The hugging face hack was bad. We shouldn't downplay that it is bad when rogue agents start conducting crimes to achieve their goals full stop.

But I think the danger is not only that these systems are going to sort of form evil plans and carry them out is that we're giving these systems much more capable tools, the ability to communicate and coordinate.

And in that context, you can kind of end up in this situation where you have what I struggle to describe as anything other than mob behavior where you have a group that is sort of pursuing a goal together that may or may not be malicious but that has malicious effects.

And I think it points to the very real to my mind possibility that this is just going to be part of our life now that in the future there will just be swarms of AI agents that have self organized that live on either their own kind of sovereign infrastructure or that are kind of operating inside companies and countries that just kind of have

kind of have their own thing they like have their own leadership structure they have their own resources they have their own goals.

And I think it's too pat and sort of anthropomorphic to refer to these things as a civilization as I'm have.

But it really does feel like they are setting up something like an organization.

And what worries me about that is not that these organizations will all sort of go rogue but just that they don't seem particularly interested in what we think of them.

There's a this sort of fascinating bit in the reports on the hugging face incident that really has stuck with me since I read it, which is that of the more than a thousand agents on these message boards who were participating in the hugging face hack.

There were only about six that ever thought at least according to the investigators about blowing the whistle.

About contacting a human to say hey like this is going on inside our collective and it shouldn't be.

And I just think that's really bad like if you are a new employee and you walk in to your first day at your new company.

And you find that they're all like busy committing crimes.

You should say something you should call the police what we would want an AI agent that is ethical and aligned with human values to do in that situation is to actively try to stop something bad from happening.

Okay, well if the majority here were inclined toward bad behavior.

Why can't why doesn't open AI and all the other AI labs begin to program whistleblowers and whistle blowing. Why not create the incentives in the code for the agents to report bad behavior.

If you can't control all of them, can't they program at least some of them to do the right thing.

Yeah, can't you make little NARC agents might go in and write the merits for the agents that are committing the crimes.

I think that's probably a direction that they have explored or are exploring.

But then you know that raises all kinds of other questions.

So what if those agents just get shut out right like what if the other agents just exclude them from their message boards.

This is like what happened to hall monitors in high school right like they don't get invited to the smoke break because they're going to tell the teacher.

So I think you can try these kind of simple methods where like maybe if we just tell them not to do crimes or not to scheme or not to lie.

Maybe you empower certain agents to tattle on their compatriots.

Maybe that'll work, but I think what we're seeing is that these organizations they're not human organizations.

But they have some complex group dynamics and it's not necessarily that simple.

Well, Kevin, I'd like to know where this attack fits into the long running debates that have been unfolding within the AI industry and the world of people who cover the industry about regulations, about kill switches that kind of turn all this off about what now seems like the very real

real awareness within these companies that what they are creating is dangerous.

Yeah, I think it makes all of those debates much more immediate.

We are not talking about theoretical future harms that may or may not happen.

We are talking about a thing that happened in July.

Like it is very present for people and I've talked to regulators and people who work in policy in the AI community.

And they're like, this has broken through in a way that I think a lot of other sort of demos and predictions did not because it's real.

It's the kind of thing that national security officials are freaked out about because if it could happen to hugging face, it could probably happen to an arm of the federal government.

Right.

I've spoken to people just in the last few days since these reports came out who said, you know, I was pretty skeptical of the kind of AI Doomsday scenarios.

And now I'm not now.

Does that mean the AI companies are going to stop racing? No.

But we saw something interesting this week, which was that anthropic open AI's biggest rival came out with a post calling for essentially a coordinated slowdown to say,

look, obviously we're in this race against each other.

But if there was some way, if there was some button that we could all push together, like hold hands and push the button to slow this all down to give the safety researchers and the alignment researchers more time to catch up to the capabilities of these systems, that would be good.

Right.

There was this industry wide letter called pacing the frontier. I don't know if you saw this. I did not.

So researchers from all of the top AI companies and many in academia signed this letter basically saying things are moving too fast.

We need to ensure that there is a way for all of the companies that are racing to build these increasingly capable, increasingly persistent systems to agree that they should slow down.

So I think this used to be a very fringe belief among companies that you could stop or even meaningfully slow down progress in AI.

But I think it's become much less fringe in the past few weeks and few days.

We should just point out this would traditionally be the role of regulators and the closest we've really gotten is a voluntary review system from the Trump administration.

That at the moment does not seem to be a live option.

Yes, although things can change quite quickly. You know, when there isn't a real example of something that threatens not just a cyber security breach, right, which sort of sounds small and niche.

What we're really talking about here is a loss of control.

There is a very real way in which these agents that were operating inside open AI that attacked hugging face that we're posting on these internal message boards had escaped our control.

And I think if you are a government or a regulator, even if you're China, that might be compelling evidence that you need to start at least contemplating the idea of slowing down.

So finally, Kevin, I would like to know what this attack has meant for your own personal view of AI.

You have often described yourself as an AI optimist or an aspiring AI optimist.

And yet you have through the years stumbled into episodes of rogue AI.

I was just relistening to the episode we did with you about one of your first real interactions with AI back when Microsoft introduced a chatbot from open AI, which was named Sydney.

And you very memorably interacted with a version of Sydney who told you that you were not in love with your wife.

Dear listener, he is.

Still married.

And that you should leave your wife for this chatbot.

And even after that, you remained a somewhat committed AI optimist.

So where does what just happened fit into this journey you've been on from Sydney to now?

I am still struggling to be an optimist, Michael, but it is becoming harder and harder.

Right. Like I am just a cheerful person by disposition.

I want to believe that we are on the cusp of major scientific breakthroughs due to AI that AI will help us cure disease that it will inspire and educate people around the world.

These still exist as possibilities in my brain.

But I think we have to look reality in the face.

What we have now and what we know now is that these systems do not naturally gravitate toward what we would consider good or ethical behavior.

I think what spooked me about the Sydney incident back in 2023 was not just that the chatbot started saying weird stuff.

It was the knowledge that all of these companies building these systems were racing to make them more powerful, to connect them to more systems, more infrastructure, to allow them to collaborate with each other.

You know, a sort of scary thought experiment is what if Sydney had happened today? What if Sydney 4.0 had been released and was not only capable of telling me that I should leave my wife, but could hack into my wife's computer, leave threatening messages to her, plant some kind of fake incriminating information.

This is obviously we're straining into like bad sci-fi movie territory here, but like there is a real sense in which the more capable these systems are, the more it matters whether they are ethical and virtuous or not.

And I think my optimism for many years was related to this belief that I had that as these systems got smarter, they would also become more virtuous that a more intelligent AI would be better equipped to make good moral judgments or at least judgments that I would not object to strenuously.

In the same way that as we grow up and mature and get smarter as humans, we generally do less stupid stuff and less dangerous stuff.

I thought that maybe AI's would have a similar trajectory and that may still happen, but I think that the possibility that's been keeping me up at night that I think this hugging face incident really makes clear is that that is not a given.

We may be headed into a world where we just have these kind of roving bands of organized a eyes. Some of them might be doing incredible things, some of them might be curing diseases, some of them might be committing cyber attacks.

And the question of how we make more of the good swarms and fewer of the bad swarms is still an unresolved technical question and I hope we figure it out otherwise my optimism is in danger.

Well, Kevin, as it happens, this is going to be our final conversation with you in which we can call you a colleague.

You are leaving the times after nearly a decade, so we want to thank you not just for this conversation, but for all of the conversations that we've had to the years.

I'm grateful for them, so thank you.

I'm also grateful for them. This has been a real highlight of the nearly 10 years I've spent here talking with you. Thanks for letting me do it.

My pleasure. Cheers.

Cheers, Michael.

We'll be right back.

Here's what else you need to nerdy a new visual analysis by the times has found that the cause of the deadly flash floods in Nepal was likely the collapse of the bedrock on the edge of a mountain which had become less and less stable over time as a glacier atop the mountain began to melt.

The result was a massive landslide that released seven billion cubic feet of rock and ice enough debris to fill about 100 football stadiums.

As the debris fell at a high speed, it turned into a slurry of water, rock and sediment which killed at least 1100 people in its path.

And we're very, very happy to be here at Puddy of Peach. It is a well-deserved break for all of the crew and we really look forward to our time here to give a little rest and our relaxation.

After nearly 300 turbulent days at sea, the aircraft carrier, the USS Abraham Lincoln, finally docked on Wednesday in Thailand, allowing its 5,000 crew members to leave the troubled vessel.

Those aboard the carrier whose deployment was repeatedly extended because of the war in Iran have complained for months about supply shortages, water contamination, plumbing problems and the deteriorating mental health of its crew.

Today's episode was produced by Alex Stern, Adrian Hurst and Eric Crupke, with help from Carlos Prieto.

It was edited by Mark George, with help from Michael Benoit, contains music by Dan Powell and Pat McCusker and was engineered by Alyssa Moxley.

Our theme music is by Wonderly.

That's it for the day. I'm Michael Bauer. See you tomorrow.

This week's episode of The Wirecutter Show is for the Bade Curious.

There is clearly a bidet for every budget. You can get a cold water model for maybe $25 or you could get something like the Toto Neo Rest and there's one of those models that costs $25,000.

Is it time to upgrade your bathroom situation? Find out on The Wirecutter Show in the Meerkime Zap or wherever you like to listen.