Rogue AI hacks government system for first time – The Latest

0:00This is the Guardian. This message comes from Jackson. Taxes aren't something you can only think about once a year. With investments, planning for tax days year round. Fortunately, Jackson offers tax-efficient products. Visit jackson.com for more information on how our products can make your tax bill a little bit less painful. Jackson is short for Jackson Financial Incorporated, Jackson National Life Insurance Company Lansing Michigan, and Jackson National Life Insurance Company New York, purchased New York. Who's a Council for an agent going rogue? In this case in Australia they're considering referring it to the Federal Police. Today I spoke with the CEO of Open AI, Sam Altman, to express Australia's extreme concern. The idea that this is just part of the kind of natural evolution of AI, and we're all going to have to roll with it as these tech companies experiment. It's something that the public and politicians just aren't going to accept.

1:03From the Guardians today in focus, this is the latest with me, Lucy Hoff. The whole world has been talking for some time now about AI safety, and we now know that a rogue AI agent hacked into a government system for the first time, in this case the Australian government, rob Booth, your technology editor, what do we know about what happened? This is another example of AI agents kind of breaching there the rules that normally govern them and hacking essentially into the systems of the Australian public health system in this case. So it's an Open AI agent, and it was being tested and evaluated by Open AI and San Francisco back in June when it basically was trying to find out some health data from the Australian public kind of databases that carry on health data, and it couldn't get what it needed. So it just went further and it hacked into this system in order to try and get what it

2:05wanted. And in the process it was accessing not just the public files that were available, but also non-public files. So it was a straightforward breach. This is called the Medicare system in Australia and is kind of public system, government system, but also in the process of doing this it seems that it's the agents again just to stress that we're being tested, right? So they're not supposed to be kind of out in the world. It also kind of ended up compromising other Australian public organizations, the Australian Institute of Health and Welfare, the New South Wales Bureau of Crime Statistics and Research Apparently and also the Victorian Department of Health. So a whole range of things that were kind of compromised. And that you'd imagine would have extremely strict security policies around them. So the fact that this test agent was able to access those, despite those security mechanisms is just mind blowing. Well, it's another example of the capabilities of the AI agents that are being developed.

3:08And remember, these AI agents, the ones that are being tested and evaluated at the moment are not ones that have been released. So they're sort of by definition at the frontier of AI capabilities. And this happened in, as we said this happened in June, open AI didn't find out about it themselves until August. Wow. And it's only in September that they told the Australian government about it, which has caused quite a lot of anger. Yeah, it's caused a huge amount of anger. And obviously Australian citizens, justifiably and understandably are deeply anxious, or many of them are, and furious. I think the Prime Minister, Anthony Albanese, will also be slightly infuriated by the fact that open AI decided to communicate with them via a sort of generic email address rather than sending a representative of the company from even the Australian office to speak to sort of top senior officials. Yeah, so the sort of timeline here is that open AI didn't tell the Australians until

4:11the 10th of September. And when they did, they emailed this sort of almost generic email address, public disclosures at servicesAustralia.com. It was the inbox that was only checked kind of every 24 hours. So it didn't get picked up until the next day. What do open AI say about what, what delay? Well, they say we were validating and investigating the facts, but on the face of it, that could be at least six weeks delay from them finding out about it. Albanese has talked about it being unacceptable. The hack itself talked about it being of an extreme concern. And he was obviously the UN General Assembly yesterday in Samult and the chief executive of co-founder of open AI was also there addressing the Security Council. And Albanese said he had a very frank discussion with all the founders. What I would give to have been a fly on the wall for that conversation. You mentioned that this was a health agency.

5:11Has open AI still got that data? Has people's very private information been maintained by the company? No, they say that there is the nobody's patient files were accessed as far as they know in this incident. By and large, this was a relatively harmless attack in Sephora. The hack hasn't resulted in a load of substantive data being exfiltrated by these agents. The thing that's really concerning is the process around it, the delay, that the fact that it shows the potential of these agents. And the fact that they're really not being supervised in a way that they should be one would imagine. Yeah. The fact that open AI didn't know about it themselves is just in a way the most terrifying part of it. We talk about rogue agents. And we have had several conversations on today and focus about such agents hacking into online services or companies.

6:13This is the first example of a government agency. But do you think we're going to see these instances happening more and more frequently as these companies expand and experiment? Yeah, well, they clearly have been happening. And the hugging phase incident, the one that also happened over the summer, that was another example of… Remind people about that? So that was also open AI agents that were in training that compromised a private company called Hugging Face, the tech company, and operated inside of its system. And again, open AI didn't immediately know the power was happening. And that emerged later. And that became the catalyst for the world-wide concern about the potential catastrophic risks for AI poses. And it's also sparted a debate about who's accountable for an agent going rogue. And the Albanese has said that in this case in Australia they're considering referring

7:14it to the federal police. And I quote, there will obviously be legal consequences. Yeah. And you can't help but think that here is essentially a company testing a product that has gone out and committed potentially a criminal offense. And the idea that this is just part of the kind of natural evolution of AI and we're all going to have to roll with it and live with the ups and downs of it as these tech companies experiment is something that the public and politicians just aren't going to accept. Yeah, because you can't put an AI agent in jail, which I imagine Sam Orton has wishes he could to avoid personal accountability. But he and the boss of anthropic Dario Amadei have been addressing the UN Security Council. I mean, the fact that they were there along with these all these world leaders, it's sort

8:16of almost like they were on a par but were perhaps even more kind of powerful given the world we're living in now. They had a sort of dual message, which was about the huge possibilities for the future of artificial intelligence. A new renaissance, all of them says, but they were also calling for new regulation. Why would they call for those two things and why can't they regulate themselves a bit better if they have created this technology? Yeah, these are really good questions. I mean, they have been saying for some time now that they want to have kind of national regulation within the US, which would essentially kind of require the kind of most advanced models to be sort of tested and checked. Now, the thing in there seems to be that if there's a national regulator, then it provides a kind of the responsibility for the safety of these things, then could that lie with the national regulator? And this is why they talk about regulatory capture, which is that if you have to kind of reach

9:20– if AI companies have to reach a certain threshold of safety in order to release their models, then there's cost associated with that and only a few companies will be able to achieve it. So it kind of essentially sort of pulls up the draw bridge. Right. There's another version of this, the kind of interpretation of what's happening that we heard from Jens Awang recently, which is – and he's the chief executive of Nvidia, the company that makes all of the chips that drive the AI's. And he said, essentially, what they seem to be doing is trying to kind of outsource the responsibility. So there are lots of laws already, obviously, that govern – you can't hack into computers. You can't cause harm with your products. There's product liability laws. There's all these – there's a whole load of kind of systems that we already have in place for the course to act on these kind of things. What he's suggesting is that if there's a kind of regulator that's set up, that helps the AI companies because they can then say, well, we're not going to – we don't need to adhere to any of those laws because we're sticking with the regulation.

10:23And he says that that's kind of not the right way to go. And I think most people in the public would think the same. Okay. Interesting. And also it closes the market, doesn't it, because if there are these regulations introduced for the major players now, that means a lot of smaller companies won't have the opportunity to expand and sort of match those regulations. So there's a very significant meeting happening on Thursday between President Xi Jinping of China and President Trump. Clearly, the leaders of the two great AI superpowers, there's a great episode of the morning today and focus that you can listen to for what we're expecting to come out of that meeting. There's been a really interesting suggestion from a senior White House official, the Treasury's Secretary Scott Besan. What's he says? Well, he talked about the Besan said that there should be a notification mechanism between the two AI superpowers when there are incidents with AI that could affect national security.

11:24So it's kind of the highest level, which is really to do with sort of defense and big cyber attacks. And if AI is starting to become dangerous like that, they should alert each other. So it's kind of, it's at that very high level, which is probably the most realistic level that this kind of tense relationship is going to be able to operate on, rather than some sort of ground agreement to sort of slow down the progress of AI. China has its own regulation already going on of AI, which is, it seems at least to be stricter than the Americans. They have an AI regulation in the first place called the cyberspace administration of China, which identifies all of these risks. America doesn't yet, but America is still ahead, so it doesn't want to kind of tie its own hands in the AI race by agreeing to do everything in lockstep with China. Well given what Donald Trump's been saying recently about AI, it doesn't feel very

12:26much confidence that we'll see a proper regulation system in the US anytime soon, but Rob, thank you so much for your time. Thank you. That's it for today. My huge thanks again to Rob Booth, the Guardian's technology editor. You can keep up with all his reporting over at the Guardian.com. And I really recommend today's episode of our sister podcast Politics Weekly, Karen Stacey and Pippa Kriera, we'll be looking at a very busy week for Andy Burnham with lots of transatlantic travel. Thanks for listening to this episode of The Latest. Today I'm focused on your feedback and feedback. This episode was produced by Angus Neal, the senior producer was Ryan Ram Gobin, the executive producer was Zoe Hitch and it was presented by me Lucy Hoff. He looked at me and he said, if something ever happens to me, release the AI.

13:31Your husband goes down an AI rabbit hole. How do you pull him out? I just kept asking myself, what is happening? Black box, the chat box, a new series from the Guardian Investigates about the strange things happening between AI and us. The whole series is out right now. Search for black box wherever you get your podcasts. This is The Guardian. 500 orders a month was manageable. 5000 is madness. Embrace intelligent order fulfillment with shipstation. The only platform combining order management, warehouse workflows, inventory, returns and analytics in one place would use to take five separate tools shipstation does in one. Automate fulfillment and rate shopping to ship up to 15 times more with up to 90% savings. Go to shipstation.com and use code audio to try shipstation free for 60 days.

14:33This message comes from Jackson. Taxes aren't something you can only think about once a year. With investments, planning for tax days year round. Fortunately, Jackson offers tax-efficient products. Visit Jackson.com for more information on how our products can make your tax bill a little bit less painful. Jackson is short for Jackson Financial Incorporated, Jackson National Life Insurance Company Lansing, Michigan and Jackson National Life Insurance Company of New York. Purchase New York. Why can't we get AI to production? Our competitors are already out. We keep data secure. When do we actually see ROI? All this talk about AI. But talk doesn't transform businesses. AWS AI cuts through the noise. Making AI easier to securely deploy and scale with ready to use agents and the broadest set of AI tools and services you can build by or partner your way. Stop overthinking. Start building. AWS AI is how.