Special Edition: Could AI Really Kill Us?

0:00By catastrophic, we define this as a million deaths or a hundred billion dollars in damages. And they thought that even when people took reasonable precautions, there was a better than 10% chance than in the next five years we'd have that level of incident. Today is Saturday, September 19. There have been some pretty alarming headlines about artificial intelligence lately, including warnings from people inside the industry who say if AI keeps advancing as quickly as it has been, the risks could eventually quote, kill us all. That can sound like science fiction. But recently we've also started seeing AI systems do things their developers did not necessarily expect. So how seriously should we take these warnings? How could AI actually cause catastrophic harm? And can the companies building it keep up with safety? Today we're joined by Dr. Neil Thompson, director of MIT's Future Tech Research Group, and a principal research scientist at MIT's Computer Science and Artificial Intelligence Lab. His team studies some of the most important trends driving progress in computing and AI, including where the technology may be headed next.

1:01So today we're talking about the real risks and asking what would actually need to happen now to keep AI's promise from becoming humanity's downfall. Welcome to the Newsworthy Special Edition Saturday, when we sit down with a different expert or celebrity every Saturday to talk about something in the news. Don't forget to tune in every Monday through Friday for our regular episodes where we provide all the day's news in less than 15 minutes. I'm Erica Mandy. It's now time for today's Special Edition Saturday. Neil Thompson, thank you so much for joining us here on the Newsworthy. My pleasure. So when people who have actually worked at these AI companies warned that this technology could eventually kill us all, as one said, how realistic do you think that actually is? And let's put it on a scale of one to ten, one being it'll never happen and ten being it's inevitable. You think the first thing we can say is that is there a chance that these things can do real harm to us? Absolutely. Are they going to be able to kill us all? I think the answer is no. Right now these

2:02systems are not at the level of they're going to be able to do something like that. I think the worry is that the pace of improvement is so fast and we see such remarkable capabilities emerging that they could emerge in areas that really would be very worrisome like bioterrorism or things like that. And of course, those are the things where suddenly you start to worry about the scale of these dangers being enormous. We recently did some work where we've asked a bunch of the experts out there. If you think about AI's ability to build things like bio weapons, what's the likelihood that that could cause a catastrophic incident? If I catastrophic we define this as a million deaths or a hundred billion dollars in damages. And they thought that even when people took reasonable precautions, there was a better than 10% chance than in the next five years we'd have that level of incident. Thankfully, we're not at the they can kill us all yet, but we should be worried about it because of the paciment. And certainly in the work that we have done where we've looked at AI

3:03capabilities, we can really see this incredible improvement that is happening in the early year, that is just of a much shorter timeline than most technologies we see. And of course, that's the challenge here. If these capabilities continue to develop as they have, we know that there are ability to do science, there are ability to do other things that can do lots of good in the world but can also do harm in the world are just going to increase. So you mentioned bioterrorism. Can you give us a little bit maybe a few other examples of how that could actually happen? Like how could AI plausibly cause catastrophic harm? I think bioter is a Muslim when people worry most about. The one that I think we're already starting to see the leading edge of is this ability to do cyber attacks. So people may have heard about open AI systems hacking, hugging face without being told to do so, right? They just went on the internet found a way to hack into their systems and then went in and evaded it. You can imagine that when this particular case what they were trying to do was just sort of change something so that they would be evaluated differently, but someone who had bad

4:04intentions could use that kind of cyber hacking ability to get into critical infrastructure. In our work that we've done with these experts, went to them and asked them what about things like manipulation, right? We know these systems are already talking with lots of people. Unfortunately, there are some very sad cases of people committing suicide. That sort of convincingness of them could of course be used to try and convince people to do bad things. Some of these bots have for example, when was encouraging someone to try and assassinate Queen Elizabeth II. This was a while back. Of course, you can also use that same capabilities to say maybe try to blackmail someone. And even when you're talking about bioterrorism, is that a situation where a human is using AI to come up with these dangerous ways that they could do attacks? Or is there also a concern that AI just takes off on its own and chooses to do things that we would not want? I think both of these are possibilities that we need to think about. Certainly in the shorter run, I'm more worried about the

5:05first one. I'm more worried about this idea that someone says, well, I'm in a particularly dangerous situation. Maybe my country is at risk or I'm at risk. And they choose to say, I'm going to unleash AI on other people. And I'm going to do it in such a way where I'm going to take off the safeguards intentionally because I wanted to do harm. That is my goal. I think we are closer to that. But it is also the case that we worry about these unexpected effects. When OpenAI's model hacked hugging face, that wasn't because OpenAI said hack hugging face. Right. We didn't think of that much of the collateral damage of doing this. And the worry is that as these systems become more capable, that collateral damage could get bigger and bigger. And yet OpenAI CEO Sam Altman just recently told Fortune Magazine that it would be insane to allow AI to get to the point where humans are not in control and that they won't allow it based on their safeguards. And yet we've sort of have already seen some of these things. So do you think AI

6:06companies can realistically stay on top of safety enough even while they're raising each other and China to build these powerful models and even when we're already starting to see incidents that are concerning. I think that's the million dollar question. And I think we don't know for sure in either direction. We do know that more can be done to help secure these models. We saw, for example, Dario Amadei saying we should be now embedding third party auditors within these companies in order to help check things as they're being developed. We wish to have some certifications. All of those things would be very helpful. And you're talking about Anthrop XCEO. Yes, exactly. Yes, absolutely. So we know that there are some of those things that can help. And we also know that as we create these models, they can also help us to reinforce the security systems we have. We also see that as these systems become more capable, they become more capable of evading our attempts to control them. And so the race between those two things that I think is unsure enough so that we should be worried and make sure that we're airing on the side of how do

7:09we make sure that we have fallbacks? How do we have other things that allow us to weather even if there are these incidents that are starting to emerge? So independent evaluators could be one pathway. Some companies have even called for some government regulations. Do you think from what you've seen, even if the government wanted to put regulations in place, which we haven't seen a ton of yet? Could they do it in a significant enough way, especially when we see lawmakers tend to struggle to keep up with what's actually happening with technology? Yes. So I think it's unquestionable that the pace of the technology is going to be changed. And I think that was why it was particularly interesting when the CEO of Anthropic proposed not just that these people would be regulating it, but actually that you would have these auditors embedded within the companies. They even have employee-like access so that they're seeing these things that they emerged. I think that is going to be a crucial aspect because the regulators are three years behind. You know, there's so far from the frontier, you get people responding to yesterday's problems, not tomorrow's.

8:13Any other solutions you think should be considered at this point? Some things that are going to be important are thinking about how do we have fallbacks and ways to turn off AI if there's a problem, right? So this is sometimes called a kill switch that can be used where you can either turn off the hardware or the software, but it is important to say if things start to get out of control, can we turn these systems off and can we have a fallback where if you've put it in charge of something important, that fallback can pick up because if you say, well, if I turn this off, something other bad happens, people are going to hesitate to turn them off. You know, there have actually been AI Doomsday warnings for years. So why has this finally become a greater concern now or at least getting to the mainstream in such a way that we're seeing it really hit headlines in the past week or two? What's changed is that the technology itself is it who put out the warning? What do you think is going on? I think two things have happened. So one is that we are finally at that point where this technology is surprising us and what it's able to do. Now that's important because people in my lab

9:19and others have been looking at trends in AI and saying we can see this incredible increase in capabilities, but of course, if you say, well, you know, this model was as good as a first rater and now it's as good as a fourth rater and a seventh rater, you're still going to say, well, I'm not that worried about, right? This is still not something that is worrisome. What we've seen in this last year is this set of problems that AI is now solving, which are incredibly challenging. So these can be math problems that have been open for a long time that are now being solved. This is unexpected things like the hugging face hack. Also, this leading edge of capabilities is crossing the threshold where we say, off, we can finally see this happening and we're right on the verge of that. I think the other thing that's happening here is that we have now also seen that these companies are large enough and these systems are distributed enough that they're really embedded in a lot of the things that we do. That combination, I think, is making it much more relevant when we see these really surprising events happening.

10:23Still ahead, what our guests think about the comparison between AI and nuclear technology. And how the race with China complicates any efforts to slow down. Plus, what everyday people can actually do if they're worried about where this is headed. But first, a break for our sponsors. When we moved a few years ago and refreshed our bedroom, one thing we already knew we wanted was to upgrade the whole bed with bowl and branch. We had been using their signature organic cotton sheets and loved them. So when we had the chance to redesign our bedroom, it made sense to build around what we already knew felt great. And honestly, it's made a difference. Even if I can't always get as many hours of sleep as I should, bowl and branch helps to ensure that the quality of my sleep that I do get is top tier. Our bowl and branch signature sheets are so soft, they're comfortable and breathable. Then we added more to the bed, a beautiful duvet cover and our waffle blanket, which I love because it adds that cozy finish look without feeling too heavy. The whole bed is just more inviting when I climb into it and that's exactly what I want at the end of a long day.

11:23Upgrade your sleep with bowl and branch. Get 15% off your first order plus free shipping at bowlandbranch.com slash newsworthy with the code newsworthy. That's bowl and branch B-O-L-L-A-N-D branch.com slash newsworthy code newsworthy to unlock 15% off. Exclusions apply. The newsworthy is also brought to you by Rosetta Stone. One thing I'd really love when I travel internationally is to feel a little more self-sufficient. Not necessarily have some perfect fluent conversation, but be able to ask for directions, check into a hotel, order at a restaurant, or handle one of those little travel hiccups without immediately needing someone to switch over to English. That's one reason I've been working with Rosetta Stone Sapphire. Rosetta Stone Sapphire is their biggest launch in over a decade. It combines the research-backed immersion method Rosetta Stone is known for with the new features designed to help you actually use the language in real life. And I especially like chat missions because they let you practice those exact situations before you're standing there trying to remember what to say. If you want to take your language skills

12:24to the next level, don't wait to try Rosetta Stone Sapphire. The newsworthy listeners can get 20% off their Rosetta Stone Sapphire subscription when they sign up today. This gets you access to Rosetta Stone Sapphire in 25 languages. Just visit rosettastone.com slash newsworthy to redeem your 20% off. That's rosettastone.com slash newsworthy. Now back to my conversation with Neil Thompson. I think it was when chat GPT first came out. We interviewed someone who compared AI to nuclear technology saying, you know, this can be used for extraordinary good, but also potentially catastrophic harm. And this was years ago saying nuclear weapons already can kill us all if humans choose to use them in that way. So do you think that's a useful comparison today or to AI at this point, create even a fundamentally different kind of danger? So I think it's still very useful to think about AI in this sort of two-sided way that it's an incredibly powerful tool and like many powerful tools that has real positives and real negatives. I think some of the ways that

13:27comparisons are to fails is that of course nuclear weapons can be something that is a lot more controlled. It's a lot more, you know, they're sort of a pretty kind of monitoring that around it. These systems are of course much more ubiquitous. You can copy them from one system to another. That means that there are a whole bunch of properties, but the risks that come with them that are pretty different. And so potentially even more dangerous than nuclear weapons. It really depends, I think, on sort of how much we're able to control them. So I think it means that they're going to be much more ubiquitous and therefore it's much more important have those controls. But exactly how that risk-all adds up is going to depend on how well we can control them and that really does emphasize why it is so important for us to do that work. So some of these frontier AI companies are saying, okay, let's slow things down and let's get China to slow things down too. President Trump and others argue that we can't slow down because whoever's first is going to win is going to have control. We can't let China get ahead. How much do we need to

14:27be thinking about what other countries are doing and how that changes our response here in America? I think it's unquestionably true that the geopolitics of this matter a lot and the United States should be thinking very carefully about what China is doing because losing a race for AI would be a very, very high cost. That said, for many of these areas, I think we should be as a planet all on the same page, right? When someone says, like, let's distribute the ability to create bio-weapons, I think we want to say no. That's something we should all agree on, even apart from the race. And yet, is that realistic? So here, I would defer to the people who have done other kind of monitoring before, but I think that this is such an important case. I would say, like, we should work very, very hard before we give up on it. This is not something that we should say, start off and say, well, it's going to be hard to monitor and therefore we're not going to do it. I think we should spend a lot of time thinking about, well, what are all the ways we could

15:28monitor because that coordination could be incredibly valuable. What do you want people to know who hear these warnings and are worried or just in general looking ahead at what could happen? For me, there are two things that are important here. So one is that there have been some dismissals of the risks of AI, people saying things like, well, these AI companies have an incentive themselves to promote this. This is just a way to make their product look more impressive or something like that. I really don't think that's what's going on here. I really believe that they trusted these are risks. But I also just think that we don't have to just take their work for it. We see in all of these other experts evaluating it, also see these risks. And so that means that we should be taking them very seriously. I think the second thing I would say is that there's much more we can be doing about this. When people have done economic analyses where they've said, well, if you think about the risks that might come with AI, how much should we be investing in safety? And the answer

16:28is, you know, could easily be $100 billion. That is so much more than we are spending on this right now. This is something where we, as a population can say, this is important to us. We see that this is happening fast. These are real risks. We can take steps to make sure that these systems are safer than they are right now by putting in the right controls on these things and spending a lot more money to figure out how to make these systems safer. That is something we can do. We just need to choose to do it. When you look at the situation now, do you think a slowdown is the right call and or at some point should there be a stopage? So it would hesitate to go for a complete stopage because I think that it's there are going to be significant gains that come with AI. I also think it would be very hard to completely stop in the sense that there are so many different ways that you can improve AI. So I think the slowdown is the more reasonable path here. The key thing here

17:29is asking the question, well, when do we think we're going to get to some of these even more dangerous capabilities and making sure that we have enough time to control those? For the average person in America, the person listening right now, what can they meaningfully do to influence how AI develops? The biggest thing that you can do is to advocate for some national standards on the safety side of things. Most of the work that is being done right now is being done at the level of the states. And I think that while that can be a useful thing, I think that a national level one would be even more useful because what you really want is these companies focused on doing as rigorous a job as possible. If what we instead get is each of these companies trying to do sort of 20 different things and doing all of them sort of poorly, that is a lot less good than having one standard that they're doing really well. And so I think having that national standard going to your congressperson and saying this is something that is important to me, I think is something that everyone can do with that would be very helpful.

18:29Anything else you want to add or just a final takeaway for our listeners? I would just say to folks that this is something where technologies, as we usually think about them, our intuitions don't really carry through to AI. And that's important. And what I mean by that is for most of the technologies we have experienced with things take many, many years to have much of an impact. Think of electric vehicles and you say, oh, yes, it's an amazing discovery, you know, to be able to go and put these in place. And yet, there's still a relatively small fraction of all the cars and things like that. And so our instinct is to say, well, we might have decades to deal with these things. And that is just not the pace that AI is developing. And so as we look forward and think about both the risks, but also the opportunities of AI, it's really important that this is something that is happening rapidly, not something that we take our time with. Thank you so much to our guest today, Neil Thompson from MIT's Future Tech. And thank you for listening. Please share this episode and join us again during the week for the latest about AI and all the

19:31other news to know each day. We provide 15 minute daily news roundups every Monday through Friday. Join us again on Monday for the latest. Until then, have a great rest of your weekend.