The debate of whether AI can kill us has been everywhere lately, but what would an actual AI catastrophe look like? This week, Brian Barrett, Zoë Schiffer, and Leah Feiger break down three real-world scenarios experts worry the most about—including hacked water supplies, bioweapons, and autonomous robots. We also discuss the highlights of Salesforce’s annual conference, where OpenAI’s Sam Altman and Anthropic’s Dario Amodei defended their positions on AI safety. Plus, Leah tells us how this backlash against AI has become a rare bipartisan flash point, to the point that Bernie Sanders and Steve Bannon shared a stage over it this week.
China Isn’t Buying Silicon Valley’s Call for an AI Slowdown
Washington Won’t Be Regulating AI Anytime Soon
Why So Many AI Researchers Think the Machines Could Kill Everyone
OpenAI Wants to Know if an AI Industry Slowdown Would Even Be Legal
You can follow Brian Barrett on Bluesky at @brbarrett , Zoë Schiffer on Bluesky at @zoeschiffer , and Leah Feiger on Bluesky at @leahfeiger . Write to us at [email protected] .
How to Listen
You can always listen to this week's podcast through the audio player on this page, but if you want to subscribe for free to get every episode, here's how:
If you're on an iPhone or iPad, open the app called Podcasts, or just tap this link . You can also download an app like Overcast or Pocket Casts and search for “uncanny valley.” We’re on Spotify too.
Transcript
Note: This is an automated transcript, which may contain errors.
Zoë Schiffer : Welcome to WIRED's Uncanny Valley . I'm Zoë Schiffer, contributing editor.
Brian Barrett : I'm Brian Barrett, executive editor.
Leah Feiger : And I'm Leah Feiger, director of politics and science.
Zoë Schiffer : Today on the show, we're diving into the debate about whether AI is going to kill us all. And instead of just adding to the pile of hot takes that already exist, we're going to walk you through real, specific scenarios to the best of our ability about how AI would actually do this, and why the tech CEOs and politicians cannot seem to agree on the way forward. OK, guys, so the debate about AI safety has officially escaped containment.
Brian Barrett : Zoë, I see what you did there. Wow.
Zoë Schiffer : You see what I did there? That's the thing that's really spread uncontrollably. AI researcher Jacob Coxon resigned from Anthropic and posted on X that, "The people building AI earnestly believe that it could kill us all by the end of the decade." Anthropic's CEO Dario Amodei has called for a slowdown . They call this, "pacing the frontier." And Sam Altman and Elon Musk have actually signed on, a rare moment of agreement. But honestly, with all of this talk about AI wiping out humanity, I have been left with a very basic question, which is, how exactly does this happen? What does AI going completely rogue actually look like?
Leah Feiger : I'm so happy we're doing this. We had such a great podcast last week with WIRED's Will Knight talking about everyone being concerned about this, but I want like a breakdown. I need you to just take me through every single scenario from bioweapons to nuclear codes, to what are we really talking about here when we're talking about AI deathbots.
Zoë Schiffer : We are going to do our very best. I did spend the most time I've ever spent reading LessWrong over the last two days trying to figure this out. They love a scenario-planning exercise, but just to get everyone up to speed on kind of the latest, there was an event this week hosted by Salesforce where Sam Altman, Dario Amodei, and NVIDIA's Jensen Huang were all in attendance, and naturally they had to talk about AI safety. Let's take a listen at what Sam Altman had to say.
Sam Altman, archival audio: I think the world is right to be afraid of this. So we have these two big challenges. We have the potential of a loss-of-control accident or some other serious thing that could go wrong. We have the potential of way too much power concentration and people fear that those developing AI could exert their worldview. And there is this sort of narrow path we have to navigate with pragmatism, with steadiness, with the ability for the world to trust that we'll make dependable decisions, consistent decisions through this. And I think people are saying, "Well, this now seems pretty important. Let's get it right."
Leah Feiger : I'm definitely in the camp that says we should be very concerned about this technology, how it's getting regulated, how it's getting utilized, the people in control. I don't disagree. And on the other hand, this also just reads like a fever-dream pitch to someone's massive IPO in the making.
Brian Barrett : I don't know. I get the marketing argument, but I think we're past that a little bit. I feel like—
Brian Barrett : If anything, well, if anything, Sam Altman has been using "the existential threat" as a reason not to go public yet. He's saying, "No, this is too dangerous. We have to wait until next year." Is that really why? I don't know, but there are probably some business reasons, too. I get why Anthropic and others saying, "Look, we're building something that's basically as strong as a nuclear weapon," felt like marketing. Something that's like, "Oh, actually I'm quitting because I think that I'm building something that will actually destroy the world," feels different. I feel like there's a substantive difference.
Leah Feiger : The quitting part feels different to me, but this is still Sam Altman. His financial future is on the line here with this and it's hard ... At least it's hard for me to parse that as being a way. "We are building the biggest, bestest, most powerful robot ever, and you guys should be kind of scared about it. And don't worry, we're going to take steps. They're not going to be regulatory steps, but there'll be steps of some kind." That still feels like a marketing ploy in some worlds.
Brian Barrett : I guess. I guess the nature of the threat, though, and we'll get into that later, is more that like, "Oh, we don't know how to control these things. They're going to go nuts and we're not going to be able to stop it." That feels like, "We're not competent enough right now to do anything about it," which feels like less of a marketing-
Leah Feiger : That's fair. What did everyone else at the seminar say, Zoë?
Zoë Schiffer : So Dario was introduced by Salesforce's CEO, Marc Benioff, who is very laissez-faire and does not seem worried about AI's existential risks.
Marc Benioff, archival audio : Dario, hello Dario. How are you?
Marc Benioff, archival audio : Great to have you. Fantastic. Great job, Dario. You're doing an amazing job leading the AI industry, guiding it all right from our Slack headquarters building. We watch you every day. Give us your state, where your mind is right now.
Dario Amodei, archival audio : Yeah, obviously it's very tempting to attack your competitor and say, "These guys are unsafe; we're safe." But I think the more responsible way to respond to it is to say, "Hey, first of all, let's look at our own record. We may not have had this big high-profile incident, but I'm sure we're not perfect. Let's look at everything, and we can always be better. Let's make our practices better." Then, let's organize the rest of the industry and say, "What can we do to set standards for everyone? What can we do to make everyone's standards better?" And then, finally, there should be an international component to it. And that's essentially the three-step plan that I laid out in the essay.
Brian Barrett : Two things. Two things real quick. One, Anthropic did have its own high-profile incident. An Anthropic agent—
Brian Barrett : ... did like extensive social engineering to try to trick a GitHub ... try to put a bad pull request into a GitHub repo? Anthropic has its issues here. The other thing is, too, Dario has this three-step plan. We can talk about that a little bit more later, but one of the steps of the plan is like, figure out China TBD, in a way that's just so unhelpful.
Leah Feiger : International. International.
Zoë Schiffer : WIRED editor Louise Matsakis just published, I thought, a really good piece on this. The idea that Dario can say, "We need an international plan. We need to figure out China . We need to negotiate in some way so that China's on board with this slowdown. But the prerequisite to negotiation is that we have to make sure that we regulate access to Chinese models in a way that keeps US AI far ahead." And China's like, "That doesn't feel like a good negotiation."
Brian Barrett : Why would we do that, Dario? Why?
Leah Feiger : Truly in that 30-second spiel, I'm just recounting every single person that he's managed to piss off, which even referencing his paper, Donald Trump was incensed by it and looked at it as such a mechanism for slowdown, that it has possibly been a linchpin for halting any sort of regulation that would happen because he was so mad about it. I have to say, hats off, being able to piss off every single person, even those that are theoretically in your economic corner is, my God, chef's kiss, impressive.
Zoë Schiffer : OK. So to close out the CEO statements, Jensen Huang had some thought of his own, and it seems like something that they all agree on is that AI safety matters as long as the companies can dictate the terms.
Jensen Huang, archival audio : Well, safety is paramount. In a lot of ways, it's job one. If you build a product or a service and you're not confident in its functionality, capability, or safety, then don't release it. And so that's a very obvious thing to do. You pace yourself until you are confident you're releasing something that the market would appreciate. The market forces are already there. We don't need any new laws. We don't need any new regulations.
Leah Feiger : I love that Jensen has this role that he currently has. Nvidia wasn't always the glorious company that it is today, but now he can really sit up there and say, "Why would you ... "
Zoë Schiffer : The crux of this argument, the crux of so many of these arguments that are anti-regulation here is that we're supposed to be relying on people who are not elected to dictate everything involving this, that it will impact everyone's lives regardless if they're using these services or not.
Zoë Schiffer : The invisible hand of the market is going to figure it out.
Brian Barrett : Oh, Leah forgot about the invisible hand again. Common mistake.
Leah Feiger : I did. I always do. And I know, I feel like I say this every few weeks, but these are not elected officials. So the fact that they are actually playing such a significant role here continues to boggle the mind, which is why I know I also keep on about so much of this is a bipartisan issue. Being regulatory or anti-regulatory is not a Democratic/Republican divide at this moment in time.
Brian Barrett : Well, and they also haven't done a lot to engender trust up until this point, full stop, right? I always go back to Sam Altman is the same guy who thinks that everyone should scan their eyeball into an orb, because eventually we're going to need eye-scanning orbs to create a new economy out of the ashes of—
Leah Feiger : Nothing incenses Brian Barrett like the orb.
Brian Barrett : The orb. I am obsessed with the orb. I think people forget the orb guy is the same guy as the AI guy, and I think holding those two in my brain at the same time makes every part of me hurt.
Zoë Schiffer : This kind of brings us to the how-is-it-actually-going-to-kill-us story, which is, a lot of these statements, in my opinion, boil down to, this could be really bad and we should do something about it. But I'm like, what specifically are you saying could happen? And I feel like there's kind of three scenarios roughly that this falls into. AI starts hacking things. It's actually already started, and both Anthropic and OpenAI have had very high-profile incidents where the agents escaped control and hacked into other systems without the knowledge of researchers, and it was discovered later what had happened. AI creates a novel bioweapon , and that weapon is released to the public in a way that is nearly impossible to control and it kills everyone. And then there's kind of the vague AI just escapes control. It starts acting on its own. It doesn't listen to our instructions, and we don't know what could happen, but we can't do anything about it and that's scary.
Leah Feiger : OK. This is super helpful for me that we're dividing it into these three categories.
Leah Feiger : I love a bucket. I want to go through every single one of these. My first question I suppose is, where do the nuclear codes getting pressed, which bucket do they fall into?
Brian Barrett : That's the one that we actually don't have to worry about, right?
Zoë Schiffer : You talk to AI researchers, and if you really, really press them, you can get to the three buckets that I just laid out. But then if you keep talking to them, they'll be like, "Well, there are these other scenarios where AI is in a robot, the robot is walking around, and then the robot can interact with the physical world in a way that's really, really scary." But that's a little further away, and they'll admit it's a little more far-fetched.
Brian Barrett : So I think one thing, Zoë, in terms of your buckets, that I want to carve out maybe a sub-bucket or a little pail within a bucket.
Brian Barrett : No, because you were saying AI creates a bioweapon, AI hacks something, AI whatever, all of which: true. But I think there's a distinction between human-guided things happening there versus AI doing it on its own. I think a lot of people are like, "Oh, how would AI release a bioweapon? That's crazy." Which is, yeah, sure, fine. But there are also bad people in the world who might use AI to make a bioweapon and then release it, right?
Zoë Schiffer : Oh, well, I mean, that scenario is one that I think is totally valid. To me, if you say giving people access to information about how to create a bioweapon, someone is going to be dumb enough and reckless enough to actually do it. I'm like, no, totally. People are doing all sorts of horrible, dangerous, malicious things all the time.
Leah Feiger : With inspiration from the internet.
Zoë Schiffer : Right. I buy that that is really, really dangerous. AI doing it completely on its own, again, we'll talk about the scenario, but that seems to me a little more far-fetched.
Brian Barrett : I'm probably going to get pilloried for this, but I do want to take us back to the 1983 film, WarGames .
Leah Feiger : No, this is exactly what I think we should be talking about. This is my ... Yes. WarGames is the ... Yes.
Brian Barrett : Because here's the thing, AI is not going to be pushing the button or turning the keys or whatever, but in WarGames , which is about AI basically, a computer almost kicking off nuclear war, it's more that it made the US think that it was under attack by Soviet nuclear weapons. And so is there a risk then, I assume, I would hope, I'm not an expert here, that we have enough safeguards in place that that would not actually happen here, and we have enough satellite technology to be able to confirm various things. It's fine. But I think the vector that's probably of concern would be AI triggering some sort of alert that causes a catastrophic series of events.
Leah Feiger : OK. I'm so happy you brought up WarGames . Everyone's been … It's Michael Creighton this and Jurassic Park that, but no, no, no. To me, it is very much all about, almost like the teaching scenario, which is how so many of these hacks occurred before, like the unanswerable question, how do I get to the bottom of this information that made these agents act the way that they did? We're being told. To take us even further back in history, because I can't help myself, when I'm thinking of the way in which nations communicate, and it's changed very, very much over the years. Archduke Franz Ferdinand of Austria shot and killed in Sarajevo in 1914. People were marching before that for World War I. Everyone was already on the march. And so the way that antagonistic nations are communicating with each other in already times of stress to have this additional war-game scenario, or this agent thought that we were doing this, so went down this path. I don't know. I'm aware that I'm very much sounding like an insane doomer right now, but these are the things that I'm thinking about, which is, will we actually use our satellites and contact Russia or whomever before getting into attack mode? Will Trump send out a Truth Social post before anything has been confirmed? Will state secrets be shared in public by Kash Patel? I have so many questions about how, maybe even specifically our current administration would be able to deal with a scenario like that, and we're just being told to hold on, wait a second. A couple of very, very wealthy men who have done some very impressive businesses have got it under control.
Brian Barrett : We've got so many tinderboxes too that are sort of ready to be set off, whether it's Ukraine , whether it's Iran , whether it's China , the conditions are ripe for something not to go great, whether that ends all humanity or sparks just an endless series of devastating wars, who can say? Zoë, what's another scenario you got?
Zoë Schiffer : When I was trying to get to what concretely would happen, two things that people brought up were, AI is used or AI breaks into on its own a water facility, changes the chemicals in a way that's really dangerous to humans and basically poisons the water supply for major cities across the US. Earlier this year, I think nine Mexican government organizations, like utilities were hacked, and there was a big cyber report that I read yesterday about this where Claude was used to plan the attack and build some of the tools. Ultimately, experts said AI didn't invent a new kind of attack, but it made the attack a lot cheaper and faster to execute. This comes just as the FBI in July put out a warning that utilities companies in the United States were being targeted. So again, this isn't one where AI on its own is necessarily doing things, but I think the fear is, this is already happening with bad actors, bad human actors in the loop, and if you have agents that can act of their own accord, this would be something we would be worried about.
Brian Barrett : You're right that it is file under things that are already happening. In the US, it was a dozen US states, at least that we know of, that around linked hackers were at least probing the water supply there, the industrial controls. To your point, I do think it's the kind of thing whe