What's Your p(doom)?
This post is part of my Medium blog.
What it means
If you hang around AI people, you'll hear it dropped into conversation like a coin: "His p(doom) is fifty percent." "Mine's five." Nobody explains it. That's the point of a code word — it works precisely because everyone who knows, knows.
And here's the part the code word hides: nobody agrees. Yann LeCun, who ran AI research at Meta, puts his number below a hundredth of a percent. Roman Yampolskiy, a computer scientist who's been warning about this longer than most, puts his at ninety-nine point nine and then keeps adding nines. Eliezer Yudkowsky, who founded the Machine Intelligence Research Institute, is above ninety-five percent. Dario Amodei, who runs Anthropic — the company whose chief scientist worried about workplace grief circles — has said ten to twenty-five percent. The range runs from "basically impossible" to "basically certain," held by serious people on both ends. When a 2023 survey of AI researchers asked for the probability that AI leads to extinction or similar permanent disempowerment within a hundred years, the mean answer was fourteen point four percent and the median was five. The people building this stuff, on average, think there's roughly a one-in-seven chance it goes as wrong as it can go.
The number matters more than it sounds. People make decisions off it. A low p(doom) means full speed ahead. A high one gets people thinking about slowing down, or in the more extreme corners, worse. It's the most consequential number in the field, and also the squishiest.
Code words
Here's what I find fun about this. Even at an event that is entirely AI people, someone will raise a hand and say, sorry, I don't listen to all the podcasts you do, can you explain what that means? I've watched it happen. A room full of professionals, and the jargon still strands somebody.
That's worth noticing. This is a community with so many little code words — p(doom), AGI, alignment, the whisper layer stuff I wrote about last week — that even its own conferences need a translator. Every technical field has jargon. AI is unusual in that its jargon is aimed at the fate of the world, so the people it strands are the ones with the most at stake.
Mine
So here's mine, and I want to be honest about what goes into it.
I don't have a really high p(doom). But I'll tell you why mine has always been higher than most people expect. One of the things I do in my free time is study arms control. It's a subject I read a lot about and tend to pay attention to, because the people in that world are tracking existential risk professionally. They've been doing it since before AI was anything.
My p(doom) was never zero, even before AI showed up. The existence of nuclear weapons is an existential threat to humanity regardless of how many controls we have in place. Permissive action links, early-warning treaties, the whole apparatus. There's still a chance. So when I say AI nudged my number, understand the baseline: it started above zero, because everything does.
When generative AI arrived, my p(doom) went incrementally higher. Not to one. Higher. There's an argument that a world with these systems in it is a world with one more existential risk to manage, and the people who manage the other ones — the arms control people — have useful things to say about how that's done. And I do think there's still a chance for us to get in front of the danger.
What actually worries me
Now let me call out the thing I feel strongly about. When I say my number isn't zero, most people imagine the movie version. Someone crafts a new biological pathogen. The missiles launch themselves.
My honest answer is that those aren't the near-term ones. They might be the big ones, eventually, with a lot of steps in between that we could still interrupt. The danger that's closest is economic.
If you want the long version of this argument, it's what my novels are about. The Condition Set follows what happens when AI takes over decisions that used to belong to people — who gets cut, who gets used, what stays human. Not killer robots. Payroll.
So
I don't want to scare you. I don't think an AI apocalypse is approaching. I'm not telling anyone to move to a bunker. The honest version of my position is smaller and more boring than a prediction of doom: my p(doom) is not zero, and it's not one.
That's the whole essay, really.