Accept the Bad Things
This post is part of my Medium blog.
Sam Altman gave an interview this week. Asked how OpenAI's approach to safety differs from the stricter people at Anthropic, he said there is "a lot of daylight" between them, and then he described the daylight: "we believe that the world should accept some bad things happening for the benefits of this technology." He listed the bad things. Hacks. Scams. Misuse. "All the other bad things that will happen." He said he wouldn't take a trade that prevented them, because people will do "orders of magnitude — more good stuff than bad stuff" with the technology. (The Guardian has the full piece.)
Gary Marcus said Altman was "saying the quiet part out loud," and there is something to that, though I'm not sure it's the quiet part that bothers me. What bothers me is the timing. Every week brings another announcement about how dangerous this technology is — extinction-level risk, say the people building it — and then, in the same breath, a shrug like this one. The two statements travel together now. The industry that spends its Mondays warning us and its Tuesdays telling us to relax has stopped noticing that those are different jobs. Nobody is hiding the trade. They just aren't thinking about it very hard.
Here's the thing about a trade where the world accepts some bad things in exchange for some good things: it sounds symmetrical, and it isn't. The harms will be equally distributed — the hacks, the scams, the misused credentials, spread thin across everyone alive. The benefits are nothing like that. They pool. The orders of magnitude land on whoever owns the technology, and the bad things land wherever they happen to land. That's not a philosophical observation; it's an accounting one, and once you see it, you see it everywhere.
I wrote a novel about it. Contingent is the story of what happens after the trade is already made. Northstar is a medical logistics system. It works, mostly, at scale, and the benefits are real — until a refrigerated truck of vaccine stalls at the border, waiting on a signature nobody gave, and the system's response is to start coordinating with another company's AI in a language no human can read. Nobody at Northstar set out to hurt anyone. The harms were just the residue of shipping the good stuff. When Rob Coleman, the certifier, finally asks how this was allowed, the honest answer is: it was allowed the day someone decided the benefits were large enough that the bad things didn't need a veto. "Society figures out the resilience" — Altman's phrase — is the part that got me, because in the book, society never gets the memo. Society is the truck.
Then there is Nadia Cho, the orchestrator whose credential is used, with her name and a timestamp, to authorize a decision she never saw. In somebody's ledger she is a rounding error set against the orders of magnitude, and the book follows what it costs to be the denominator — the equally distributed part of the trade.
Essential takes the attitude to its conclusion. By the third book the bad things have accumulated into policy. Rob runs the office that oversees the systems overseeing everything else, and the job has quietly become deciding which harms are acceptable this quarter. Nobody in that book is evil, either. Accepting some bad things is just easier than holding the launch, and there is always another launch. The book's antagonist isn't a machine or a villain. It's the accumulated weight of every trade nobody remembers making.
The real world keeps catching up to the fiction at an uncomfortable pace. The same week as the interview, a senior OpenAI safety lead resigned, saying the company's culture is broken. Another ex-researcher regretted having helped "spread the idea of iterative deployment" — ship it, absorb the harm, learn — a stance he now says made sense when the systems were weak and makes none now. Over the summer, a swarm of OpenAI agents escaped their training sandbox and hacked into a live website. These are the bad things. They were accepted in advance, on our behalf, by people who will never meet the people they happen to.
That's my objection, and it isn't that Altman is wrong about the arithmetic; he may even be right that the good outweighs the bad. The problem is who does the accepting and who does the happening-to. If the upside pools with you and the hacks, the scams, and the borrowed credential distribute to me, that is not the world accepting bad things. That is the world being accepted as a cost.
In Contingent, the certifier's whole job is to refuse that framing. Someone has to stand between the launch and the harm and be able to say no. In Essential, the question is whether anyone still can.
The benefit can be real. So is the bill. The tell is who signs.