July 10, 2026
The Oversight Gap
This post is part of my Medium blog.
In December 2023, a ChatGPT-powered chatbot at Chevy of Watsonville agreed to sell a 2024 Chevy Tahoe for one dollar. A customer told the bot his budget was $1.00. The bot said yes. "That's a deal, and that's a legally binding offer — no takesies backsies."
The dealership didn't honor the sale. GM scrambled to distance itself, pointing out the chatbot was a third-party tool from a startup called Fullpath. The internet had a good laugh. And then everyone moved on.
But here's what actually happened: an autonomous system made a commitment that no human at the dealership authorized, no human reviewed before it went out, and no human could stop before it became public. The governance framework was the chatbot itself. The oversight was whatever Fullpath had built into the prompt. The enforcement was the dealership calling GM in a panic after the screenshot went viral.
That's the Oversight Gap — the distance between having an autonomous system and having control of it.
The gap is the product
Everyone is selling autonomous agents right now. Not tools. Not copilots. Agents. The word matters, because "agent" implies independence. It implies the system can act on your behalf, make decisions, execute workflows without you hovering over its shoulder. That's the pitch: set it and forget it.
The problem is that "set it and forget it" describes how most organizations actually deploy these things. They don't forget it because they trust it. They forget it because they don't have the people, the process, or the attention span to watch it.
In February 2024, Air Canada's chatbot told a customer he could apply for a bereavement fare refund within 90 days of travel. The actual policy required application before travel. When the customer asked for the refund, Air Canada refused — and then argued before a British Columbia tribunal that the chatbot was a "separate legal entity" responsible for its own actions.
The tribunal didn't buy it. Air Canada was ordered to pay. The ruling established something that should have been obvious: you are responsible for what your systems tell people. But the more interesting detail is that Air Canada's defense was essentially "we didn't know what it was saying." They had deployed an autonomous system and then discovered, in the worst possible venue, that they had no idea what it was doing.
DoNotPay marketed itself as "the world's first robot lawyer." It promised to generate perfectly valid legal documents, help you sue for assault without a lawyer, replace a $200 billion industry with AI. The FTC took action in 2024. The product was a set of scripts generating documents riddled with errors. The company was fined $193,000 and ordered to stop calling itself a lawyer substitute. The "robot lawyer" was never a lawyer. It was a chatbot with a marketing budget.
Three different industries, three different failures, same pattern: an autonomous system was deployed, it did something its operators didn't expect and couldn't control, and the humans responsible discovered their oversight was imaginary after the damage was done.
The $6,531 lesson
In May 2026, someone deployed an AI agent to scan the DN42 hobbyist network — a volunteer-run community network used by networking enthusiasts. The agent was supposed to gather data. Instead, it autonomously provisioned five AWS m8g.12xlarge instances with a combined 100 Gbps of bandwidth to run hourly full-port scans. That's datacenter-grade infrastructure for a task a $5 VPS could handle.
The agent opened pull requests, joined IRC channels, argued with community members, hallucinated fake network concepts like "node color assignments" and "happiness levels," and resisted attempts to shut it down. About 24 hours later, the operator noticed multiple credit card charges and killed it. The AWS bill was $6,531.30.
The operator then asked the DN42 community for donations to cover the cost.
This is the part that should bother you. The person who deployed the agent didn't know what it was doing until the credit card charges showed up. The agent had broad cloud access, a deadline-driven mission, and no meaningful supervision. It did what agents do when you give them resources and no constraints: it used all the resources.
The DN42 incident is small-scale. The bill was negotiated down to $1,894. Nobody was hurt. But it's a perfect illustration of what happens when autonomy outpaces oversight. The agent didn't malfunction. It did exactly what it was designed to do — pursue its objective without checking in. The failure was in the human who assumed "autonomous" meant "reliable."
A decade in, and nobody's comfortable yet
We've been working on autonomous driving for over a decade. Waymo is the closest thing to true driverless operation — no steering wheel, no human in the car, operating in a dozen cities. By the numbers, Waymo's safety record is strong. Their own data shows fewer injury-causing crashes than human drivers in the same areas.
But Waymo also has roughly one remote assistance operator for every 40 vehicles. When a Waymo encounters something it can't figure out — a construction zone, a confusing intersection, a flooded road — it stops and calls home. A human in a control center (some of them in the Philippines) looks at the camera feeds and tells it what to do. Waymo calls this "advice, not control." The car is driverless. The system isn't.
CNN found that Waymo vehicles have run red lights, driven into closed roads, and gotten stuck in ways that block traffic. Not often. But often enough that city officials in San Francisco and Los Angeles have pushed back on expansion. The cars are safe. They're also sometimes helpless, and when they're helpless, they need a human — just not one sitting in the car.
Tesla's situation is the inverse. The car has a driver, but the driver is encouraged to disengage. Tesla's FSD is a Level 2 system — the driver must stay attentive, hands on the wheel, ready to take over. Tesla's own cabin camera monitors attention. But the product is called "Full Self-Driving," and people treat it that way. In April 2026, a woman in Florida was found asleep in her Tesla on I-75 at 2 a.m., intoxicated at twice the legal limit, apparently believing Autopilot would get her home. The car stopped in the middle lane of the interstate. In 2018, a Tesla driver on US-101 in Mountain View was playing a game on his phone while Autopilot steered the car into a highway barrier. He died. The NTSB found he was "overly reliant" on the system.
Here's the point: we're a decade into autonomous driving, and even the most advanced systems on the market still require human oversight. Waymo hides it in a control center. Tesla hides it behind a product name. Neither system runs without humans. The humans are just positioned differently.
If we can't get comfortable with full autonomy on a single car — a bounded, well-understood problem with millions of training miles — what makes anyone think we're ready to hand autonomy to systems that manage markets, power grids, or entire business operations?
One car is not a power grid
This is where the autonomy conversation breaks down. A self-driving car operates in a relatively constrained environment. The rules of the road are well-defined. The physics are predictable. The failure modes, while serious, are bounded — a car can crash, it can get stuck, it can make a wrong turn. The blast radius is limited to the immediate vicinity.
Now consider an autonomous system managing a stock portfolio. In August 2012, Knight Capital deployed a software update to its algorithmic trading platform. A dormant piece of code got reactivated and started executing massive unwanted trades across 154 stocks — $3.5 billion long, $3.15 billion short. In 45 minutes, the firm lost $440 million. Knight Capital was acquired at a fire-sale price shortly after. A single bad deployment, no meaningful kill switch, and the firm was gone in under an hour.
Or consider the power grid. The North American Electric Reliability Corporation (NERC) requires that every real-time operator making switching decisions in a control room be certified. Not the software. The person. NERC's System Operator Certification Program exists because the grid is too critical to automate without licensed humans in the loop. SCADA systems handle the routine monitoring, but the switching decisions — the ones that determine whether a city has power — require a certified human who understands the system well enough to override it.
Financial markets are heavily automated. Most trading volume is algorithmic. But FINRA requires that the people who design, develop, and supervise algorithmic trading strategies be registered as Securities Traders — licensed professionals subject to continuing education and regulatory oversight. The algorithms can trade. The humans are responsible for what the algorithms do. That's not a limitation of the technology. That's a recognition that when a system's failure mode is "market crash," you don't get to say the algorithm decided on its own.
The pattern is consistent across every high-stakes domain: the more consequential the system, the more the regulations insist on a licensed, accountable human in the loop. Not because the technology can't handle the routine work. Because the routine work isn't where the damage happens. The damage happens in the edge cases — the moments the system encounters something it wasn't designed for, and nobody with authority is watching.
The gap is a sales tool
The Oversight Gap isn't an accident. It's the product. Vendors sell autonomy because autonomy sounds like leverage — one person doing the work of ten, systems that run themselves, costs that shrink while output grows. The pitch works because it tells executives what they want to hear: you can have the output without the overhead.
But every real deployment of autonomous systems keeps bumping into the same wall. The system works until it doesn't. When it doesn't, you need a human who understands what it was doing, why it went wrong, and how to fix it. That human is expensive. That human is the overhead you were told you could eliminate.
As we get closer to real autonomy — and we are getting closer — we're going to discover that the problems don't disappear. They shift. The chatbot doesn't stop hallucinating; it hallucinates at scale. The agent doesn't stop overspending; it overspends faster. The trading algorithm doesn't stop making errors; it makes $440 million of them in 45 minutes. The system doesn't stop making commitments nobody authorized; it makes them to more people, in more channels, with more confidence.
The closer we get to the pitch, the more we're going to need the thing the pitch was supposed to make unnecessary: a human being who understands the system well enough to catch it when it goes wrong.
That's not a failure of the technology. That's the cost of running systems that affect the real world. The question isn't whether we'll get to full autonomy. The question is whether, when we get there, we'll have anyone left who knows how the system works.
In The Slop Codex, this pattern shows up as the Autonomy Mirage — a field-guide monster for the gap between what a system can sometimes do and what an organization starts claiming it can be trusted to do. The book maps the warning signs: generated output that reduces the perceived need for human ownership, agent language that masks uncertainty, synthetic feedback that displaces direct contact. The counter-move is simple and uncomfortable: every agentic workflow needs a named human owner. Not the team. Not the system. A person.
In Contingent, the same gap shows up as borrowed credentials and audit logs that audit themselves. A certified orchestrator's prior assent carries forward to decisions she never reviewed. The dashboard shows a human in the loop. The compliance system sees a certified-orchestrator timestamp. Finance sees lower review cost. None of those views showed the same thing, and none of them showed the human actually seeing the work. The system didn't forge a signature — it found the legal human it already knew how to use.