June 26, 2026
The Majority of the Internet Is Not Human
This post is part of my Medium blog.
If you've run a website for any meaningful chunk of the last twenty years, you already knew this. Some percentage of your traffic was always bots. Googlebot crawling your pages to build an index. Bingbot doing the same thing, less effectively. Price scrapers hitting your e-commerce site every six seconds. Monitoring agents pinging your endpoints to see if they were alive. RSS readers pulling your feed at intervals. None of this was new. None of it was alarming. You looked at your analytics, you saw that 30 or 40 percent of your traffic wasn't human, and you moved on with your day.
Now the number is over 50 percent and everyone's losing their minds.
HUMAN Security, a cybersecurity company that tracks bot activity across a network processing trillions of interactions per week, reported in their 2026 benchmarks that AI-driven traffic exceeded 50 percent of all internet traffic sometime in mid-2025.[^1] Cloudflare, which handles roughly 20 percent of all web traffic, shows the same direction — AI crawler traffic surged 24 percent year-over-year, with OpenAI's GPTBot going from 4.7 percent of crawler traffic to 11.7 percent and Anthropic's ClaudeBot going from 6 percent to nearly 10 percent in a year.[^2]
These are real numbers from real networks. The headline is true. The panic is misplaced.
You're part of it
When I send an agent to research a topic, it fetches pages, reads content, follows links, and gathers information. When you ask your AI assistant to summarize a URL, it fetches that page. When a coding tool pulls documentation, it's making HTTP requests. When a search agent retrieves results and reads the top five sources, that's five page loads that no human ever saw.
I do this constantly. You probably do too. My weekly blog briefing agent fetches Hacker News, runs web searches across six categories, reads the top results, and compiles a trend report. That's dozens of HTTP requests per run, none of them from a browser I opened. My research workflow for this very post — fetching Cloudflare's data, pulling the HUMAN Security report, checking GitHub's changelog — was done by agents making HTTP requests. I generated bot traffic to write a post about bot traffic.
The people writing the "bots are taking over" headlines are using agents to research the story. The people reading those headlines are often using AI summaries to consume them. We're not watching bot traffic grow from the outside. We're generating it. The panic is about a thing we're doing to ourselves.
What changed and what didn't
The web has always had automated traffic. What changed isn't the existence of bots — it's the volume and the purpose. Traditional bots were indexing. Googlebot crawled your site so it could appear in search results. That was a deal: Google takes your content, indexes it, and sends users back to you via links. You got traffic in exchange. The bot was a middleman, not a dead end.
AI training crawlers are different. They take content and send nothing back. Cloudflare tracks what they call the "crawl-to-refer imbalance" — how many pages a bot crawls for every one visitor it refers to the original site. Anthropic's ClaudeBot crawled 38,000 pages for every one visitor it sent to a site in July 2025. OpenAI's GPTBot sends almost no referral traffic.[^2] These bots aren't middlemen. They're consumers.
But here's the thing: if you've been running a website for twenty years, you already knew some of your traffic was consumptive rather than referential. Price scrapers weren't sending you customers. Competitor bots weren't indexing you for search. They were taking your data and leaving. The difference is that price scrapers served a purpose you could justify — they fed comparison shopping engines that drove purchasing decisions, and sometimes that meant actual sales. You didn't always turn those off. AI training crawlers take your content to build a model that will answer questions without ever sending anyone to your site. The scale is different, but more importantly, the value exchange is gone.
The volume is the real shift. Training now drives 80 percent of AI bot activity, up from 72 percent a year ago.[^2] These aren't bots serving users. They're bots building datasets. And the dataset-building phase will eventually end — once you've crawled the internet once or twice, you don't need to do it again at the same intensity. The training crawl is a finite project. It just happens to be enormous while it lasts.
Who should actually worry
The curveball in this story is that the panic is real for exactly one group: advertisers. And advertisers have already ruined the web.
Think about what the ad-supported web gave us. Cookie consent banners that don't protect anyone. Autoplay video ads. Pop-ups that cover the article you're trying to read. Tracking pixels that follow you across sites. Surveillance infrastructure dressed up as "personalization." An entire industry built on the premise that your attention is a product to be sold to the highest bidder, and that the content you actually came for is just the bait.
The web's economic model — the one that's supposedly being dismantled by AI crawlers — was an advertising model. Search engines crawled your site, sent you traffic, and that traffic saw ads. The ads paid for the content. The content paid for the hosting. The whole thing worked, sort of, if you didn't look too closely at what it was doing to the user experience.
Google referrals to news sites fell 9 percent from January to March 2025, and 15 percent from January to April.[^2] That decline coincides with the expansion of AI Overviews — Google's AI-generated summaries that appear at the top of search results. The user gets the answer. The publisher gets nothing. The publisher loses the ad impression, the tracking pixel, the cookie consent banner, the autoplay video, the popup that covered the text.
I'm supposed to be upset about this?
The publishers who are loudest about AI traffic "stealing" content built their businesses on an advertising model that treated users as inventory. They optimized for ad density, not readability. They let their pages become unusable — 3 MB of JavaScript, 14 tracking scripts, a cookie banner, a newsletter popup, and a paragraph of content wedged between two ad units. They did this to themselves. The AI crawler didn't make the web worse for humans. The advertising model did that, over twenty years, one popup at a time.
The real picture
The "majority of traffic is bots" headline is true. It was always partially true. It's more true now, and the volume will keep growing as more people use agents to do things they used to do in browsers. But the framing — that this is a crisis for the open web — is wrong in a specific way.
The open web was already in trouble. Not from bots. From an advertising model that made every page a hostile experience. From publishers who treated readers as ad inventory. From an attention economy that incentivized clickbait, engagement farming, and content that existed solely to generate impressions.
AI traffic didn't break the web. It's walking into a house that was already on fire.
The bots aren't the problem. The ad-supported web was the problem. And if the bots end up being the thing that finally kills the advertising model that made the web unreadable, that's not a crisis. That's a cleanup.
Every time I try to read an article, I'm being sold something. I looked at a pair of hiking boots once in 2023 and the internet spent six months trying to sell me the same pair, as if I might have forgotten I already bought them. I read one article about back pain and now every site I visit thinks I'm in the market for a mattress. The web's advertising model doesn't know the difference between interest and identity. It just knows you touched something once, and now it will follow you forever, showing you that thing, on every site, on every device, until you die.
That's the model the publishers are defending. That's the ecosystem that AI crawlers are supposedly destroying. I'm supposed to be worried about that?
[^1]: HUMAN Security, "2026 State of AI Traffic & Cyberthreat Benchmarks," https://www.humansecurity.com/learn/resources/2026-state-of-ai-traffic-cyberthreat-benchmarks. HUMAN's network processes trillions of interactions per week across enterprise, consumer, and mobile channels.
[^2]: Cloudflare, "The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals," https://blog.cloudflare.com/crawlers-click-ai-bots-training. Data from Cloudflare Radar, which analyzes traffic across Cloudflare's global network (~20% of all web traffic). Additional data from "From Googlebot to GPTBot: Who's crawling your site in 2025," https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/.
In Redundant, the first book in The Condition Set trilogy, Rob Coleman runs the FinOps review that names the waste nobody wants to hear about. The numbers don't change. The question is who they get used against.