They Indexed My Gitea Instance. I Got the Bill.
A story about AI crawlers, cloud bills, and why the web is getting worse.
Last week my Google Cloud bill suddenly spiked.
Not because my server got popular.
Not because I accidentally provisioned an expensive VM.
Not because I made a mistake.
The culprit?
Around five gigabytes of outbound traffic.
I run a tiny Gitea instance on an e2-micro VM. It's mostly for my own projects, documentation, experiments, and the occasional public repository. It's not GitHub. It's not serving millions of users. It's a hobby server.
So naturally my first thought was:
"Who the hell downloaded five gigabytes?"
The Investigation
At first I assumed it had to be a bug. Google Billing doesn't tell you what generated the traffic. It just tells you:
Network Data Transfer Out.
Great. So I started digging.
I checked Nginx logs.
I checked Gitea logs.
I checked monitoring graphs.
Nothing looked unusual at first. Then I started looking at User-Agent strings. And that's when it clicked.
Twenty Thousand Requests From Claude
My server wasn't being used by humans. It was being crawled.
Claude-SearchBot 20,015 requests
Amazon SearchBot 5,036 requests
MJ12bot 940 requests
SERanking 282 requests
That's over 26,000 automated requests.
Not from people. From bots.
They Didn't Just Read My Homepage
This wasn't a polite crawl. The bots were walking:
commit history
raw source files
generated binaries
archives
images
RSS feeds
Some requests were for DLLs.
Some were for ZIP archives.
Some were for generated SVGs.
They weren't interested in me. They were harvesting data.
The Best Part?
I Paid For It.
Every request looked like this.
Claude
↓
Cloudflare
↓
My Gitea Instance
↓
Google bills me
The AI company gets training data.
Cloudflare gets traffic.
Google gets paid.
I'm the only one paying the bandwidth bill.
No one asked.
No one warned me.
No one compensated me.
This Isn't About Five Gigabytes
The amount isn't important; the principle is. Imagine this at scale.
Thousands of hobby developers.
Small blogs.
Self-hosted documentation.
Academic mirrors.
Open-source projects.
Everyone pays to serve AI crawlers. The companies doing the crawling pay nothing to the origin server. The incentives are backwards.
We Built a Web Where Reading Costs the Author
For decades the web had an implicit agreement.
Search engines indexed pages.
They sent people back.
Traffic flowed both ways.
The publisher got visitors.
The search engine got an index.
AI changes that relationship.
The crawler doesn't want readers.
It wants data.
Once it has copied enough information, the user never visits your site.
The feedback loop is broken.
The crawler extracts value.
The publisher pays the infrastructure bill.
Open Source Was Never Meant To Subsidize AI
I publish code because I want people to learn from it.
Clone it.
Fork it.
Read it.
Open source is built on sharing.
It was never an agreement that trillion-dollar companies could continuously scrape everything while sending the bandwidth bill back to hobby developers.
Those are two very different ideas.
The Invisible Cost
Most people won't notice.
Five gigabytes.
Ten gigabytes.
A few dollars.
But multiply that by:
thousands of self-hosted Git servers
personal blogs
documentation sites
package mirrors
Suddenly the economics become obvious.
The AI industry has found a way to outsource part of its infrastructure costs onto everyone else.
So What Did I Do?
I blocked them.
Cloudflare has AI bot blocking.
robots.txt.
Rate limiting.
Nginx rules.
Not because I dislike people reading my work.
Because I dislike paying for companies to scrape it.
This Isn't Anti-AI
I use AI every day.
I build tooling.
I use language models constantly.
This isn't an argument against AI.
It's an argument against a business model where the costs are socialized and the benefits are privatized.
If you want my bandwidth...
Pay for it.
Or don't crawl my server.