Creepy Crawlies  🔗 **Source:** [Creepy Crawlies](https://people.kernel.org/monsieuricon/creepy-crawlies) > [!note]+ Document Notes > --- > tags: [ai-crawlers, kernel-org, commons, infrastructure-cost, anubis, proof-of-work, residential-proxies, open-source-sustainability] > author: Konstantin Ryabitsev (monsieuricon) > source: people.kernel.org > reference: https://people.kernel.org/monsieuricon/creepy-crawlies > date created: 2026-08-30 > --- > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > # 🏷️ Title: Creepy crawlies > **Attribution.** Published 2026-08-29 on people.kernel.org under the handle **monsieuricon**. Reader records the author only as the domain. The handle is Konstantin Ryabitsev's, and the content — operational authority over git.kernel.org's five-node fleet, plus the closing "Oblig. Canadian thing to say" — is consistent with his role running Linux Foundation kernel infrastructure. **Attribution is inference from the handle and content, not a byline in the captured markup.** > **Register.** **First-party operator report with instrumented numbers.** This is the person who runs the servers publishing his own capacity figures, which is a much stronger evidentiary position than the commentary that usually surrounds this topic. He is careful to mark his estimates as estimates. > **Why this one matters more than the usual AI-crawler complaint.** He opens by saying so: "You've probably heard me complain about the 'AI crawlers' before, but now I actually have some hard numbers." > ## 🧭 Table of Contents > 1. Summary > 2. Main Content > 3. References > 4. Paraphrase > 5. Newsletter > 6. Action Items > 7. Hypotheses > 8. Methodology > 9. Questions > 10. Future Research > 11. Implications > 12. Learning Styles Assessment > 13. Accessibility Assessment > 14. DEIB Assessment > 15. Logical Validity > 16. DARVO Analysis > 17. Conclusion > 18. Refactor Appendix > 19. Template Library > ## 🔗 References (in-page) > Two outbound links, both load-bearing: > - **Spur** — `spur.us/blog/smart-tv-apps-residential-proxy-sdks`, cited for proxy-SDK monetization ("your TV is probably doing it") > - **Anubis** — `anubis.techaro.lol`, the proof-of-work challenge system deployed in front of kernel.org > No other citations. Every number is the author's own measurement. > ## 📝 Summary > **What happened.** Ryabitsev publishes capacity figures for git.kernel.org under AI-scraper load. The headline: across 90 cores on 5 geo-distributed nodes, **14–16 cores are permanently occupied rendering git commits for scrapers** — about 20% of total capacity, spiky rather than flat. Roughly **6M daily requests** arrive demanding random commits; **66% are turned away by an Anubis proof-of-work challenge and 33% now solve it and get through**; and on generous assumptions **legitimate traffic is about 2% of the total**. > **Why it matters.** The finding is not that scraping is expensive — it is that **the scrapers are choosing the most expensive possible access path when a nearly free one is offered and advertised.** Everything is clonable. A `git clone` would give them the entire history in one transfer. Instead they render 1.48 million commits as HTML, across 922 forks, for a duplicate copy of the same data. > **Stakeholders.** kernel.org's infrastructure team; every project offering public git or archives at scale; anonymous users, who are about to lose functionality; owners of consumer devices being monetised as residential proxy exit nodes without meaningful awareness. > **Risks.** The mitigation is now the harm: kernel.org is "turning off features to reduce the number of crawlable URLs" and expects anonymous users to lose capability. The commons is being closed to protect it, and the cost of that lands on people who did nothing. > **Bottom line.** A four-stage defensive escalation — user-agent bans, IP bans, ASN bans, proof-of-work — has been defeated at each stage, and the last one failed in a specific and instructive way: **proof-of-work taxes whichever party has less compute, and that is not the scrapers.** > ## 📖 Main Content > ### 🗒️ Notes > **The measurement, stated up front.** > > [!quote] > > TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. > **Why kernel.org specifically, and the phrase worth keeping.** Linux development happens in the open, which makes it a training-data goldmine — but the reason he gives is sharper than availability: > > [!quote] > > is easy to filter in order to guarantee pure unadulterated pre-AI content. Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data. > That reframes the whole problem. It is not that kernel.org is large or public; it is that its history is **datable to before the contamination**, which is a property no amount of new content can replace and which becomes scarcer every day. > **The efficiency argument, which is the article's core.** Everything is deliberately clonable — "we may not be around forever, so here — clone the repos." You can clone the entirety of LKML. "It's just git repos all the way down." The efficient path is offered, documented and free: clone, walk every commit, done. > Instead: "let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it." > **The combinatorics.** linux.git is about **1.48 million commits**, with roughly **922 forks** on git.kernel.org — cheap on the backend because the objects are shared, and catastrophic for a URL-space walker: "several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits." And that is before cgit's patches, plain renders, and arbitrary commit-to-commit diffs, which he notes were "perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt." > **The escalation ladder, and how each rung fell.** > 1. **User-agent bans.** Easy at first — "the bots helpfully told you who they were." Then they started pretending to be vanilla browsers. > 2. **IP bans.** Defensible, because an IP walking every commit of an eight-year-old abandoned fork "is not really some lone Chrome on Windows user." > 3. **ASN bans.** The bots fanned out to subnets, so whole ASNs went — an IP from Google Compute claiming to be Firefox is not ambiguous. Occasionally caught legitimate automated link checking. > 4. **Residential proxy SDKs**, and this is where it broke: > > [!quote] > > They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat. > Millions of residential and mobile IPs, each making four or five requests and never reappearing. Banning is pointless because identification arrives after the traffic is done. He names the mechanism: "proxy SDK monetization. It's big business, and your TV is probably doing it." > **Proof-of-work, and its instructive failure.** Anubis was deployed in front of everything: solve for a string that, combined with the requester's IP and a server secret, hashes to a sha256 with four leading zeroes. The intent was economic — flip the cost balance by making bots burn cycles. > It worked immediately and completely. "the bots just gave up. For a few months, it was bliss." > Then bots began solving difficulty 4. Difficulty was raised to 5, and the cost landed visibly on the wrong party: "Difficulty 5 takes a few seconds to solve on a mobile device, and the phone gets uncomfortably warm as it's doing the number crunching." That bought a few more months. Then the bots started solving difficulty 5 too. > **Where it stands.** > > [!quote] > > Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting thr You've probably heard me complain about the “AI crawlers” before, but now I actually have some hard numbers I can put up to show their impact. In a few words, it's bad enough to create a constant “background radiation” of system load, permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model. TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. CPU background radiation #### Why is git.kernel.org “interesting” to crawlers Linux development happens in the open — from git repositories you can clone, to discussion archives you can follow in real time. To a large language model, this is a goldmine of learning data, because all of this is not only immediately available, but is easy to filter in order to guarantee pure unadulterated pre-AI content. Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data. #### The stupidest way of doing it We make almost everything clonable, because hey — we may not be around forever, so here — clone the repos. Also, clone the archives. Grab a copy just so we're not the only ones who have it all. Seriously, it's just a “git clone” away — and then you'll have the whole history. For example, did you know you can clone the entirety of LKML and then do whatever you want with it? It's just git repos all the way down. So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done. But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it. The stupidest way of doing it At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork. Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers are doing. But wait, it's not just commits itself. You can also ask for patches, plain renders, diffs between arbitrary commits — cgit is happy to let you, which was perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt, and is *AWFUL* right about now, because we can generate 1.2 *METRIC BAJILLION* valid URLs just for a single fork of linux.git. How many valid URLs is linux.git? #### Block them Initially, this was the solution — look through the logs, find out which IPs are obvious scraper bots, and fail2ban them. At first, this was easy, because the bots helpfully told you who they were via their user-agent. Then, they wised up and started pretending that they were random vanilla browsers. So, we started banning them by IP — after all, it's easy to figure out that an IP that is trying to grab every possible commit in a 8-year-old abandoned fork of linux is not *really* some lone Chrome on Windows user who is just furiously clicking every link that comes across their screen. The bots then started fanning out to entire subnets, but this was still meh, because obviously an IP coming from Google Compute is just *pretending* to be a Firefox user. Banning the whole ASN was justified, even if this occasionally caught a random legitimate instance trying to automate link checking in commits. #### Enter... your TV? And... that's when things turned really, *really* ugly. Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back. They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat. They still do that — welcome to the wonderful world of “proxy SDK monetization.” It's big business, and [your TV is probably doing it](https://spur.us/blog/smart-tv-apps-residential-proxy-sdks). #### Make them pay When this first became a problem, oh, about a year ago, we naively thought that there was a way to make it stop. Just make the bots perform a task that would flip the economy of the whole thing upside-down by making them burn some cycles doing throwaway math. Like, calculate what string, when combined with their own IP and a secret we provide, would generate a sha256 sum with 4 leading zeroes. In other words, we put [Anubis](https://anubis.techaro.lol) in front of everything. Anubis painfulness graph It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were *mildly annoyed* but tolerated it, and the Anubis stack was easy enough to deploy everywhere. A few months later, the bots were back, solving difficulty 4. No problem, we said, let's raise difficulty to 5. The legitimate users were more annoyed now. Difficulty 5 takes a few seconds to solve on a mobile device, and the phone gets uncomfortably warm as it's doing the number crunching. However, it was effective and bought us a few more months of peace. Then... the bots started solving difficulty 5. #### Where we are now Anatomy of git.kernel.org requests Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. It's impossible to tell with certainty which of these are bots and which are real humans — but chances are, if it's asking for an old commit in a random old fork, it's *probably* not a real developer trying to do their work. With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers. #### How bad is it? Hits vs. bytes At this point, we're not quite overwhelmed — if you visit git.kernel.org, it will *likely* be snappy and responsive. The thing that usually takes us down are not scraper bots, but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. (Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.) However, you should know that out of the total of the 90 cores across 5 geo-distributed nodes, there are 14-16 cores that are constantly doing nothing but rendering commits for scrapers. On average, that's 20% of our entire capacity — except the swarms descend in waves and the actual graph is a lot more spiky than a 20% flatline. #### Where does that leave us? Unclear. Maybe the AI bubble bursts and we suddenly have a lot fewer entities out there trying to train their models. Alternatively, maybe they smarten up and stop consuming our data in the dumbest way possible. In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity. Worst of all, there are no simple solutions to the problem. Companies offering custom “AI” models still pop up daily, all of them hungry for training data. App makers are still looking for ways to turn a profit, so they will continue to turn your household appliances into attack vectors. That said, we promise to still offer all of our data for download to anyone who asks. You just may have to jump through more hoops to get it. Sorry. (Oblig. Canadian thing to say.), Creepy Crawlies, Konstantin Ryabitsev (monsieuricon), 01m199zj4yyyhzcbh74nqxa1x5.