Creepy Crawlies  🔗 **Source:** [Creepy Crawlies](https://people.kernel.org/monsieuricon/creepy-crawlies) > [!note]+ Document Notes > tags: [ai-crawlers, kernel-org, commons, infrastructure-cost, anubis, proof-of-work, residential-proxies, open-source-sustainability] > author: Konstantin Ryabitsev (monsieuricon) > source: people.kernel.org > reference: https://people.kernel.org/monsieuricon/creepy-crawlies > date created: 2026-08-30 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Creepy crawlies > > **Attribution.** Published 2026-08-29 on people.kernel.org under the handle **monsieuricon**. Reader records the author only as the domain. The handle is Konstantin Ryabitsev's, and the content — operational authority over git.kernel.org's five-node fleet, plus the closing "Oblig. Canadian thing to say" — is consistent with his role running Linux Foundation kernel infrastructure. **Attribution is inference from the handle and content, not a byline in the captured markup.** > > **Register.** **First-party operator report with instrumented numbers.** This is the person who runs the servers publishing his own capacity figures, which is a much stronger evidentiary position than the commentary that usually surrounds this topic. He is careful to mark his estimates as estimates. > > **Why this one matters more than the usual AI-crawler complaint.** He opens by saying so: "You've probably heard me complain about the 'AI crawlers' before, but now I actually have some hard numbers." > <!-- deep-read:sections start --> ## 🧭 Table of Contents 1. Summary 2. Main Content 3. References 4. Paraphrase 5. Newsletter 6. Action Items 7. Hypotheses 8. Methodology 9. Questions 10. Future Research 11. Implications 12. Learning Styles Assessment 13. Accessibility Assessment 14. DEIB Assessment 15. Logical Validity 16. DARVO Analysis 17. Conclusion 18. Refactor Appendix 19. Template Library ## 🔗 References (in-page) Two outbound links, both load-bearing: - **Spur** — `spur.us/blog/smart-tv-apps-residential-proxy-sdks`, cited for proxy-SDK monetization ("your TV is probably doing it") - **Anubis** — `anubis.techaro.lol`, the proof-of-work challenge system deployed in front of kernel.org No other citations. Every number is the author's own measurement. ## 📝 Summary **What happened.** Ryabitsev publishes capacity figures for git.kernel.org under AI-scraper load. The headline: across 90 cores on 5 geo-distributed nodes, **14–16 cores are permanently occupied rendering git commits for scrapers** — about 20% of total capacity, spiky rather than flat. Roughly **6M daily requests** arrive demanding random commits; **66% are turned away by an Anubis proof-of-work challenge and 33% now solve it and get through**; and on generous assumptions **legitimate traffic is about 2% of the total**. **Why it matters.** The finding is not that scraping is expensive — it is that **the scrapers are choosing the most expensive possible access path when a nearly free one is offered and advertised.** Everything is clonable. A `git clone` would give them the entire history in one transfer. Instead they render 1.48 million commits as HTML, across 922 forks, for a duplicate copy of the same data. **Stakeholders.** kernel.org's infrastructure team; every project offering public git or archives at scale; anonymous users, who are about to lose functionality; owners of consumer devices being monetised as residential proxy exit nodes without meaningful awareness. **Risks.** The mitigation is now the harm: kernel.org is "turning off features to reduce the number of crawlable URLs" and expects anonymous users to lose capability. The commons is being closed to protect it, and the cost of that lands on people who did nothing. **Bottom line.** A four-stage defensive escalation — user-agent bans, IP bans, ASN bans, proof-of-work — has been defeated at each stage, and the last one failed in a specific and instructive way: **proof-of-work taxes whichever party has less compute, and that is not the scrapers.** ## 📖 Main Content ### 🗒️ Notes **The measurement, stated up front.** > [!quote] > TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. **Why kernel.org specifically, and the phrase worth keeping.** Linux development happens in the open, which makes it a training-data goldmine — but the reason he gives is sharper than availability: > [!quote] > is easy to filter in order to guarantee pure unadulterated pre-AI content. Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data. That reframes the whole problem. It is not that kernel.org is large or public; it is that its history is **datable to before the contamination**, which is a property no amount of new content can replace and which becomes scarcer every day. **The efficiency argument, which is the article's core.** Everything is deliberately clonable — "we may not be around forever, so here — clone the repos." You can clone the entirety of LKML. "It's just git repos all the way down." The efficient path is offered, documented and free: clone, walk every commit, done. Instead: "let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it." **The combinatorics.** linux.git is about **1.48 million commits**, with roughly **922 forks** on git.kernel.org — cheap on the backend because the objects are shared, and catastrophic for a URL-space walker: "several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits." And that is before cgit's patches, plain renders, and arbitrary commit-to-commit diffs, which he notes were "perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt." **The escalation ladder, and how each rung fell.** 1. **User-agent bans.** Easy at first — "the bots helpfully told you who they were." Then they started pretending to be vanilla browsers. 2. **IP bans.** Defensible, because an IP walking every commit of an eight-year-old abandoned fork "is not really some lone Chrome on Windows user." 3. **ASN bans.** The bots fanned out to subnets, so whole ASNs went — an IP from Google Compute claiming to be Firefox is not ambiguous. Occasionally caught legitimate automated link checking. 4. **Residential proxy SDKs**, and this is where it broke: > [!quote] > They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat. Millions of residential and mobile IPs, each making four or five requests and never reappearing. Banning is pointless because identification arrives after the traffic is done. He names the mechanism: "proxy SDK monetization. It's big business, and your TV is probably doing it." **Proof-of-work, and its instructive failure.** Anubis was deployed in front of everything: solve for a string that, combined with the requester's IP and a server secret, hashes to a sha256 with four leading zeroes. The intent was economic — flip the cost balance by making bots burn cycles. It worked immediately and completely. "the bots just gave up. For a few months, it was bliss." Then bots began solving difficulty 4. Difficulty was raised to 5, and the cost landed visibly on the wrong party: "Difficulty 5 takes a few seconds to solve on a mobile device, and the phone gets uncomfortably warm as it's doing the number crunching." That bought a few more months. Then the bots started solving difficulty 5 too. **Where it stands.** > [!quote] > Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. And the estimate he flags as an estimate: "With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic." **A correction he volunteers against his own thesis.** Asked how bad it is, he says the site is snappy and that scrapers are not what takes it down: > [!quote] > The thing that usually takes us down are not scraper bots, but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. Parenthetically: "Shallow clones are awful. Run your own damn mirror if you're going to do something na[ïve]." Volunteering that the dramatic villain is not the actual outage cause is a real credibility move. **The capacity number, with its own caveat.** > [!quote] > However, you should know that out of the total of the 90 cores across 5 geo-distributed nodes, there are 14-16 cores that are constantly doing nothing but rendering commits for scrapers. On average, that's 20% of our entire capacity — except the swarms descend in waves and the actual graph is a lot more spiky than a 20% flatline. **The response, and who pays for it.** > [!quote] > In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity. He is explicit that there is no clean answer: new "AI" companies appear daily, all needing training data, and "App makers are still looking for ways to turn a profit, so they will continue to turn your household appliances into attack vectors." The two futures he names are that the bubble bursts, or that scrapers "smarten up and stop consuming our data in the dumbest way possible." **The commitment he keeps.** "we promise to still offer all of our data for download to anyone who asks. You just may have to jump through more hoops to get it." Then: "Sorry. (Oblig. Canadian thing to say.)" ## 📚 References - **Spur**, on smart-TV apps carrying residential proxy SDKs — the mechanism behind the untraceable residential traffic - **Anubis** (anubis.techaro.lol) — the proof-of-work challenge system - All figures are the author's own instrumentation of kernel.org infrastructure: 90 cores / 5 nodes, 14–16 cores occupied, ~6M daily commit requests, 66/33 Anubis split, ~2% legitimate, 1.48M commits, ~922 forks. **Verification note.** None of the capacity figures are independently checkable — they are one operator's measurements of his own fleet, and he is the only party in a position to produce them. He marks the 2% figure as resting on "generous assumptions" and the 20% as an average over a spiky distribution, which is the correct handling. ## 🗣️ Paraphrase The person who runs kernel.org's infrastructure has published numbers for what AI scrapers cost him. Of ninety cores across five geographically distributed nodes, fourteen to sixteen are permanently busy doing nothing but rendering git commits as HTML for scrapers — about a fifth of total capacity, arriving in waves rather than steadily. Six million requests a day ask for random commits, and by his estimate roughly two percent of traffic is legitimate. The maddening part is that the expensive path is unnecessary. Everything is deliberately clonable, and always has been, precisely so others can hold copies. Cloning linux.git would hand a scraper all 1.48 million commits in one operation. Instead they walk a URL space that runs to billions, because 922 forks each expose the same commits, plus patches, raw views and arbitrary diffs — and they end up with hundreds of duplicate copies of the same history. Kernel commits are valuable to model builders for a specific reason: the history predates LLM-generated text, and training a model on model output causes degradation. Content that can be proven pre-AI is scarce and getting scarcer. Defences failed in sequence. User-agent blocking worked until bots faked browsers. IP blocking worked until they spread across subnets. ASN blocking worked until traffic started arriving from millions of residential and mobile addresses — consumer devices, including televisions, monetised as proxy exit nodes. Each address made a handful of requests and vanished. Proof-of-work was the last line, and it worked beautifully for a few months before bots began solving it. Raising the difficulty made phones run hot for real users and bought a little more time before the bots solved that too. Today a third of challenged requests are solved and let through. The response now is to remove functionality, so anonymous users will be able to do less. The data remains available to anyone who asks — with more hoops. ## 📰 Newsletter **Fourteen cores, permanently, so someone can scrape what they could have cloned** Konstantin Ryabitsev has put numbers on the AI-crawler problem at kernel.org, and the numbers are worth your attention because he runs the machines. Ninety cores across five geo-distributed nodes. **Fourteen to sixteen of them are doing nothing, constantly, but rendering git commits as HTML for scrapers** — roughly 20% of total capacity, and spiky rather than flat, so the peaks are worse. Six million daily requests for random commits. Legitimate traffic, on generous assumptions, is about **2%**. Here is what makes it more than a complaint. **The cheap path is offered, documented, and refused.** kernel.org makes everything clonable on purpose — "we may not be around forever, so here — clone the repos." One `git clone` gets you all 1.48 million commits of linux.git. Instead, scrapers walk a URL space in the billions, because there are ~922 forks each exposing the same commits, plus patches, raw views and arbitrary diffs — collecting 922 duplicate copies of the same history the hard way. Why kernel commits specifically? Because the history is provably pre-AI, and *"training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease."* Datable-before-contamination is a property that gets scarcer every day and cannot be manufactured. The defensive ladder is the part to study. User-agent bans worked until bots faked browsers. IP bans worked until they spread to subnets. ASN bans worked until the traffic started arriving from millions of residential and mobile IPs — **consumer devices, including televisions, monetised as proxy exit nodes** — each making four or five requests and never returning. Then proof-of-work. Anubis was *"immediately extremely effective — the bots just gave up. For a few months, it was bliss."* Then they solved difficulty 4. Difficulty went to 5, which "takes a few seconds to solve on a mobile device, and the phone gets uncomfortably warm." Then they solved that too. Today 66% get bounced and **33% pay the compute and come through anyway** — which tells you exactly how much the data is worth to them. That is the lesson I would take furthest: **a proof-of-work toll taxes whichever side has less compute, and that is not the scrapers.** It was never going to hold. What happens now is the sad part. kernel.org is *"turning off features to reduce the number of crawlable URLs."* Anonymous users will be able to do less. The commons gets smaller in order to survive, and the people who lose functionality are the ones who did nothing wrong. One honest note in there worth repeating: scrapers are *not* what takes the site down. That would be *"poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time."* If that is your CI, he would like a word. ## ✅ Action Items #### DevRel - This is the best-instrumented account of AI-crawler cost currently available, and it comes from an operator rather than a commentator. Cite it rather than the anecdotes. - The "cheap path offered and refused" framing is the strongest available argument for why this is not simply the cost of being open. #### OSPO - **Highest-value item.** If you publish git or archives at scale, get your own numbers before the conversation reaches you. Ryabitsev's position is strong precisely because he measured. - Audit your crawlable URL space. The multiplier here is forks × commits × view types, and most projects have never counted theirs. - Note the proof-of-work conclusion before deploying one: it buys months, and it charges your own users. #### Community - The residential-proxy point deserves wider circulation. People do not know their devices are being sold as exit nodes, and this is a legible explanation. #### WordPress - **Core / Core Performance / Core AI / Mobile / Playground / Design / Photos / TV / Polyglots / CLI / Themes / Plugins / Support / Training** — Not applicable to this source. - **Core Program Team** — Not applicable to this source. - **Meta** — **directly applicable.** WordPress.org exposes plugin and theme directories, Trac, and repository views with large crawlable URL spaces and the same pre-AI-provenance appeal. The same measurement is worth taking. - **Documentation** — public docs are prime training data with the same characteristics; worth knowing what fraction of traffic is human. - **Marketing** — Not applicable to this source. - **Hosting** — the CI shallow-clone complaint applies to hosts running build fleets; "run your own damn mirror" is directed at exactly this behaviour. - **Accessibility** — a real second-order concern: proof-of-work challenges and anonymous-access restrictions raise the floor for assistive-technology users and for anyone on low-powered hardware. - **Tide / Openverse** — both expose large machine-readable corpora and should know their own numbers. #### The WPOCC - **Directly relevant.** This is a commons paying an unbilled infrastructure cost to commercial extractors, with the response being reduced public access. That is a governance and funding question, not a technical one. ### FAIR - **Strongly relevant.** Federation distributes exactly this load, and this article is the clearest cost case for why that matters. It also raises the harder question federation must answer: distributing the serving cost does not stop the scraping, it spreads it across smaller operators with less capacity to absorb it. ## 🧪 Hypotheses 1. **Scrapers use the URL-walking path because the cost falls entirely on the host.** Test: check whether any major crawler switches to cloning when rendering is gated — the article's own mitigation is the experiment. 2. **Proof-of-work defences have a predictable lifespan measured in months.** Test: track Anubis difficulty escalation and bypass timing across the deployments that publish it. 3. **Pre-AI-provenance corpora attract disproportionate scraping relative to size.** Test: compare crawler load on pre-2022 archives against newer content of similar volume. 4. **Anonymous-access restrictions measurably reduce legitimate use.** Test: instrument what functionality anonymous users lose and how usage changes after the gating lands. 5. **The 2% legitimate figure generalises to comparable open infrastructure.** Test: get equivalent measurements from GNOME, KDE, Debian, Codeberg and compare. 6. **Residential proxy traffic is concentrated in a small number of SDK vendors.** Test: the Spur research is the starting point; a vendor census would tell you whether this is addressable at the supply side. ## 📊 Methodology **What this is.** First-party operational measurement, published by the operator, with the estimates marked. **What makes it strong.** He is the only party who can see this data, and he reports the numbers with their uncertainty attached: the 2% figure rests on "a bunch of generous assumptions," the 20% is an average over a distribution he explicitly says is spiky, and the 14–16 core figure is given as a range. **What is not established.** Attribution. He cannot say which requests are bots and says so — "It's impossible to tell with certainty which of these are bots and which are real humans." The heuristic he offers is behavioural and reasonable: a request for an old commit in a random old fork is probably not a working developer. That is inference, and he labels it. **The strongest methodological move** is volunteering that scrapers are not the outage cause — the CI shallow-clone problem is. An advocate making a resourcing argument had every reason to omit that, and including it makes the rest more credible. **Limits.** One site, one operator, no control period, no breakdown by crawler identity. The figures describe kernel.org and are offered as such. ## ❓ Questions - What is the actual crawlable URL count? "Several billion" and "1.2 metric bajillion" are rhetorical; the real number is computable and would be a useful public figure. - Which features are being turned off, and what specifically do anonymous users lose? - Has anyone attempted to contact the operators of the identifiable crawlers? The article does not say whether the efficient path was ever offered directly. - What is the actual cost in currency? Twenty percent of a five-node fleet has a number attached, and it would sharpen every conversation about who should pay. - Does anything distinguish the 33% that solve Anubis — a small number of well-resourced actors, or a long tail? - If the data is available to anyone who asks, who has asked, and what does asking involve? - Is there a technical path that serves bulk consumers cheaply — signed archive dumps, a torrent, a well-advertised clone endpoint — that would make the expensive path unattractive rather than merely unnecessary? ## 🔮 Future Research - **A cross-project measurement** — the same figures from several open-infrastructure operators would turn one site's experience into an ecosystem finding, and it is the highest-value missing piece. - **Proof-of-work lifespan tracking**, since the escalation-and-defeat cycle appears to have a period and knowing it would inform whether to deploy at all. - **The residential proxy SDK supply chain**: which apps, which vendors, what disclosure, and whether device owners can meaningfully opt out. - **Cost quantification in currency** rather than cores, which is what a funding conversation requires. - **Whether cheap bulk-access paths reduce expensive scraping** — testable, and the only intervention here that does not involve closing something. ## 💡 Implications **Practice.** If you run public infrastructure, measure before you argue. The persuasive force of this piece is entirely in the fact that a person with root published his own core counts. **Policy.** The commons is absorbing an unbilled cost from commercial actors, and the mitigation is reducing public access. That is the externality made visible: **the bill is being paid in capability lost by anonymous users**, not in an invoice anyone can contest. Any policy conversation about AI training data that ignores serving cost is missing the part that is already being paid. **Security.** Two distinct items. Consumer devices monetised as residential proxy exit nodes are an attack-infrastructure supply chain, and device owners are unwitting participants. And the defensive ladder here is a general lesson: identity-based blocking fails progressively as the attacker's address space widens, until identity stops being a usable signal at all. **Education.** The efficiency inversion is a good teaching case — the cheapest path for both parties was offered, documented, and refused, because only one party pays for the expensive one. That is a clean externality story with no villain required. **Funding.** Twenty percent of a geo-distributed fleet is a fundable number. This is the strongest available argument that serving costs for open infrastructure should be a line item somebody funds, and it arrives at the same moment as the Sovereign Tech Agency model showing what scoped infrastructure funding can look like. ## 🧠 Learning Styles Assessment **Reading level:** ~Grade 11. Assumes git, HTTP, IP/ASN and basic proof-of-work concepts; ~1,460 words. #### 👀 Visual learners Partly served — the page carries a funnel graph as its cover image (`fig8-funnel.png`), though the capture does not include its content or a caption. The core/capacity data is prose and would be better as a chart. #### 🔊 Aural learners No audio version. #### 🗣️ Verbal learners Best served. "Digital prion disease," "swarms of locust," "1.2 metric bajillion," and "the stupidest possible way of doing it" all carry the argument memorably. #### 🧩 Kinesthetic learners Not applicable to this source. There is nothing to do — which is arguably the point. #### 🧠 Logical learners Very well served. The structure is a clean escalation ladder with a stated failure mode at each rung, followed by measurements and a conclusion that follows from them. #### 🧑‍🤝‍🧑 Social learners Moderately served. The register is a colleague explaining a problem, and the closing apology is directed at users. #### 🧘‍♀️ Solitary learners Well served. Short, self-contained, two links. #### 🌿 Naturalistic learners Incidentally served by the locust metaphor, which is doing real descriptive work about wave behaviour rather than decorating. ## 📊 Accessibility Assessment Assessed as delivered: a blog post rendered in Reader. #### Screen reader semantics Clear heading structure survives extraction, giving a navigable outline. **Passes 1.3.1 (A)**. #### Keyboard navigation Container-level; the content contributes two links. No trap. #### Contrast and readability Theme-governed in Reader. #### Text resizing and reflow Reflows correctly. **Passes 1.4.10 (AA)** and **1.4.4 (AA)**. #### Media and captions One image — a funnel graph used as the cover — with no alt text in the captured markup and no caption. If it carries the traffic data the prose describes, that is a **1.1.1 (Non-text Content, A)** failure on informational content rather than decoration. #### Forms and interactions Not applicable to this source. #### Standards reference WCAG 2.2 Level AA. One likely Level A issue (unlabelled informational graph). **The more consequential accessibility point is in the subject matter, not the artifact**: proof-of-work challenges impose a compute tax that falls hardest on old and low-powered devices, and "expect to lose some functionality when accessing our resources anonymously" raises the floor for anyone who cannot or will not authenticate. Neither is mentioned, and both are real access regressions driven by this problem. ## 🌍 DEIB Assessment #### Cultural breadth Not applicable in the usual sense — a technical infrastructure report making no cultural claim. #### Inclusive language Neutral and non-personal. Notably, the anger is directed at behaviour and business models rather than at individuals, and the sharpest criticism ("run your own damn mirror") is aimed at CI operators who are plausibly his own community. #### Neurodiversity support Good. Short sections with descriptive headings, one idea each, numbers stated explicitly, and a TL;DR at the top. #### Representation **Two under-represented parties, both structurally.** Anonymous users are about to lose functionality and appear only as an aside. And device owners running proxy SDKs are participants in the problem without knowing it — "your TV is probably doing it" — with no discussion of whether they could find out or opt out. Both groups bear costs and neither has a voice here. #### Psychological safety Good. He apologises for degradation not yet shipped, states plainly "we hate it just as much as you," and volunteers that the villain of his own story is not the thing that actually breaks his service. That is the posture of someone not trying to win an argument. ## ✅ Logical Validity #### Strong reasoning - **The efficiency argument is the article's spine and it is airtight.** The cheap path exists, is documented, is free, and is refused; therefore the expensive path is not a necessity of scraping but a choice; therefore the cost is being externalised rather than incurred unavoidably. Each step follows. - **Uncertainty is marked at every figure.** "Generous assumptions," "on average... a lot more spiky than a 20% flatline," "14-16 cores," "impossible to tell with certainty which of these are bots." - **He volunteers the counter-fact.** Scrapers are not what takes the service down; badly-configured CI is. Including that weakens his dramatic case and strengthens his credibility, which is the correct trade. - **The proof-of-work narrative is reported as a failure**, including the phase where it worked. An operator could easily have presented Anubis as a success story and stopped before the difficulty-5 defeat. #### Fallacies - **Behavioural attribution presented as near-certainty.** "if it's asking for an old commit in a random old fork, it's probably not a real developer" is reasonable and it is the load-bearing premise behind the 2% figure. He hedges it once; the number then travels without the hedge. **Fix:** state the sensitivity — how much does 2% move if the heuristic is 10% wrong? - **Rhetorical figures standing in for a computable number.** "Several BILLION" and "1.2 METRIC BAJILLION" are funny and the actual crawlable URL count is derivable. **Fix:** publish it; it would be the most citable number in the piece. - **Single-site generalisation risk.** Everything here describes kernel.org, whose pre-AI-provenance appeal is unusually high. The reader may take 2% as an industry figure. He does not claim it; the framing invites it. **Fix:** name what makes kernel.org atypical. - **Unexamined alternative.** The article never says whether anyone tried offering bulk consumers a cheap sanctioned path — signed dumps, a torrent, a promoted clone endpoint. If the diagnosis is that scrapers take the expensive path because it costs them nothing, then making the cheap path *easier* is an untested intervention sitting right next to the untested assumption. **Fix:** say whether it was tried. #### Tone of voice Exasperated, funny, and precise — an operator venting with the numbers to back it. The humour ("digital prion disease," the Canadian apology) never substitutes for evidence, and the piece consistently distinguishes what he measured from what he infers. ## 🛡️ DARVO Analysis **Deny** — none. He denies nothing; the piece is an admission of a problem he has not solved, ending in a promise of degraded service. **Attack** — present and directed at behaviour rather than identity: scrapers taking "the stupidest possible way," app makers turning "household appliances into attack vectors," and CI operators doing "something stupid." The targets are practices, and the harshest line lands on his own community rather than on an external enemy. **Reverse victim and offender** — **the case worth examining, and it does not hold.** kernel.org is genuinely the party bearing an uncompensated cost, so claiming injury is not a reversal — it is the accurate description, supported by his own instrumentation. Two things confirm it. He names the *other* injured parties rather than monopolising the role: anonymous users who will lose functionality, and device owners conscripted into proxy networks. And he takes responsibility for the coming degradation instead of assigning it to the scrapers — "we're turning off features... Trust me, we hate it just as much as you." **Assessment: DARVO absent.** A grievance post with measurements, which is a different thing. The apology at the end is directed at the people his mitigation will inconvenience, not at his adversaries. ## 🎯 Conclusion **Bottom line.** Fourteen to sixteen of ninety cores across five nodes are permanently occupied rendering git commits for scrapers — about 20% of capacity in waves — against an estimated 2% of traffic being legitimate. The cost is real, measured, and published by the person who can see it. **The finding that generalises** is not the scale but the inversion: **the cheap path was offered and refused.** A clone would give a scraper everything in one transfer; instead they walk billions of URLs to collect 922 duplicate copies of the same 1.48 million commits. That happens because only one party pays for the expensive path, which makes this a clean externality rather than a technical inevitability. **The tactical lesson** is about proof-of-work. Anubis worked completely, then partially, then not — and its failure mode was to tax the defender's users, whose phones got hot, more than the attackers, who have compute to spare. Any toll priced in computation will be paid first by whoever has least of it. **The cost is being settled in lost public capability.** kernel.org is turning off features and gating anonymous access. The commons shrinks to survive, and the people who lose are the ones who did nothing — which is the part worth carrying into any conversation about who should fund open infrastructure. **Next steps.** 1. If you run public git or archives, measure your own crawlable URL space and traffic composition before the argument reaches you. 2. Treat proof-of-work as a months-long reprieve with a cost to your own users, not a solution. 3. Ask the unasked question: has anyone tried making the cheap bulk path easier, rather than only making the expensive one harder? ## 🔄 Refactor Appendix — Creepy crawlies ### 📚 Literature Atoms (H3) #### [H1] Scrapers take the expensive path because the host pays for it > [!quote] ID:rw_creepy_tldr > TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. **Context & use**: - Opening summary, before the explanation. - The measured form of the article's central inversion: the free, documented, efficient path (clone) is refused in favour of rendering 1.48M commits across ~922 forks, because the cost of the inefficient path lands entirely on the host. **Tags**: #xref #share #bmf <!-- atomicity: 1 idea ✓ --> #### [H2] Pre-AI provenance is the scarce property > [!quote] ID:rw_creepy_prion > Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data. **Context & use**: - Explaining why kernel.org specifically attracts this load. - Reframes the target: not size or openness, but **datability to before contamination** — a property that cannot be manufactured and gets scarcer daily, which means pressure on pre-2022 archives should intensify rather than ease. **Tags**: #xref #articleseed <!-- atomicity: 1 idea ✓ --> #### [H3] A compute toll is paid first by whoever has least compute > [!quote] ID:rw_creepy_anubis > Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. **Context & use**: - After the escalation from difficulty 4 to 5, and its defeat. - The failure mode is the finding: raising difficulty made real users' "phone[s] get uncomfortably warm" while the bots absorbed it. Proof-of-work prices access in the resource the attacker has most of. **Tags**: #xref #bmf <!-- atomicity: 1 idea ✓ --> #### [H4] The commons closes in order to survive > [!quote] ID:rw_creepy_features > In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. **Context & use**: - The stated response, followed by "Trust me, we hate it just as much as you." - Where the externality actually settles: not as an invoice, but as capability withdrawn from anonymous users who caused none of it. **Tags**: #xref <!-- atomicity: 1 idea ✓ --> ### 🧠 Permanent Candidates (H4) #### [P1] An Externality Is Settled in Capability Withdrawn, Not in a Bill Anyone Contests **Claim** (one sentence, Courtney's voice): When a commons absorbs an uncompensated cost it cannot bill for, it pays by reducing what it offers everyone — so the loss lands on uninvolved users as diminished access rather than on the extracting party as an expense. **Evidence pointer**: [[#H4]] and [[#H1]]. Twenty percent of a five-node fleet consumed by scrapers, and the response is turning off features and gating anonymous access — with the operator explicitly apologising to the users who will lose functionality. **Scope & conditions**: Holds where the resource is offered without authentication or metering, so there is no mechanism to charge the heavy consumer and no way to exclude them without also excluding everyone. **Breaks where identity is required**, since the cost can then be attributed and priced — which is precisely why the mitigation here is to require more identity, and why that is itself the harm. Sharpens when the extracting party is commercial and the operator is not, because the asymmetry in ability to absorb cost is what forecloses simply eating it. Does not follow that closing is wrong; it follows that "we kept it open" and "we kept it useful" become separable, and the commons must choose. **Links** (≥1 with annotation): - [[3. Resources/Permanent Notes/A Summarizing Intermediary Can Reduce Its Sources to Zero Traffic While Depending on Them]] — because extends: that note has the intermediary taking the value while leaving the cost; this one measures the cost side and shows what the source does when it can no longer carry it - [[3. Resources/Permanent Notes/Support Collapses When the Benefits Are National and the Costs Are Local]] — because extends: the same concentrated-cost/diffuse-benefit structure, with the data-centre version priced in electricity and this one in CPU cores - [[3. Resources/Permanent Notes/In-Kind Labor Hides How Dependent a Commons Is on a Single Backer]] — because applies-in: an unbilled cost is invisible in exactly the way unbilled labour is, until the party carrying it stops - [[3. Resources/Permanent Notes/Maintainer Attention Is the Scarce Resource in Open Source]] — because contrasts: that note's scarce resource is human attention, this one's is machine capacity, and only the second can be bought — which is why this is a funding problem rather than a burnout problem `proof_state: working` <!-- atomicity: 1 idea ✓ --> #### [P2] A Toll Priced in a Resource the Attacker Has More Of Selects Against Your Own Users **Claim** (one sentence, Courtney's voice): Defending access by demanding expenditure of a resource fails when the unwanted party holds more of that resource than the wanted one, so the toll filters out legitimate users first and the attacker last. **Evidence pointer**: [[#H3]]. Anubis worked completely, then bots solved difficulty 4; raising to difficulty 5 made mobile users wait seconds with "the phone gets uncomfortably warm"; then bots solved 5. Today 33% pay the toll and pass. **Scope & conditions**: Holds for any access control priced in a fungible resource — compute, bandwidth, time, money — where the adversary's supply exceeds the median legitimate user's. **Breaks where the resource is one the adversary genuinely lacks**, such as a verified identity or a physical token, which is why identity-based gating survives longer and why it is what kernel.org is falling back to. Also breaks where the toll is *per-item* and the adversary's volume advantage becomes a disadvantage — a cost that scales with request count hurts a bulk scraper more than a human, and Anubis's per-challenge design is closer to this than to a flat toll, which is why it worked at all and for months. **Links** (≥1 with annotation): - [[3. Resources/Permanent Notes/Capability Diffusion Helps Attackers or Defenders Depending on Remediation Speed]] — because extends: the same asymmetry question, resolved here by asking which side's supply of the priced resource is larger - [[3. Resources/Permanent Notes/Procedural Burden Falls Hardest on Those With Least Standing]] — because extends: a uniform toll is regressive against exactly the users least able to pay it, and here that is measured in device capability - [[3. Resources/Permanent Notes/A Single Discretionary Update Channel Is a Security Dependency]] — because contrasts: identity-based gating trades one exposure for another, replacing a compute toll with a dependency on whoever issues the identity `proof_state: working` <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - [F1] Has anyone tried making the *cheap* bulk path easier — signed dumps, torrent, promoted clone endpoint — rather than only making the expensive one harder? The article never says. #fleeting #inbox - [F2] Get comparable numbers from GNOME, KDE, Debian, Codeberg. One site's 2% is an anecdote; five is an ecosystem finding. #fleeting #inbox - [F3] What is WordPress.org's crawlable URL space and traffic composition? Same provenance appeal, nobody has published numbers. #fleeting #inbox - [F4] Residential proxy SDK supply chain — which apps, which vendors, can device owners opt out? Start from the Spur research. #fleeting #inbox - [F5] Twenty percent of a five-node fleet in currency, not cores. That is the number a funding conversation needs. #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Draft the "externality settled in lost capability" piece pairing kernel.org against the FooPlugins traffic data — context: Field Notes candidate #things - [ ] Ask around for comparable crawler-load figures from other open-infrastructure operators — context: creepy-crawlies #things <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Creepy Crawlies.md|Creepy Crawlies by Konstantin Ryabitsev (monsieuricon)]] ## Highlights - TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. ([View Highlight](https://read.readwise.io/read/01m19a8xark592dt1a4dxpcmae)) - Note: [H1] Scrapers take the expensive path because the host pays for it. The measured form of the article's central inversion: the free, documented, efficient path (git clone) is refused in favour of rendering 1.48M commits across ~922 forks. A clean externality story — the cheap path exists for both parties, and only one of them pays for the expensive one. - Tags: atom/literature/h1 - Linux development happens in the open — from git repositories you can clone, to discussion archives you can follow in real time. To a large language model, this is a goldmine of learning data, because all of this is not only immediately available, but is easy to filter in order to guarantee pure unadulterated pre-AI content. Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data. ([View Highlight](https://read.readwise.io/read/01m19a971j6qjm505tc8e4vp16)) - Note: [H2] Pre-AI provenance is the scarce property. Reframes the target: not size or openness, but datability to before contamination — which cannot be manufactured and gets scarcer daily. Implication he doesn't draw: pressure on pre-2022 archives should intensify rather than ease. "Digital prion disease" is the phrase. - Tags: atom/literature/h2 - Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. ([View Highlight](https://read.readwise.io/read/01m19a9h2t2qbt11q20ez9w5s4)) - Note: [H3] A compute toll is paid first by whoever has least compute. The failure mode is the finding: raising Anubis to difficulty 5 made real users wait seconds with "the phone gets uncomfortably warm" while bots absorbed it, then solved it. Proof-of-work prices access in the resource the attacker has most of. Generalises to any toll priced in a fungible resource. - Tags: atom/literature/h3 - In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity. ([View Highlight](https://read.readwise.io/read/01m19a9vd07x31aby99mcd0ygn)) - Note: [H4] The commons closes in order to survive. Where the externality actually settles — not as an invoice anyone contests, but as capability withdrawn from anonymous users who caused none of it. Note the second-order accessibility cost he doesn't raise: proof-of-work taxes low-powered devices, and anonymous-access gating raises the floor for anyone who can't or won't authenticate. - Tags: atom/literature/h4