Reddit Will Block the Internet Archive  ## Metadata - Author: [[Jay Peters]] - Full Title: Reddit Will Block the Internet Archive - Category: #articles - Document Note: --- tags: [Assign relevant tags or keywords to the notee to facilitate organization and retrieval. Add a # before each tag. Tags cannot have special characters. Do not use the tag #articles] - #Reddit - #InternetArchive - #WaybackMachine Author: Jay Peters Source: The Verge Reference: https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit date created: 2025-08-16 12:00 --- # 🧭 Title: Reddit will block the Internet Archive ## 🧾 Table of Contents: - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [♿ Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-diversity-equity-inclusion-and-belonging-assessment) - [🧩 Logical validity (incl. tone & fallacies)](#-logical-validity-incl-tone--fallacies) - [🎯 Conclusion](#-conclusion) --- ## 📝 Summary: - 🔹 **What happened:** Reddit says it **caught AI companies scraping** Reddit content from the Internet Archive’s **Wayback Machine** and will **block the Archive from indexing most Reddit pages** (posts, comments, profiles). Only the **reddit.com homepage** will remain indexable. - 🔹 **Why Reddit says it matters:** It frames the move as **privacy and policy compliance**, including deletion of removed content and preventing **policy-violating scraping** through third parties. - 🔹 **Context:** Reddit has **monetized access** to its corpus via **paid data deals** (e.g., with Google and OpenAI) and has **tightened robots/crawler rules**. It **sued Anthropic** in June 2025 over alleged scraping. - 🔹 **Impact:** Researchers, journalists, moderators, and the public will have **reduced archival visibility** into Reddit discussions, moderation history, and web history more broadly. --- ## 📖 Main Content: - 🔹 **Scope of the block** - Reddit will prevent the Wayback Machine from **crawling post pages, comments, and profiles**. - **Homepage-only** archiving remains, which mainly captures **what was trending** on a day, not the underlying threads. - 🔹 **Reddit’s stated rationale** - Reddit argues that **some AI companies used Wayback to evade platform policies**, scraping Reddit data indirectly. - It cites **user privacy**, honoring **deleted/removed content**, and the Archive’s need to **“defend their site” and comply** with platform rules. - 🔹 **Negotiation status** - Reddit says limits will **ramp up** and that it **notified the Internet Archive** beforehand. - The Wayback Machine’s director **Mark Graham** indicates a **“longstanding relationship”** and **ongoing discussions**—leaving room for changes. - 🔹 **Wider pattern** - Since 2024, Reddit has **restricted bots/search engines** and **licensed data** to selected AI companies (e.g., **Google**) while **pressuring others** to pay. - The **Anthropic lawsuit (June 2025)** fits the trend: **enforce terms + monetize data**. - 🔹 **Who’s affected** - **Researchers & journalists:** lose a key tool to verify past claims, community sentiment, and policy enforcement timelines. - **Open-source maintainers & digital archivists:** less reproducibility for studies that cross-reference Reddit threads. - **Moderators & transparency advocates:** harder to audit **post removals**, **policy shifts**, and **content provenance**. ### 🗒️ Notes - 🔹 **Key dates (timestamped links)** - **2025-08-11 13:00 EDT** — Publication of Verge report: https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit - **2025-06-04** — Reddit **sues Anthropic** over alleged scraping: https://www.theverge.com/ai-artificial-intelligence/679768/reddit-sues-anthropic-alleging-its-bots-accessed-reddit-more-than-100000-times-since-last-july - **2024-07-24** — Reddit **blocking major search engines** unless they pay: https://www.theverge.com/2024/7/24/24205244/reddit-blocking-search-engine-crawlers-ai-bot-google - **2024-06-25** — Reddit **escalates fight** vs AI bots via robots/crawler changes: https://www.theverge.com/2024/6/25/24185984/reddit-robots-txt-fight-ai-bots-scraping-crawlers - **2024-02-21/22** — Reddit–**Google** ~$60M/year data deal reports: - https://www.theverge.com/2024/2/22/24080165/google-reddit-ai-training-data - https://www.theverge.com/2024/2/22/24079831/google-is-reportedly-reddits-60-million-per-year-ai-content-licensing-customer - **2017-04-17** — Internet Archive explains **robots.txt isn’t ideal** for web archives: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ - 🔹 **Archive policy backdrop** - IA historically honored **robots.txt** but evolved to **ignore it in many cases**, emphasizing explicit removal requests—especially for public-interest archiving. - Reddit’s concerns include **removed content** and **privacy expectations**—areas where IA offers **request-based removals** and policy processes. - 🔹 **Strategic read** - The block reduces **free third-party access paths** to Reddit data (direct or via IA), aligning with **data licensing** strategy. - Public-interest archivability vs. platform control creates a **governance tension** with implications for **transparency** and **historical record**. --- ## 📖 References: - 🔹 The Verge report (Jay Peters): https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit - 🔹 IA robots.txt policy explainer (Mark Graham): https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ - 🔹 Reddit–Google licensing coverage: https://www.theverge.com/2024/2/22/24080165/google-reddit-ai-training-data - 🔹 Reddit bot/search engine restrictions: https://www.theverge.com/2024/6/25/24185984/reddit-robots-txt-fight-ai-bots-scraping-crawlers - 🔹 Reddit blocks search engines unless paid: https://www.theverge.com/2024/7/24/24205244/reddit-blocking-search-engine-crawlers-ai-bot-google - 🔹 Reddit sues Anthropic: https://www.theverge.com/ai-artificial-intelligence/679768/reddit-sues-anthropic-alleging-its-bots-accessed-reddit-more-than-100000-times-since-last-july --- ## 🗣️ Paraphrase: - 🔹 **Para 1 (policy shift):** Reddit says AI firms abused the Wayback Machine to harvest Reddit data against its rules. So, the company will block the Internet Archive from indexing nearly all Reddit content, keeping only the homepage open for archiving. - 🔹 **Para 2 (privacy & compliance angle):** Reddit argues the Archive must better protect its platform and respect platform policies—like honoring deletions—before full access resumes. Until then, Reddit will limit IA access to safeguard users. --- ## 📰 Newsletter: - 🔹 **Headline:** *Archiving vs. AI: Reddit Limits Wayback Machine — What Devs Should Know* - 🔹 **Blurb:** This week, Reddit moved to **limit the Internet Archive’s Wayback Machine** to the **homepage-only** view, citing **AI scraping** and **privacy**. For developers, product folks, and docs teams, this is a reminder to **design for link rot** and **proof trails**. Practical tips: - Save critical references with **Perma.cc** and **Memento Time Travel** (mementoweb.org) to retain citations over time. - Use the Internet Archive’s **Save Page Now** for your project docs and changelogs (when permitted): https://web.archive.org/save - In WordPress, surface **permalink stability** and **changelogs** in your readmes; consider adding **“Last-Updated” badges** and **checksum/signing** for release artifacts. - Learning web dev? Bookmark **MDN** (https://developer.mozilla.org/) and **Learn WordPress** workshops (https://learn.wordpress.org/). - Read the Verge piece: https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit. Friendly take: respect privacy, support archives, and **keep receipts**. You - Summary: It’s another move to protect against AI scraping. - URL: https://www.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit The company says that AI companies have scraped data from the Wayback Machine, so it’s going to limit what the Wayback Machine can access. Aug 11, 2025, 1:00 PM EDT Image: The Verge Jay Peters  is a news editor covering technology, gaming, and more. He joined The Verge in 2019 after nearly two years at Techmeme. Reddit says that it has caught AI companies scraping its data from the Internet Archive’s Wayback Machine, so it’s going to start blocking the Internet Archive from indexing the vast majority of Reddit. The Wayback Machine will no longer be able to crawl post detail pages, comments, or profiles; instead, it will only be able to index the Reddit.com homepage, which effectively means Internet Archive will only be able to archive insights into which news headlines and posts were most popular on a given day. ”Internet Archive provides a service to the open web, but we’ve been made aware of instances where AI companies violate platform policies, including ours, and scrape data from the Wayback Machine,” spokesperson Tim Rathschmidt tells *The Verge*. The Internet Archive’s mission is to keep a digital archive of websites on the internet and [“other cultural artifacts,”](https://archive.org/about/) and the Wayback Machine is a tool you can use to look at pages as they appeared on certain dates, but Reddit believes not all of its content should be archived that way. “Until they’re able to defend their site and comply with platform policies (e.g., respecting user privacy, re: deleting removed content) we’re limiting some of their access to Reddit data to protect redditors,” Rathschmidt says. The limits will start “ramping up” today, and Reddit says it reached out to the Internet Archive “in advance” to “inform them of the limits before they go into effect,” according to Rathschmidt. He says Reddit has also “raised concerns” about the ability of people to scrape content from the Internet Archive in the past. Reddit has a recent history of cutting off access to scraper tools as AI companies have begun to use (and abuse) them en masse, but it’s willing to provide that data if companies pay. Last year, Reddit struck [a deal with Google](https://www.theverge.com/2024/2/22/24080165/google-reddit-ai-training-data) for both Google Search and AI training data early last year, and a few months later, it started blocking major search engines from crawling its data [unless they pay](https://www.theverge.com/2024/7/24/24205244/reddit-blocking-search-engine-crawlers-ai-bot-google). It also said its infamous [API changes from 2023](https://www.theverge.com/2023/4/18/23688463/reddit-developer-api-terms-change-monetization-ai), which forced some third-party apps to shut down, [leading to protests](https://www.theverge.com/23779477/reddit-protest-blackouts-crushed), were because those APIs were abused to train AI models. Reddit also struck an AI deal with [OpenAI](https://www.theverge.com/2024/5/16/24158529/reddit-openai-chatgpt-api-access-advertising), but it [sued Anthropic](https://www.theverge.com/ai-artificial-intelligence/679768/reddit-sues-anthropic-alleging-its-bots-accessed-reddit-more-than-100000-times-since-last-july) in June, claiming Anthropic was still scraping from Reddit even after Anthropic [said it](https://www.theverge.com/2024/7/31/24210565/reddit-microsoft-anthropic-perplexity-pay-ai-search) wasn’t scraping anymore. “We have a longstanding relationship with Reddit and continue to have ongoing discussions about this matter,” Mark Graham, director of the Wayback Machine, says in a statement to *The Verge*. ***Update, August 11th**: Added statement from the Wayback Machine.* **Follow topics and authors** from this story to see more like this in your personalized homepage feed and to receive email updates. * Jay Peters, Reddit Will Block the Internet Archive, Jay Peters, 01k2ssxnb57y9gzpnjfxbhx9dk.