Some Thoughts on the Open Web - CourtneyR.dev's 2nd Brain

Some Thoughts on the Open Web 🔗 **Source:** [Some Thoughts on the Open Web](https://www.mnot.net/blog/2026/01/20/open_web) > [!note]+ Document Notes > ```yaml > tags: [open-web, internet-governance, artificial-intelligence] > author: Mark Nottingham > source: mnot.net > reference: [https://www.mnot.net/blog/2026/01/20/open_web] > date created: 2026-01-20 14:30 > > --- > > # 🏷️ Title: Some Thoughts on the Open Web >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - Mark Nottingham, “Some Thoughts on the Open Web” – https://www.mnot.net/blog/2026/01/20/open_web ## 📝 Summary - What happened. - The author reflects on how the “Open Web” emerged, why it’s valuable, and how AI and shifting incentives are shrinking it. - Why it matters. - Open access to information is a public good; its erosion affects education, innovation, and democratic discourse. - Key stakeholders. - Publishers, creators, users, browser/search/AI vendors, standards bodies, and regulators. - Risks and tradeoffs. - Over‑extraction by platforms and AI vs. sustainability for publishers; openness vs. control and privacy; blocking bots vs. information access. - Bottom line. - We must redesign incentives so people *want* to publish openly, recognizing many motives and forms of “open,” instead of trying to coerce openness. ## 📖 Main Content - Core arguments: - Historically, access to and publication of information were hard and gatekept; the Web massively lowered these barriers and normalized easy, low‑friction access to content. - The Open Web is a “public good” that benefits both consumers and producers by reducing costs and friction in both directions. - People open content for diverse reasons: contributing to a commons, building reputation, social connection, ad revenue, or as a funnel into paid offerings. Policies must respect this diversity rather than enforce “purity tests.” - There are many kinds of “open”: free access, registration walls, ad‑supported, limited free views, etc. Small frictions (mandatory logins everywhere) cumulatively reduce openness but can also address privacy and sustainability. - Participation in the Open Web must be voluntary; content owners respond to incentives and can’t realistically be forced to open content. Attempts at coercion (e.g., withholding features) push them toward walled gardens and paywalls. - AI radically shifts incentives: it scrapes open content for training and then competes with that content, undermining models based on reputation, traffic, and ad/subscription funnels. - As a result, many are blocking bots, tightening paywalls, and withdrawing from the Open Web, shrinking the shared commons. - Serving content isn’t free; creators must sustain themselves, and infrastructure costs (even if cheap per request) accumulate at scale. Selective blocking of heavy or objectionable crawlers is now widespread and arguably necessary. - The real tension is about control: between maximal control by publishers and maximal freedom for reusers. Neither extreme is desirable; a sustainable balance is needed. - The path forward is to strengthen incentives to keep content open, revisiting assumptions about what “open” means and what incentive structures work under new technological shocks like AI. - Key terms: - Open Web: Web content that is easily accessible (often free and not heavily restricted), indexable, and reusable to some extent. - Public good: Something broadly available and beneficial where one person’s use doesn’t diminish another’s (non‑rivalrous). - Incentive structure: The mix of economic, social, and technical conditions that make it attractive or unattractive to publish openly. - Uncertainties / contested areas: - How equitable ad‑supported vs. pay‑per‑view models really are across income levels and regions. - Where to draw the line between legitimate crawler blocking and de‑facto enclosure. - What legal and technical norms for AI training and reuse will emerge or be enforced. ### 🗒️ Notes - The Web transformed access to knowledge by slashing friction in both publishing and discovery, disrupting traditional gatekeepers while creating new intermediaries (search engines, marketplaces). - A social norm emerged that Web content is easy to access without paywalls or complex logins; this norm isn’t universal but has had strong network effects. - The Open Web benefits: - Farmers getting up‑to‑date agricultural guidance. - Students globally accessing materials once limited to elite institutions. - Creators who gain reputation and contribute to a global commons. - Motivations for open publishing: - Altruism / contribution to commons. - Reputation and discovery. - Social connection and community. - Advertising revenue. - Freemium tactics leading into closed, paid experiences. - “Open” is a spectrum: - Completely open, no tracking. - Ad‑supported with tracking. - Search‑indexable but individual users must log in. - Metered/paywalled models with a few free views. - Advertising models can be more equitable than micro‑payments by lowering financial barriers to access, but they introduce privacy concerns and dependence on surveillance infrastructure. - Cumulative friction (e.g., ubiquitous logins and meters) gradually erodes the Open Web norm and can lock users into a patchwork of silos. - Voluntariness: - Publishers choose openness when benefits (reach, reputation, revenue) outweigh downsides (costs, misappropriation, scraping). - Coercive strategies—denying capabilities, punitive policies—risk pushing content into closed ecosystems rather than encouraging openness. - AI as a disruptive shock: - Trains on publicly accessible content without direct compensation or consent in most cases. - Generates answers that substitute for visiting the source, displacing traffic and monetization. - Converts “commons” contributions into private corporate advantage, eroding trust. - Responses: - Bot blocking, robots.txt hardening, and paywalls. - Withdrawal from truly open formats (like unmetered RSS) in some sectors. - Persistent but unevenly distributed commitment to openness among some creators. - Cost realities: - Information is non‑rivalrous, but production and delivery have real human and infrastructure costs. - High crawler demand, especially from AI companies, can create unsustainable bandwidth and operational loads. - Selective blocking of abusive or extractive clients has become common; removing that capability would likely harm the overall supply of open content. - Control tension: - One extreme: publishers want pixel‑perfect control over presentation, use cases, and audiences. - Other extreme: open‑access advocates want fully free reuse for any purpose. - Sustainable openness requires a middle path that leaves publishers enough control and benefit to participate willingly. - Way forward: - Focus on incentive design, not moralizing or coercion. - Accept multiple, pragmatic definitions of “open” that can evolve with the ecosystem. - Plan for resilience to future shocks beyond AI. ## 📚 References - “Some Thoughts on the Open Web” – https://www.mnot.net/blog/2026/01/20/open_web - Tim Berners-Lee, “Information Management: A Proposal” (original Web proposal) - W3C – https://www.w3.org/ - IETF – https://www.ietf.org/ ## 🗣️ Paraphrase - Paragraph A paraphrase. - The author explains that “Open Web” means different things, but current debates mainly center on how easily people can publish and access information without barriers. - Paragraph B paraphrase. - He argues that while the Web created an extraordinary public good by lowering friction, new forces—especially AI and changing business models—are undermining the incentives that previously kept content open, so we must rethink how to keep openness attractive and sustainable. ## 📰 Newsletter In this week’s WordPress‑adjacent reading, Mark Nottingham’s “Some Thoughts on the Open Web” is a must‑read for anyone building on or around WordPress. He traces how the Web’s low‑friction access to information became a public good—and why AI, paywalls, and heavy scraping are pressuring publishers to lock content down. For WordPress creators, this raises hard questions: how do we keep sites indexable and shareable while maintaining viable business models and protecting from abusive crawlers and model training? Think about: - Smarter caching/CDN setups to keep costs low. - Bot‑management rules that distinguish legitimate search from extractive scraping. - Clear content licensing and terms pages on your WordPress site. - Exploring open‑friendly monetization like memberships, ethical ad networks, or donation models. Recommended follow‑up resources for WordPress folks: - WordPress.org documentation on privacy and consent tools. - Guides on implementing robots.txt, XML sitemaps, and caching plugins responsibly. - Community talks on publishing sustainability in the age of AI. Open publishing is at the heart of WordPress’s mission; this essay is a timely reminder that “open” only survives if it also works for the people doing the publishing. ## ✅ Action Items #### DevRel - Clarify and document your project’s stance on Open Web participation, AI training, and crawler policies. - Prepare talks/blogs explaining how developers can be open‑friendly while protecting their content. - Produce example robots.txt and HTTP header configurations balancing openness with rate‑limiting and bot control. - Create tutorials on sustainable monetization for open‑access documentation and blogs. - Gather feedback from developer communities about AI scraping and share with product/standards teams. #### OSPO - Draft policies for when corporate content should be open vs. access‑controlled, including treatment of AI crawlers. - Clarify license guidance around data scraping, model training, and downstream use. - Integrate crawler and AI‑usage considerations into security and compliance reviews. - Maintain an inventory of open vs. closed internal web properties and their policies. - Engage upstream standards bodies (IETF/W3C) on mechanisms to express AI‑training preferences. #### Community - Host discussions on sustainable open publishing and AI impacts with maintainers and users. - Provide maintainers with simple templates for content usage policies and bot rules. - Encourage community‑run archives and mirrors for at‑risk open resources (where licenses allow). - Share case studies of projects that remain open while staying financially viable. #### WordPress - Core: ensure tools like robots.txt, XML sitemaps, and privacy settings are first‑class and easy to configure. - Core Performance: keep open sites cheap to serve through performance improvements and caching guidance. - Core AI: define guidance on how AI features in the ecosystem respect site owners’ content and choices. - Core Program Team: coordinate cross‑team efforts on openness, privacy, and AI policies. - Mobile: surface key privacy/open‑web controls in mobile publishing workflows. - Playground: provide labs demonstrating best practices for open yet protected WordPress setups. - Community: include Open Web resilience as a theme in meetups and WordCamps. - Training: add modules on Open Web economics, AI, and sustainable content strategies. - Marketing: highlight success stories of sites thriving while staying indexable and open. - Support: create help articles on bot blocking, caching, and monetization for open content. - Design: ensure consent and paywall UX is clear, honest, and minimally disruptive. - Documentation: document recommended open/closed patterns, including robots and caching examples. - Photos / TV / Polyglots / Accessibility / Meta / CLI / Hosting / Tide / Openverse / Themes / Plugins: - Align content policies and tools with a shared view of openness, crawler handling, and attribution. #### The WPCC - Programs: sponsor maintainers who commit to keeping high‑value educational content openly available. - Projects: run governance clinics on content licensing and AI policy for WordPress ecosystems. - Funding experiments: pilot grants that reward open educational resources and transparent bot policies. ### FAIR - Technical independence: encourage local archiving and provenance metadata for at‑risk open resources. - Policies: recommend transparent funding and sunset clauses for initiatives that rely on content remaining open. - Governance artifacts: provide reusable charters and agreements covering AI training rights and crawler behavior. ## 🧪 Hypotheses - H1: Publishers facing high AI scraping without compensation are more likely to move content behind paywalls or block crawlers. - H2: Clear, widely adopted technical signals about AI training preferences will slow the retreat from the Open Web. - H3: Ad‑supported open content provides more equitable access across income levels than micro‑payment models, at equal quality levels. - H4: Improving performance and hosting efficiency decreases the probability that small publishers will restrict or close their content. - H5: Communities with strong open norms and governance (e.g., WordPress ecosystem) will maintain higher levels of open content despite AI shocks. ### 📊 Methodology - Use longitudinal web crawls to track shifts in paywall prevalence and robots policies in response to AI‑related events. - Survey publishers about reasons for changing openness policies, including AI, costs, and platform behavior. - Run controlled experiments comparing user access patterns under ad‑supported vs. micro‑paywall models in diverse regions. - Measure correlation between performance optimizations (CDN use, caching) and openness (indexability, paywalls) in small sites. - Compare ecosystems (e.g., WordPress vs. proprietary CMS) on levels of open publishing and attitudes toward AI scraping. ### ❓ Questions - What technical and legal tools can express “open for humans, restricted for AI training” in a meaningful, enforceable way? - How can small publishers negotiate with large AI vendors over training and attribution, if at all? - Are there sustainable funding models for open content beyond ads and subscriptions? - What level of friction (logins, meters) is compatible with still calling something part of the “Open Web”? - How can standards bodies incorporate these incentive questions without overstepping into policy‑only realms? ### 🔮 Future Research - Comparative studies of regions where paywalls dominate vs. where open access norms remain strong. - Economic modeling of AI’s impact on traffic and revenue for different classes of publishers. - Exploration of new protocols/metadata specifically for AI training consent and compensation. - Social research into user trust and willingness to support open content directly (donations, memberships). ### 💡 Implications - Practice: Site owners must revisit crawler policies, monetization, and licensing to remain open without being exploited. - Policy: Regulators may need to address AI training on open content, balancing innovation and creator rights. - Security: High‑volume scraping can undermine availability and raise infrastructure costs; rate‑limiting and bot controls become more critical. - Education: Institutions relying on open web materials may face shrinking resources and should consider archiving and funding strategies. - Funding: Philanthropy and public funding might play larger roles in preserving open, high‑value content. ### 🧠 Learning Styles Assessment - Reading level (rough estimate): Upper high‑school / early college (Flesch‑Kincaid ~11–13) due to policy/technical vocabulary. #### 👀 Visual learners - Present: Conceptual contrasts (open vs. closed, spectrum of openness) that could map to diagrams. - Improve: Add flowcharts showing incentive changes; diagrams of stakeholders and content flows. #### 🔊 Aural learners - Present: None inherent. - Improve: Short audio summary or podcast discussion of the piece. #### 🗣️ Verbal learners - Present: Clear definitions, analogies (e.g., public good), and structured argument sections. - Improve: More concrete examples and short case narratives. #### 🧩 Kinesthetic learners - Present: None. - Improve: Hands‑on exercises like configuring robots.txt, setting up open‑friendly caching, or experimenting with paywall plugins. #### 🧠 Logical learners - Present: Causal reasoning about incentives; explicit tradeoffs; discussion of costs and benefits. - Improve: Tables comparing business models; decision trees for openness choices. #### 🧑‍🤝‍🧑 Social learners - Present: Implicit invitations to community debate. - Improve: Guided discussion prompts for meetups, working groups, and classrooms. #### 🧘‍♀️ Solitary learners - Present: Reflective essay style suitable for individual reading and note‑taking. - Improve: Reflection questions at the end (e.g., “Why am I open?” “What would make me close my content?”). #### 🌿 Naturalistic learners - Present: Limited direct links. - Improve: Analogies to environmental commons and resource management. ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Typical blog structure with headings and paragraphs should be reasonably readable. - Improve: Confirm landmark roles, skip links, and descriptive image alt text. #### Keyboard navigation - Present: Likely basic keyboard support via standard HTML. - Improve: Ensure visible focus indicators and skip‑to‑content links. #### Contrast and readability - Present: Depends on site theme; mnot.net is generally minimalist. - Improve: Verify contrast ratios, font sizes, and line spacing for long‑form reading. #### Text resizing and reflow - Present: Responsive layout likely handles moderate zoom. - Improve: Test at 200%+ zoom and with reduced‑motion preferences. #### Media and captions - Present: Primarily text; minimal media. - Improve: Provide audio version or downloadable text for offline use. #### Forms and interactions - Present: Minimal, likely just navigation and maybe comments. - Improve: Ensure any forms have labels, clear errors, and adequate target sizes. #### Standards reference - Map improvements to WCAG 2.2 AA: 1.4.x (contrast, text size), 2.1.x (keyboard), 2.4.x (navigation), 1.3.x (structure). ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global framing (farmers, students worldwide), but from a Western policy/infra perspective. - Improve: Include more regional examples and non‑Western platform contexts. #### Inclusive language - Present: Neutral, technical, and accessible language. - Improve: Define policy/standards acronyms and avoid assuming familiarity with institutions. #### Neurodiversity support - Present: Logical headings and chunked sections. - Improve: Add summaries, bullet lists, and consistent visual cues. #### Representation - Present: Abstract discussion; no imagery or personal profiles. - Improve: Use examples featuring varied creators and regions. #### Psychological safety - Present: Non‑confrontational tone; no triggering content. - Improve: Provide code of conduct and moderation norms where discussion occurs. ### ✅ Logical Validity #### Strong reasoning - Inductive reasoning from observed changes (bot blocking, paywalls) to claim that the Open Web is shrinking due to incentive shifts. - Causal analysis linking AI scraping and generative answers to reduced traffic and value for publishers. - Balanced argument that both extreme control and extreme openness are harmful, guiding toward a middle ground. #### Fallacies - No major explicit fallacies; some claims about the prevalence and effects of practices (e.g., bot blocking) are asserted without quantitative data but are plausible. #### Tone of voice - Reflective, analytic, and measured. - Acknowledges multiple motivations and stakeholder views without demonizing any side. ### 🛡️ DARVO Analysis - Deny: The article does not deny harms or responsibility; it explicitly recognizes publisher and platform incentives. - Attack: No attacks on individuals or groups; critique is systemic (platform incentives, AI business models). - Reverse victim/offender: The piece avoids casting any side purely as victim or villain; it frames everyone as operating under changing incentives and seeks a balanced path. ## 🎯 Conclusion The essay argues that the Open Web is a fragile public good now under pressure from AI, changing business models, and rising costs. Because openness must be voluntary, the way forward is to redesign technical, economic, and social incentives so that creators still *want* to publish openly, even as technology and platforms evolve. For readers and builders, the next steps are to clarify your own openness goals, update crawler and licensing policies, and explore sustainable ways to keep valuable content available. --- ```yaml tags: [open-web, internet-governance, artificial-intelligence] author: Mark Nottingham source: mnot.net reference: [https://www.mnot.net/blog/2026/01/20/open_web] date created: 2026-01-20 14:30 --- # 🏷️ Title: Some Thoughts on the Open Web ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - Mark Nottingham, “Some Thoughts on the Open Web” – https://www.mnot.net/blog/2026/01/20/open_web ## 📝 Summary - What happened. - The author reflects on how the “Open Web” emerged, why it’s valuable, and how AI and shifting incentives are shrinking it. - Why it matters. - Open access to information is a public good; its erosion affects education, innovation, and democratic discourse. - Key stakeholders. - Publishers, creators, users, browser/search/AI vendors, standards bodies, and regulators. - Risks and tradeoffs. - Over‑extraction by platforms and AI vs. sustainability for publishers; openness vs. control and privacy; blocking bots vs. information access. - Bottom line. - We must redesign incentives so people *want* to publish openly, recognizing many motives and forms of “open,” instead of trying to coerce openness. ## 📖 Main Content - Core arguments: - Historically, access to and publication of information were hard and gatekept; the Web massively lowered these barriers and normalized easy, low‑friction access to content. - The Open Web is a “public good” that benefits both consumers and producers by reducing costs and friction in both directions. - People open content for diverse reasons: contributing to a commons, building reputation, social connection, ad revenue, or as a funnel into paid offerings. Policies must respect this diversity rather than enforce “purity tests.” - There are many kinds of “open”: free access, registration walls, ad‑supported, limited free views, etc. Small frictions (mandatory logins everywhere) cumulatively reduce openness but can also address privacy and sustainability. - Participation in the Open Web must be voluntary; content owners respond to incentives and can’t realistically be forced to open content. Attempts at coercion (e.g., withholding features) push them toward walled gardens and paywalls. - AI radically shifts incentives: it scrapes open content for training and then competes with that content, undermining models based on reputation, traffic, and ad/subscription funnels. - As a result, many are blocking bots, tightening paywalls, and withdrawing from the Open Web, shrinking the shared commons. - Serving content isn’t free; creators must sustain themselves, and infrastructure costs (even if cheap per request) accumulate at scale. Selective blocking of heavy or objectionable crawlers is now widespread and arguably necessary. - The real tension is about control: between maximal control by publishers and maximal freedom for reusers. Neither extreme is desirable; a sustainable balance is needed. - The path forward is to strengthen incentives to keep content open, revisiting assumptions about what “open” means and what incentive structures work under new technological shocks like AI. - Key terms: - Open Web: Web content that is easily accessible (often free and not heavily restricted), indexable, and reusable to some extent. - Public good: Something broadly available and beneficial where one person’s use doesn’t diminish another’s (non‑rivalrous). - Incentive structure: The mix of economic, social, and technical conditions that make it attractive or unattractive to publish openly. - Uncertainties / contested areas: - How equitable ad‑supported vs. pay‑per‑view models really are across income levels and regions. - Where to draw the line between legitimate crawler blocking and de‑facto enclosure. - What legal and technical norms for AI training and reuse will emerge or be enforced. ### 🗒️ Notes - The Web transformed access to knowledge by slashing friction in both publishing and discovery, disrupting traditional gatekeepers while creating new intermediaries (search engines, marketplaces). - A social norm emerged that Web content is easy to access without paywalls or complex logins; this norm isn’t universal but has had strong network effects. - The Open Web benefits: - Farmers getting up‑to‑date agricultural guidance. - Students globally accessing materials once limited to elite institutions. - Creators who gain reputation and contribute to a global commons. - Motivations for open publishing: - Altruism / contribution to commons. - Reputation and discovery. - Social connection and community. - Advertising revenue. - Freemium tactics leading into closed, paid experiences. - “Open” is a spectrum: - Completely open, no tracking. - Ad‑supported with tracking. - Search‑indexable but individual users must log in. - Metered/paywalled models with a few free views. - Advertising models can be more equitable than micro‑payments by lowering financial barriers to access, but they introduce privacy concerns and dependence on surveillance infrastructure. - Cumulative friction (e.g., ubiquitous logins and meters) gradually erodes the Open Web norm and can lock users into a patchwork of silos. - Voluntariness: - Publishers choose openness when benefits (reach, reputation, revenue) outweigh downsides (costs, misappropriation, scraping). - Coercive strategies—denying capabilities, punitive policies—risk pushing content into closed ecosystems rather than encouraging openness. - AI as a disruptive shock: - Trains on publicly accessible content without direct compensation or consent in most cases. - Generates answers that substitute for visiting the source, displacing traffic and monetization. - Converts “commons” contributions into private corporate advantage, eroding trust. - Responses: - Bot blocking, robots.txt hardening, and paywalls. - Withdrawal from truly open formats (like unmetered RSS) in some sectors. - Persistent but unevenly distributed commitment to openness among some creators. - Cost realities: - Information is non‑rivalrous, but production and delivery have real human and infrastructure costs. - High crawler demand, especially from AI companies, can create unsustainable bandwidth and operational loads. - Selective blocking of abusive or extractive clients has become common; removing that capability would likely harm the overall supply of open content. - Control tension: - One extreme: publishers want pixel‑perfect control over presentation, use cases, and audiences. - Other extreme: open‑access advocates want fully free reuse for any purpose. - Sustainable openness requires a middle path that leaves publishers enough control and benefit to participate willingly. - Way forward: - Focus on incentive design, not moralizing or coercion. - Accept multiple, pragmatic definitions of “open” that can evolve with the ecosystem. - Plan for resilience to future shocks beyond AI. ## 📚 References - “Some Thoughts on the Open Web” – https://www.mnot.net/blog/2026/01/20/open_web - Tim Berners-Lee, “Information Management: A Proposal” (original Web proposal) - W3C – https://www.w3.org/ - IETF – https://www.ietf.org/ ## 🗣️ Paraphrase - Paragraph A paraphrase. - The author explains that “Open Web” means different things, but current debates mainly center on how easily people can publish and access information without barriers. - Paragraph B paraphrase. - He argues that while the Web created an extraordinary public good by lowering friction, new forces—especially AI and changing business models—are undermining the incentives that previously kept content open, so we must rethink how to keep openness attractive and sustainable. ## 📰 Newsletter In this week’s WordPress‑adjacent reading, Mark Nottingham’s “Some Thoughts on the Open Web” is a must‑read for anyone building on or around WordPress. He traces how the Web’s low‑friction access to information became a public good—and why AI, paywalls, and heavy scraping are pressuring publishers to lock content down. For WordPress creators, this raises hard questions: how do we keep sites indexable and shareable while maintaining viable business models and protecting from abusive crawlers and model training? Think about: - Smarter caching/CDN setups to keep costs low. - Bot‑management rules that distinguish legitimate search from extractive scraping. - Clear content licensing and terms pages on your WordPress site. - Exploring open‑friendly monetization like memberships, ethical ad networks, or donation models. Recommended follow‑up resources for WordPress folks: - WordPress.org documentation on privacy and consent tools. - Guides on implementing robots.txt, XML sitemaps, and caching plugins responsibly. - Community talks on publishing sustainability in the age of AI. Open publishing is at the heart of WordPress’s mission; this essay is a timely reminder that “open” only survives if it also works for the people doing the publishing. ## ✅ Action Items #### DevRel - Clarify and document your project’s stance on Open Web participation, AI training, and crawler policies. - Prepare talks/blogs explaining how developers can be open‑friendly while protecting their content. - Produce example robots.txt and HTTP header configurations balancing openness with rate‑limiting and bot control. - Create tutorials on sustainable monetization for open‑access documentation and blogs. - Gather feedback from developer communities about AI scraping and share with product/standards teams. #### OSPO - Draft policies for when corporate content should be open vs. access‑controlled, including treatment of AI crawlers. - Clarify license guidance around data scraping, model training, and downstream use. - Integrate crawler and AI‑usage considerations into security and compliance reviews. - Maintain an inventory of open vs. closed internal web properties and their policies. - Engage upstream standards bodies (IETF/W3C) on mechanisms to express AI‑training preferences. #### Community - Host discussions on sustainable open publishing and AI impacts with maintainers and users. - Provide maintainers with simple templates for content usage policies and bot rules. - Encourage community‑run archives and mirrors for at‑risk open resources (where licenses allow). - Share case studies of projects that remain open while staying financially viable. #### WordPress - Core: ensure tools like robots.txt, XML sitemaps, and privacy settings are first‑class and easy to configure. - Core Performance: keep open sites cheap to serve through performance improvements and caching guidance. - Core AI: define guidance on how AI features in the ecosystem respect site owners’ content and choices. - Core Program Team: coordinate cross‑team efforts on openness, privacy, and AI policies. - Mobile: surface key privacy/open‑web controls in mobile publishing workflows. - Playground: provide labs demonstrating best practices for open yet protected WordPress setups. - Community: include Open Web resilience as a theme in meetups and WordCamps. - Training: add modules on Open Web economics, AI, and sustainable content strategies. - Marketing: highlight success stories of sites thriving while staying indexable and open. - Support: create help articles on bot blocking, caching, and monetization for open content. - Design: ensure consent and paywall UX is clear, honest, and minimally disruptive. - Documentation: document recommended open/closed patterns, including robots and caching examples. - Photos / TV / Polyglots / Accessibility / Meta / CLI / Hosting / Tide / Openverse / Themes / Plugins: - Align content policies and tools with a shared view of openness, crawler handling, and attribution. #### The WPCC - Programs: sponsor maintainers who commit to keeping high‑value educational content openly available. - Projects: run governance clinics on content licensing and AI policy for WordPress ecosystems. - Funding experiments: pilot grants that reward open educational resources and transparent bot policies. ### FAIR - Technical independence: encourage local archiving and provenance metadata for at‑risk open resources. - Policies: recommend transparent funding and sunset clauses for initiatives that rely on content remaining open. - Governance artifacts: provide reusable charters and agreements covering AI training rights and crawler behavior. ## 🧪 Hypotheses - H1: Publishers facing high AI scraping without compensation are more likely to move content behind paywalls or block crawlers. - H2: Clear, widely adopted technical signals about AI training preferences will slow the retreat from the Open Web. - H3: Ad‑supported open content provides more equitable access across income levels than micro‑payment models, at equal quality levels. - H4: Improving performance and hosting efficiency decreases the probability that small publishers will restrict or close their content. - H5: Communities with strong open norms and governance (e.g., WordPress ecosystem) will maintain higher levels of open content despite AI shocks. ### 📊 Methodology - Use longitudinal web crawls to track shifts in paywall prevalence and robots policies in response to AI‑related events. - Survey publishers about reasons for changing openness policies, including AI, costs, and platform behavior. - Run controlled experiments comparing user access patterns under ad‑supported vs. micro‑paywall models in diverse regions. - Measure correlation between performance optimizations (CDN use, caching) and openness (indexability, paywalls) in small sites. - Compare ecosystems (e.g., WordPress vs. proprietary CMS) on levels of open publishing and attitudes toward AI scraping. ### ❓ Questions - What technical and legal tools can express “open for humans, restricted for AI training” in a meaningful, enforceable way? - How can small publishers negotiate with large AI vendors over training and attribution, if at all? - Are there sustainable funding models for open content beyond ads and subscriptions? - What level of friction (logins, meters) is compatible with still calling something part of the “Open Web”? - How can standards bodies incorporate these incentive questions without overstepping into policy‑only realms? ### 🔮 Future Research - Comparative studies of regions where paywalls dominate vs. where open access norms remain strong. - Economic modeling of AI’s impact on traffic and revenue for different classes of publishers. - Exploration of new protocols/metadata specifically for AI training consent and compensation. - Social research into user trust and willingness to support open content directly (donations, memberships). ### 💡 Implications - Practice: Site owners must revisit crawler policies, monetization, and licensing to remain open without being exploited. - Policy: Regulators may need to address AI training on open content, balancing innovation and creator rights. - Security: High‑volume scraping can undermine availability and raise infrastructure costs; rate‑limiting and bot controls become more critical. - Education: Institutions relying on open web materials may face shrinking resources and should consider archiving and funding strategies. - Funding: Philanthropy and public funding might play larger roles in preserving open, high‑value content. ### 🧠 Learning Styles Assessment - Reading level (rough estimate): Upper high‑school / early college (Flesch‑Kincaid ~11–13) due to policy/technical vocabulary. #### 👀 Visual learners - Present: Conceptual contrasts (open vs. closed, spectrum of openness) that could map to diagrams. - Improve: Add flowcharts showing incentive changes; diagrams of stakeholders and content flows. #### 🔊 Aural learners - Present: None inherent. - Improve: Short audio summary or podcast discussion of the piece. #### 🗣️ Verbal learners - Present: Clear definitions, analogies (e.g., public good), and structured argument sections. - Improve: More concrete examples and short case narratives. #### 🧩 Kinesthetic learners - Present: None. - Improve: Hands‑on exercises like configuring robots.txt, setting up open‑friendly caching, or experimenting with paywall plugins. #### 🧠 Logical learners - Present: Causal reasoning about incentives; explicit tradeoffs; discussion of costs and benefits. - Improve: Tables comparing business models; decision trees for openness choices. #### 🧑‍🤝‍🧑 Social learners - Present: Implicit invitations to community debate. - Improve: Guided discussion prompts for meetups, working groups, and classrooms. #### 🧘‍♀️ Solitary learners - Present: Reflective essay style suitable for individual reading and note‑taking. - Improve: Reflection questions at the end (e.g., “Why am I open?” “What would make me close my content?”). #### 🌿 Naturalistic learners - Present: Limited direct links. - Improve: Analogies to environmental commons and resource management. ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Typical blog structure with headings and paragraphs should be reasonably readable. - Improve: Confirm landmark roles, skip links, and descriptive image alt text. #### Keyboard navigation - Present: Likely basic keyboard support via standard HTML. - Improve: Ensure visible focus indicators and skip‑to‑content links. #### Contrast and readability - Present: Depends on site theme; mnot.net is generally minimalist. - Improve: Verify contrast ratios, font sizes, and line spacing for long‑form reading. #### Text resizing and reflow - Present: Responsive layout likely handles moderate zoom. - Improve: Test at 200%+ zoom and with reduced‑motion preferences. #### Media and captions - Present: Primarily text; minimal media. - Improve: Provide audio version or downloadable text for offline use. #### Forms and interactions - Present: Minimal, likely just navigation and maybe comments. - Improve: Ensure any forms have labels, clear errors, and adequate target sizes. #### Standards reference - Map improvements to WCAG 2.2 AA: 1.4.x (contrast, text size), 2.1.x (keyboard), 2.4.x (navigation), 1.3.x (structure). ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global framing (farmers, students worldwide), but from a Western policy/infra perspective. - Improve: Include more regional examples and non‑Western platform contexts. #### Inclusive language - Present: Neutral, technical, and accessible language. - Improve: Define policy/standards acronyms and avoid assuming familiarity with institutions. #### Neurodiversity support - Present: Logical headings and chunked sections. - Improve: Add summaries, bullet lists, and consistent visual cues. #### Representation - Present: Abstract discussion; no imagery or personal profiles. - Improve: Use examples featuring varied creators and regions. #### Psychological safety - Present: Non‑confrontational tone; no triggering content. - Improve: Provide code of conduct and moderation norms where discussion occurs. ### ✅ Logical Validity #### Strong reasoning - Inductive reasoning from observed changes (bot blocking, paywalls) to claim that the Open Web is shrinking due to incentive shifts. - Causal analysis linking AI scraping and generative answers to reduced traffic and value for publishers. - Balanced argument that both extreme control and extreme openness are harmful, guiding toward a middle ground. #### Fallacies - No major explicit fallacies; some claims about the prevalence and effects of practices (e.g., bot blocking) are asserted without quantitative data but are plausible. #### Tone of voice - Reflective, analytic, and measured. - Acknowledges multiple motivations and stakeholder views without demonizing any side. ### 🛡️ DARVO Analysis - Deny: The article does not deny harms or responsibility; it explicitly recognizes publisher and platform incentives. - Attack: No attacks on individuals or groups; critique is systemic (platform incentives, AI business models). - Reverse victim/offender: The piece avoids casting any side purely as victim or villain; it frames everyone as operating under changing incentives and seeks a balanced path. ## 🎯 Conclusion The essay argues that the Open Web is a fragile public good now under pressure from AI, changing business models, and rising costs. Because openness must be voluntary, the way forward is to redesign technical, economic, and social incentives so that creators still *want* to publish openly, even as technology and platforms evolve. For readers and builders, the next steps are to clarify your own openness goals, update crawler and licensing policies, and explore sustainable ways to keep valuable content available.