Markdown, llms.txt and AI Crawlers  🔗 **Source:** [Markdown, llms.txt and AI Crawlers](https://dri.es/markdown-llms-txt-and-ai-crawlers?__readwiseLocation=) > [!note]+ Document Notes > tags: #markdown #ai-crawlers #seo #web-development #content-optimization > author: Dries Buytaert > source: dri.es > reference: https://dri.es/markdown-llms-txt-and-ai-crawlers > date created: 2024-12-19 14:30 > > # 🏷️ Title: Markdown, llms.txt and AI crawlers > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Analyze AI crawler ROI — 1,241 pages crawled per citation received #ai-crawlers - [H2] Evaluate Markdown implementation effort vs benefits — no clear benefit demonstrated #markdown - [H3] Focus resources on content quality over technical optimization — clear writing still works best #content-strategy ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Markdown, llms.txt and AI crawlers](https://dri.es/markdown-llms-txt-and-ai-crawlers) - [llms.txt specification](https://llmstxt.org/) ## 📝 Summary - Dries Buytaert analyzed one month of AI crawler data after implementing Markdown support on his site - AI crawlers fetch 1,241 pages for every citation they provide back to creators - a poor ROI - Adding Markdown increased crawler traffic by 7% rather than reducing it - The llms.txt standard receives virtually no traffic from actual AI systems (only SEO tools) - Current recommendation: focus on content quality over technical optimization for AI crawlers ## 📖 Main Content Dries Buytaert conducted a real-world experiment by adding Markdown support to his personal website in January 2024 and analyzing the subsequent AI crawler behavior through Cloudflare logs. The results challenge common assumptions about optimizing content for AI systems. The core finding reveals an unfavorable exchange rate between AI crawler consumption and traffic returned to creators. For every citation or link back that AI answer engines provide, their crawlers had fetched 1,241 pages from Buytaert's site. This represents a massive imbalance where content creators provide substantial value to AI systems while receiving minimal traffic in return. Contrary to expectations that smaller Markdown files would reduce bot traffic, the implementation actually increased crawler activity by approximately 7%. Bots fetch both HTML and Markdown versions when available, treating them as complementary rather than alternative resources. The analysis reveals different crawler behaviors across AI companies. OpenAI operates three distinct bots: OAI-SearchBot for indexing search content, GPTBot for training data collection, and ChatGPT-User for real-time page fetching during live conversations. Amazon's bot showed the highest percentage of Markdown requests at 10.9%, while others like ByteDance's Bytespider completely ignored Markdown versions. The llms.txt standard, designed to help sites describe their content for AI systems, proved ineffective in practice. Across Buytaert's site and Acquia's entire hosting fleet (400 million requests), llms.txt requests came almost exclusively from SEO audit tools rather than the AI systems it was designed to serve. ## 🗒️ Notes #### 📊 AI Crawler Traffic Analysis - Key idea: AI crawlers provide poor ROI with 1,241 pages fetched per citation returned - Evidence: "For every citation they sent, their crawlers had fetched 1,241 pages" #### 📈 Markdown Impact on Bot Traffic - Key idea: Adding Markdown increases rather than decreases total crawler traffic - Evidence: "My crawler traffic increased by about 7%. Offering a lighter format does not replace the heavier one" #### 🤖 OpenAI's Multi-Bot Strategy - Key idea: OpenAI operates specialized bots for different functions - Evidence: "OAI-SearchBot indexes content for search, GPTBot crawls for training data, and ChatGPT-User fetches pages in real time" #### 🔍 Content Negotiation Failure - Key idea: No AI crawler uses proper HTTP content negotiation for Markdown - Evidence: "No AI crawler uses content negotiation. Not one" #### 📋 llms.txt Standard Ineffectiveness - Key idea: The llms.txt standard is ignored by actual AI systems - Evidence: "Every one came from SEO audit tools. Not a single request came from an AI answer engine or crawler" #### 🎯 Content Quality Over Technical Optimization - Key idea: Traditional content excellence remains more valuable than AI-specific technical features - Evidence: "What still works is what has always worked: clear writing, authoritative content, and timely publishing" #### 📊 Crawler Distribution Analysis - Key idea: Amazon and OpenAI dominate AI crawler traffic with different Markdown adoption rates - Evidence: "AmazonBot 16,872 total requests, ChatGPT-User 13,864 total requests" #### 🔗 Auto-Discovery Link Usage - Key idea: AI crawlers discover Markdown only through dedicated URLs, not HTTP headers - Evidence: "They only discover the Markdown pages through the dedicated URLs, and only via the auto-discovery link" ## 📚 References - [Original article](https://dri.es/markdown-llms-txt-and-ai-crawlers) - [llms.txt specification](https://llmstxt.org/) - [Acquia hosting platform data](https://acquia.com/) ## 🗣️ Paraphrase - AI systems consume far more content than they return in traffic, creating an imbalanced value exchange for content creators - Technical optimizations like Markdown support and llms.txt files provide minimal practical benefits for AI discoverability - Content creators should prioritize writing quality and authoritative information over AI-specific technical implementations ## 📰 Newsletter **AI Crawler Reality Check: Data Beats Hype for Content Strategy** Dries Buytaert's thorough analysis of AI crawler behavior offers WordPress creators valuable data-driven insights. His findings reveal that AI systems fetch over 1,200 pages for every citation they provide back - highlighting the current imbalance between content consumption and traffic generation. **Key takeaway for WordPress sites**: Focus your energy on content excellence rather than AI-specific technical features. The llms.txt standard and Markdown optimization show minimal real-world impact, while quality writing and timely publishing continue to drive results. **Practical next steps**: Audit your content strategy priorities and ensure you're investing time in substance over speculative SEO tactics. WordPress's built-in semantic markup and clean HTML structure already provide strong foundations for AI systems. Resources: [WordPress SEO Guide](https://wordpress.org/documentation/article/search-engine-optimization/), [Content Strategy](https://learn.wordpress.org/), [Performance Optimization](https://make.wordpress.org/core/handbook/best-practices/performance/) ## ✅ Action Items #### WordPress - Core: Evaluate need for native Markdown support based on crawler data analysis - Performance: Document that lighter formats don't reduce total bot traffic - SEO: Update guidance to prioritize content quality over AI-specific optimizations - Documentation: Create realistic expectations guide for AI crawler optimization #### Community - Maintainers: Share data-driven insights on AI optimization ROI with plugin developers - Contributors: Focus development resources on proven value additions over speculative features - Content Creators: Emphasize writing quality and authoritative content in community guidelines ## 🧪 Hypotheses - AI crawler traffic will continue increasing regardless of content format optimizations - The citation-to-crawl ratio will remain unfavorable to content creators in the short term - Traditional SEO factors will retain more value than AI-specific technical implementations - Content negotiation adoption by AI crawlers will remain minimal without industry pressure ### 📊 Methodology - Longitudinal analysis comparing pre/post Markdown implementation crawler patterns - Cross-platform validation using Acquia's hosting fleet data (400M+ requests) - Categorical analysis of crawler behavior across different AI vendors and bot types ### ❓ Questions - Will AI companies adjust their crawler-to-citation ratios as content creator pushback increases? - How might WordPress core balance AI optimization features against development resource allocation? - What crawler behavior changes might emerge as AI answer engines face sustainability pressure? ### 🔮 Future Research - Quarterly tracking of AI crawler ROI metrics across different content management systems - Analysis of crawler respect for robots.txt and rate limiting across AI vendors - Investigation of premium content models that could rebalance the creator-AI value exchange ### 💡 Implications - Practice: Content teams should deprioritize AI-specific technical optimizations in favor of quality improvements - Policy: Industry discussions needed around sustainable crawler practices and content creator compensation - Education: WordPress community education should emphasize proven strategies over speculative AI optimization - Funding: Development resources better allocated to core functionality than experimental AI features ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical content with data analysis) #### 👀 Visual learners - Present: Comprehensive data tables showing crawler statistics and percentages - Improve: Add visual timeline of crawler traffic changes, crawler behavior flowcharts #### 🔊 Aural learners - Present: Clear narrative structure walking through experimental findings - Improve: Podcast discussion or audio summary of key insights #### 🗣️ Verbal learners - Present: Technical terms explained in context, clear definitions of bot types - Improve: Add glossary of crawler terminology, analogies for crawler behavior patterns #### 🧩 Kinesthetic learners - Present: Step-by-step methodology description, actionable recommendations - Improve: Add implementation checklist, hands-on crawler analysis tutorial #### 🧠 Logical learners - Present: Data tables, statistical analysis, cause-and-effect relationships - Improve: Add decision tree for AI optimization priorities, ROI calculation framework #### 🧑‍🤝‍🧑 Social learners - Present: LinkedIn discussion reference, community implications - Improve: Add discussion prompts for team meetings, peer comparison frameworks #### 🧘‍♀️ Solitary learners - Present: Self-contained analysis with clear conclusions - Improve: Add reflection questions for individual content strategy assessment #### 🌿 Naturalistic learners - Present: Ecosystem perspective on AI crawler behavior patterns - Improve: Add sustainability considerations for content-AI relationships ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Well-structured headings, data tables with proper markup - Improve: Add table captions explaining crawler data context #### Keyboard navigation - Present: Clean semantic HTML structure - Improve: Ensure data tables are fully keyboard navigable #### Contrast and readability - Present: Clean text presentation with good contrast - Improve: Verify data table readability at high zoom levels #### Text resizing and reflow - Present: Responsive design supporting text scaling - Improve: Test data table reflow at 200% zoom #### Media and captions - Present: Text-based content with clear data presentation - Improve: Consider audio description for data insights #### Forms and interactions - Present: Minimal interactive elements - Improve: N/A for this content type ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global perspective on AI crawler behavior - Improve: Include examples from non-English sites, diverse content creators #### Inclusive language - Present: Technical content with clear explanations - Improve: Add plain-language summary for non-technical readers #### Neurodiversity support - Present: Well-chunked content with clear section headers - Improve: Add executive summary for quick comprehension #### Representation - Present: Focus on technical analysis - Improve: Include diverse content creator perspectives on AI impact #### Psychological safety - Present: Factual, non-judgmental analysis - Improve: Acknowledge varying comfort levels with AI technology ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Sound statistical analysis from comprehensive data set - Inductive strength: Valid generalization from specific crawler behavior patterns - Causal reasoning: Clear cause-effect relationships between Markdown implementation and traffic changes - Analogical strength: Appropriate comparison between different AI vendor behaviors #### Fallacies - Sample bias: Potential limitation using single site data, though mitigated by Acquia fleet validation - Hasty generalization: Conclusions appropriately scoped to current timeframe and conditions #### Tone of voice - Overall tone: Data-driven and objective with practical recommendations based on evidence ### 🛡️ DARVO Analysis - Deny: No denial patterns present - straightforward technical analysis - Attack: No attack patterns - collaborative tone toward industry - Reverse victim/offender: No victim/offender dynamics in technical content - Emotional cues: Factual presentation with appropriate concern for creator economics ## 🎯 Conclusion Buytaert's analysis provides crucial reality-checking for content creators considering AI optimization strategies. The data clearly shows that current AI crawler behavior heavily favors AI systems over content creators, with minimal return on the substantial resources consumed. For WordPress site owners and developers, the practical implication is clear: invest in proven content quality strategies rather than speculative AI-specific technical implementations. The fundamentals of clear writing, authoritative content, and consistent publishing continue to provide the best return on effort. This research also highlights broader questions about sustainable relationships between AI systems and content creators that the industry will need to address as AI adoption continues growing. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad ### [H1] AI crawlers provide poor ROI for content creators > [!quote] [Markdown, llms.txt and AI crawlers](https://dri.es/markdown-llms-txt-and-ai-crawlers) by [[Dries Buytaert]] via [[dri.es]]. Published 2024-03-05. Accessed 2024-12-19. > "For every citation they sent, their crawlers had fetched 1,241 pages." Quantifies the imbalanced value exchange between AI systems and content creators through empirical data analysis. [view highlight](https://dri.es/markdown-llms-txt-and-ai-crawlers#H1) #xref #articleseed ID:rw_ai_crawler_roi ### [H2] llms.txt standard fails in practice > [!quote] [Markdown, llms.txt and AI crawlers](https://dri.es/markdown-llms-txt-and-ai-crawlers) > "Not a single request came from an AI answer engine or crawler." Demonstrates gap between proposed standards and actual AI system implementation through traffic analysis. [view highlight](https://dri.es/markdown-llms-txt-and-ai-crawlers#H2) #xref ID:rw_llms_txt_failure --- ### 🔖 Atomic Split Pad (Zettel Candidates) #### [Z1] Content optimization should prioritize proven strategies - Applies to current AI optimization landscape with uncertain ROI - Based on crawler traffic analysis and citation data - Links: [[Content Strategy]] [[SEO Best Practices]] #### [Z2] AI systems consume more content than they return in traffic - Universal pattern across different AI vendors and crawler types - Quantified through comprehensive traffic analysis - Links: [[Content Economics]] [[AI Industry Impact]] --- ### ✏️ Fleeting Queue - [F1] Investigate WordPress core Markdown support implications — based on crawler data #fleeting #inbox - [F2] Review community AI optimization guidance accuracy — update with empirical findings #fleeting #inbox - [F3] Track AI crawler behavior changes over time — establish baseline metrics #fleeting #inbox <!-- deep-read:sections end --> 📄 **Full text:** ![[Full Document Contents/Articles/Markdown, llms.txt and AI Crawlers|Markdown, llms.txt and AI Crawlers, dri.es]] ## Highlights - People also asked whether serving Markdown reduced my bot traffic since the files are smaller. It does not. Bots fetch both versions, and my crawler traffic increased by about 7%. Offering a lighter format does not replace the heavier one. It simply gives bots more to crawl. ([View Highlight](https://read.readwise.io/read/01kk27m4xkj4325w9ax5k4adcn)) - No AI crawler uses content negotiation. Not one. They only discover the Markdown pages through the dedicated URLs, and only via the autodiscovery link. ([View Highlight](https://read.readwise.io/read/01kk27mznqkv8hcnyfq3a570am)) - So should you add Markdown support to your site? Probably not. There is no clear benefit today. It does not reduce crawl traffic, and I can't demonstrate that it improves how AI systems use your content. ([View Highlight](https://read.readwise.io/read/01kk27kgbdnc1jaje4pmgcsj7n)) - Tags: 👻 ai highlighted