Nobody loves the native search 🔗 **Source:** [Nobody loves the native search](https://www.youtube.com/watch?v=UQ-X3wPoxuQ) > [!note]+ Document Notes > tags: [wordpress-core, search, mysql, performance] > author: Thörn Lüsteal > source: YouTube > reference: https://www.youtube.com/watch?v=UQ-X3wPoxuQ > date created: 2024-12-28 15:30 > > # 🏷️ Title: Nobody loves the native search > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Report search markup pollution issue to WordPress core — 00:00 #wordpress-core - [H2] Test REGEX_REPLACE solution on MySQL 8.0.4+ — 10:45 #development - [H3] Advocate for basic search settings in core — 13:20 #feature-request ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [WordPress Search Issues](https://www.youtube.com/watch?v=UQ-X3wPoxuQ) - [MySQL REGEX_REPLACE Documentation](https://dev.mysql.com/doc/refman/8.0/en/regexp.html) ## 📝 Summary - WordPress native search has significant limitations including inability to filter by author, date, categories, or custom post types without coding - Block editor creates HTML comments that pollute search results, making terms like "paragraph," "image," and "background" essentially useless - Core developers acknowledge this as a known issue but suggest expensive solutions like Elasticsearch - A potential middle-ground solution using MySQL's REGEX_REPLACE function could remove HTML markup before searching - The speaker advocates for basic search settings in WordPress core rather than requiring custom code ## 📖 Main Content WordPress's native search functionality has fundamental flaws that affect user experience across millions of websites. The search only covers title, excerpt, and content using simple full-text SQL queries, with no built-in options for filtering by author, date, tags, categories, or custom post types. A critical issue emerged with the block editor: HTML comments containing block names and settings are stored in post content and included in searches. This means searching for "syntax highlighter" returns every post containing code blocks, even when the term doesn't appear in the visible content. Similar pollution occurs with common terms like "paragraph," "image," and "background." Core developers Gary Pendergast and Daniel Bachhuber acknowledge this as a known limitation, suggesting solutions like Elasticsearch or custom indexing. However, these approaches are impractical for typical WordPress users and clients. The speaker proposes a middle-ground solution using MySQL 8.0.4+'s REGEX_REPLACE function to strip HTML markup before searching. While this has performance implications, it could work acceptably for smaller sites without requiring dedicated search infrastructure. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=0) — [H1] WordPress Search Limitations - Key idea: Native WordPress search lacks basic filtering capabilities - Evidence: "You can't filter for an author date tags or categories you can't exclude a page or a post" #### ⏱️ [01:30](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=90) — [H2] How WordPress Search Works - Key idea: Search uses simple full-text SQL query on title, excerpt, and content - Evidence: "It searches the title The excerpt and the content with a very very simple full text SQL query" #### ⏱️ [02:45](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=165) — [H2] Block Editor Search Pollution Problem - Key idea: Block editor HTML comments contaminate search results - Evidence: "I searched for syntax highlighter and I was wondering I got so many results and they had the even they don't have the search term in them" #### ⏱️ [04:20](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=260) — [H3] Root Cause of False Positives - Key idea: Block names and settings stored as HTML comments are searchable - Evidence: "The blog editor Saves The Blog name and some settings in the HTML comment in the Raw content" #### ⏱️ [05:15](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=315) — [H2] Search Term Pollution Impact - Key idea: Common words become useless for search due to markup pollution - Evidence: "You can't really search for paragraph or image or background you will get every every blog post or nearly every blog post" #### ⏱️ [06:30](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=390) — [H2] Core Developer Response - Key idea: Leading developers acknowledge this as a known MySQL limitation - Evidence: "MySQL string searching isn't capable of dealing with this kind of contextual passing" - Gary Pendergast #### ⏱️ [07:45](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=465) — [H3] Suggested Solutions from Core - Key idea: Core developers suggest Elasticsearch or custom indexing - Evidence: "If you require 100% accurate search results the best option is to use a dedicated search engine like elastic search" #### ⏱️ [08:30](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=510) — [H2] Elasticsearch Impracticality - Key idea: Elasticsearch setup is unrealistic for typical WordPress users - Evidence: "Setting up elastic search is not what the typical user or client I have would do" #### ⏱️ [09:15](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=555) — [H1] REGEX_REPLACE Solution - Key idea: MySQL 8.0.4+ offers REGEX_REPLACE function for markup removal - Evidence: "With the MySQL 8.0.4 and MariaDB 10.0.7 onwards there is a new function called regex replace" #### ⏱️ [10:45](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=645) — [H3] Performance Trade-offs - Key idea: REGEX solution has performance impact but could work for smaller sites - Evidence: "This will of course have some import performance impact but for smaller sites this could be acceptable" #### ⏱️ [11:30](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=690) — [H3] Implementation Challenges - Key idea: Various edge cases need consideration for markup removal - Evidence: "File names and alternative text... short codes could be also removed... some page Builders are using Json" #### ⏱️ [12:15](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=735) — [H2] Focus Phases Limitation - Key idea: WordPress focus phases left search functionality behind - Evidence: "We have those focus on the four phases... but now after some years I think we need to look at those things that we have left behind" #### ⏱️ [13:20](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=800) — [H1] Core Search Settings Request - Key idea: WordPress needs basic search configuration options in core - Evidence: "Important features like SEO sitemaps or the search would be much better with at least some basic settings" ## 📚 References - [WordPress Core Development](https://make.wordpress.org/core/) - [MySQL REGEX_REPLACE Function](https://dev.mysql.com/doc/refman/8.0/en/regexp.html) - [Elasticsearch Documentation](https://www.elastic.co/guide/) ## 🗣️ Paraphrase - WordPress's default search is too basic, lacking filters for authors, dates, categories, and custom post types that users expect - The block editor inadvertently polluted search results by storing block metadata as HTML comments in post content, making common words unsearchable - While core developers recognize the problem, their suggested solutions like Elasticsearch are too complex for typical WordPress sites ## 📰 Newsletter **WordPress Search Needs an Overhaul** WordPress's native search functionality has fallen behind user expectations, lacking basic filtering options and suffering from block editor pollution that makes common terms like "paragraph" and "image" return irrelevant results. Core developers acknowledge the issue stems from MySQL's limitations with contextual parsing, but their suggested solutions like Elasticsearch aren't practical for most sites. A proposed middle-ground using MySQL's REGEX_REPLACE function could strip HTML markup before searching, though with performance trade-offs. **Key Takeaway**: WordPress core needs basic search configuration options rather than forcing users to code solutions or implement complex search infrastructure. **Next Steps**: Consider advocating for search improvements in WordPress core development discussions, or explore existing search plugins that address these limitations. Resources: [WordPress Core](https://make.wordpress.org/core/), [Search Plugins](https://wordpress.org/plugins/tags/search/), [Performance Best Practices](https://make.wordpress.org/core/handbook/references/php-compatibility-and-wordpress-versions/) ## ✅ Action Items #### WordPress - Core: Add basic search filtering options to WordPress admin - Core: Implement REGEX_REPLACE solution for MySQL 8.0+ installations - Core: Create search configuration settings in WordPress admin - Core: Address block editor HTML comment pollution in search queries - Core: Develop progressive enhancement for improved search on supported databases - Documentation: Document search limitations and workarounds for developers - Community: Gather feedback on search improvement priorities from users #### WordPress - Plugins: Review existing search plugins for markup pollution solutions - Themes: Ensure search templates handle empty/poor results gracefully - Hosting: Evaluate MySQL/MariaDB versions for REGEX_REPLACE support ## 🧪 Hypotheses - REGEX_REPLACE implementation will improve search relevance while maintaining acceptable performance on smaller sites - Basic search filtering options in core would reduce reliance on search plugins - Progressive enhancement approach could provide better search for modern database setups without breaking compatibility - Removing HTML comments from search queries will significantly reduce false positives ### 📊 Methodology - Performance testing REGEX_REPLACE queries on various database sizes - A/B testing search relevance with and without markup removal - User survey on most-needed search filtering features - Database version analysis across WordPress installations ### ❓ Questions - What percentage of WordPress sites run MySQL 8.0.4+ or MariaDB 10.0.7+ to support REGEX_REPLACE? - How significant is the performance impact of REGEX_REPLACE on typical WordPress search queries? - Which basic search filtering options would provide the highest user value with lowest implementation complexity? ### 🔮 Future Research - Comprehensive analysis of search plugin adoption and feature usage - Investigation into block editor content storage optimization for search - Development of lightweight search indexing for WordPress core ### 💡 Implications - Practice: Developers may need to implement custom search solutions or recommend plugins more frequently - Policy: WordPress core development priorities may need adjustment to address search functionality gaps - Education: Users need better documentation about search limitations and available solutions - Funding: Investment in search infrastructure improvements could benefit the entire WordPress ecosystem ### 🧠 Learning Styles Assessment Reading level estimate: Grade 12 #### 👀 Visual learners - Present: Clear problem examples and solution comparisons - Improve: Add database architecture diagrams, search result comparison screenshots #### 🔊 Aural learners - Present: Video presentation with clear narration - Improve: Provide audio summary of key technical concepts #### 🗣️ Verbal learners - Present: Technical terms explained with context and examples - Improve: Add glossary of MySQL and WordPress search terminology #### 🧩 Kinesthetic learners - Present: Practical problem demonstration with real search examples - Improve: Add step-by-step REGEX_REPLACE implementation guide #### 🧠 Logical learners - Present: Clear cause-and-effect relationship between block editor and search pollution - Improve: Add decision tree for choosing search solutions #### 🧑‍🤝‍🧑 Social learners - Present: References to community discussions and developer input - Improve: Add links to relevant WordPress Slack channels or forums #### 🧘‍♀️ Solitary learners - Present: Comprehensive problem analysis and potential solutions - Improve: Add self-assessment checklist for search issues #### 🌿 Naturalistic learners - Present: Evolution of WordPress search problems over time - Improve: Connect search performance to overall site ecosystem health ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure and technical explanations - Improve: Add more descriptive alt text for any code examples #### Keyboard navigation - Present: Standard video controls accessible via keyboard - Improve: Ensure any interactive elements are fully keyboard accessible #### Contrast and readability - Present: Clear verbal presentation - Improve: Ensure any visual aids meet contrast requirements #### Text resizing and reflow - Present: Video format adapts to different screen sizes - Improve: Provide transcript for better text resizing support #### Media and captions - Present: Audio content available - Improve: Add closed captions and transcript for accessibility #### Forms and interactions - Present: No complex interactions required - Improve: N/A for video content ### 🌍 DEIB Assessment #### Cultural breadth - Present: German developer perspective on global WordPress issues - Improve: Include examples from different regions and languages #### Inclusive language - Present: Technical language appropriate for developer audience - Improve: Define more technical terms for broader accessibility #### Neurodiversity support - Present: Clear problem-solution structure - Improve: Add visual breaks and section summaries #### Representation - Present: Technical focus with specific developer credited - Improve: Include diverse perspectives on search user needs #### Psychological safety - Present: Respectful critique of WordPress limitations - Improve: Frame issues as opportunities rather than failures ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear cause-effect relationship between block editor HTML comments and search pollution - Inductive strength: Uses specific examples to demonstrate broader search problems - Causal reasoning: Correctly identifies technical root causes of search issues - Analogical strength: Good comparison between simple and complex search solutions #### Fallacies - Appeal to authority: Relies heavily on core developer quotes without exploring alternative perspectives - False dichotomy: Presents choice mainly between status quo and Elasticsearch without exploring middle options thoroughly #### Tone of voice - Overall tone: Professional, technical, constructively critical with specific examples and proposed solutions ### 🛡️ DARVO Analysis - Deny: No denial patterns present - Attack: No attacks on accusers present - Reverse victim/offender: No victim/offender reversals present - Emotional cues and rhetoric: Constructive technical criticism focused on solutions rather than blame ## 🎯 Conclusion WordPress's native search functionality needs significant improvement to meet modern user expectations. The block editor's HTML comment storage creates pollution that makes basic search terms unusable, while the lack of filtering options forces users toward complex solutions or custom code. The proposed REGEX_REPLACE solution offers a practical middle ground for sites running modern database versions, though WordPress core should ultimately provide basic search configuration options. This represents a broader need to revisit foundational features that were deprioritized during the block editor focus phases. Site owners should evaluate existing search plugins or consider custom implementations while advocating for core improvements through WordPress development channels. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad ### [H1] WordPress native search lacks essential filtering capabilities > [!quote] [Nobody loves the native search](https://www.youtube.com/watch?v=UQ-X3wPoxuQ) by [[Thörn Lüsteal]] via [[YouTube]]. Published 2024. Accessed 2024-12 <!-- deep-read:sections end -->