Nobody loves the native search  ## Metadata - Author: [[YouTube]] - Full Title: Nobody loves the native search - Category: #articles - Document Note: --- tags: [wordpress-core, search, mysql, performance] author: Thörn Lüsteal source: YouTube reference: https://www.youtube.com/watch?v=UQ-X3wPoxuQ date created: 2024-12-28 15:30 --- # 🏷️ Title: Nobody loves the native search ## 🧰 Action Inbox - [H1] Report search markup pollution issue to WordPress core — 00:00 #wordpress-core - [H2] Test REGEX_REPLACE solution on MySQL 8.0.4+ — 10:45 #development - [H3] Advocate for basic search settings in core — 13:20 #feature-request ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [WordPress Search Issues](https://www.youtube.com/watch?v=UQ-X3wPoxuQ) - [MySQL REGEX_REPLACE Documentation](https://dev.mysql.com/doc/refman/8.0/en/regexp.html) ## 📝 Summary - WordPress native search has significant limitations including inability to filter by author, date, categories, or custom post types without coding - Block editor creates HTML comments that pollute search results, making terms like "paragraph," "image," and "background" essentially useless - Core developers acknowledge this as a known issue but suggest expensive solutions like Elasticsearch - A potential middle-ground solution using MySQL's REGEX_REPLACE function could remove HTML markup before searching - The speaker advocates for basic search settings in WordPress core rather than requiring custom code ## 📖 Main Content WordPress's native search functionality has fundamental flaws that affect user experience across millions of websites. The search only covers title, excerpt, and content using simple full-text SQL queries, with no built-in options for filtering by author, date, tags, categories, or custom post types. A critical issue emerged with the block editor: HTML comments containing block names and settings are stored in post content and included in searches. This means searching for "syntax highlighter" returns every post containing code blocks, even when the term doesn't appear in the visible content. Similar pollution occurs with common terms like "paragraph," "image," and "background." Core developers Gary Pendergast and Daniel Bachhuber acknowledge this as a known limitation, suggesting solutions like Elasticsearch or custom indexing. However, these approaches are impractical for typical WordPress users and clients. The speaker proposes a middle-ground solution using MySQL 8.0.4+'s REGEX_REPLACE function to strip HTML markup before searching. While this has performance implications, it could work acceptably for smaller sites without requiring dedicated search infrastructure. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=0) — [H1] WordPress Search Limitations - Key idea: Native WordPress search lacks basic filtering capabilities - Evidence: "You can't filter for an author date tags or categories you can't exclude a page or a post" #### ⏱️ [01:30](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=90) — [H2] How WordPress Search Works - Key idea: Search uses simple full-text SQL query on title, excerpt, and content - Evidence: "It searches the title The excerpt and the content with a very very simple full text SQL query" #### ⏱️ [02:45](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=165) — [H2] Block Editor Search Pollution Problem - Key idea: Block editor HTML comments contaminate search results - Evidence: "I searched for syntax highlighter and I was wondering I got so many results and they had the even they don't have the search term in them" #### ⏱️ [04:20](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=260) — [H3] Root Cause of False Positives - Key idea: Block names and settings stored as HTML comments are searchable - Evidence: "The blog editor Saves The Blog name and some settings in the HTML comment in the Raw content" #### ⏱️ [05:15](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=315) — [H2] Search Term Pollution Impact - Key idea: Common words become useless for search due to markup pollution - Evidence: "You can't really search for paragraph or image or background you will get every every blog post or nearly every blog post" #### ⏱️ [06:30](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=390) — [H2] Core Developer Response - Key idea: Leading developers acknowledge this as a known MySQL limitation - Evidence: "MySQL string searching isn't capable of dealing with this kind of contextual passing" - Gary Pendergast #### ⏱️ [07:45](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=465) — [H3] Suggested Solutions from Core - Key idea: Core developers suggest Elasticsearch or custom indexing - Evidence: "If you require 100% accurate search results the best option is to use a dedicated search engine like elastic search" #### ⏱️ [08:30](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=510) — [H2] Elasticsearch Impracticality - Key idea: Elasticsearch setup is unrealistic for typical WordPress users - Evidence: "Setting up elastic search is not what the typical user or client I have would do" #### ⏱️ [09:15](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=555) — [H1] REGEX_REPLACE Solution - Key idea: MySQL 8.0.4+ offers REGEX_REPLACE function for markup removal - Evidence: "With the MySQL 8.0.4 and MariaDB 10.0.7 onwards there is a new function called regex replace" #### ⏱️ [10:45](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=645) — [H3] Performance Trade-offs - Key idea: REGEX solution has performance impact but could work for smaller sites - Evidence: "This will of course have some import performance impact but for smaller sites this could be acceptable" #### ⏱️ [11:30](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=690) — [H3] Implementation Challenges - Key idea: Various edge cases need consideration for markup removal - Evidence: "File names and alternative text... short codes could be also removed... some page Builders are using Json" #### ⏱️ [12:15](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=735) — [H2] Focus Phases Limitation - Key idea: WordPress focus phases left search functionality behind - Evidence: "We have those focus on the four phases... but now after some years I think we need to look at those things that we have left behind" #### ⏱️ [13:20](https://www.youtube.com/watch?v=UQ-X3wPoxuQ&t=800) — [H1] Core Search Settings Request - Key idea: WordPress needs basic search configuration options in core - Evidence: "Important features like SEO sitemaps or the search would be much better with at least some basic settings" ## 📚 References - [WordPress Core Development](https://make.wordpress.org/core/) - [MySQL REGEX_REPLACE Function](https://dev.mysql.com/doc/refman/8.0/en/regexp.html) - [Elasticsearch Documentation](https://www.elastic.co/guide/) ## 🗣️ Paraphrase - WordPress's default search is too basic, lacking filters for authors, dates, categories, and custom post types that users expect - The block editor inadvertently polluted search results by storing block metadata as HTML comments in post content, making common words unsearchable - While core developers recognize the problem, their suggested solutions like Elasticsearch are too complex for typical WordPress sites ## 📰 Newsletter **WordPress Search Needs an Overhaul** WordPress's native search functionality has fallen behind user expectations, lacking basic filtering options and suffering from block editor pollution that makes common terms like "paragraph" and "image" return irrelevant results. Core developers acknowledge the issue stems from MySQL's limitations with contextual parsing, but their suggested solutions like Elasticsearch aren't practical for most sites. A proposed middle-ground using MySQL's REGEX_REPLACE function could strip HTML markup before searching, though with performance trade-offs. **Key Takeaway**: WordPress core needs basic search configuration options rather than forcing users to code solutions or implement complex search infrastructure. **Next Steps**: Consider advocating for search improvements in WordPress core development discussions, or explore existing search plugins that address these limitations. Resources: [WordPress Core](https://make.wordpress.org/core/), [Search Plugins](https://wordpress.org/plugins/tags/search/), [Performance Best Practices](https://make.wordpress.org/core/handbook/references/php-compatibility-and-wordpress-versions/) ## ✅ Action Items #### WordPress - Core: Add basic search filtering options to WordPress admin - Core: Implement REGEX_REPLACE solution for MySQL 8.0+ installations - Core: Create search configuration settings in WordPress admin - Core: Address block editor HTML comment pollution in search queries - Core: Develop progressive enhancement for improved search on supported databases - Documentation: Document search limitations and workarounds for developers - Community: Gather feedback on search improvement priorities from users #### WordPress - Plugins: Review existing search plugins for markup pollution solutions - Themes: Ensure search templates handle empty/poor results gracefully - Hosting: Evaluate MySQL/MariaDB versions for REGEX_REPLACE support ## 🧪 Hypotheses - REGEX_REPLACE implementation will improve search relevance while maintaining acceptable performance on smaller sites - Basic search filtering options in core would reduce reliance on search plugins - Progressive enhancement approach could provide better search for modern database setups without breaking compatibility - Removing HTML comments from search queries will significantly reduce false positives ### 📊 Methodology - Performance testing REGEX_REPLACE queries on various database sizes - A/B testing search relevance with and without markup removal - User survey on most-needed search filtering features - Database version analysis across WordPress installations ### ❓ Questions - What percentage of WordPress sites run MySQL 8.0.4+ or MariaDB 10.0.7+ to support REGEX_REPLACE? - How significant is the performance impact of REGEX_REPLACE on typical WordPress search queries? - Which basic search filtering options would provide the highest user value with lowest implementation complexity? ### 🔮 Future Research - Comprehensive analysis of search plugin adoption and feature usage - Investigation into block editor content storage optimization for search - Development of lightweight search indexing for WordPress core ### 💡 Implications - Practice: Developers may need to implement custom search solutions or recommend plugins more frequently - Policy: WordPress core development priorities may need adjustment to address search functionality gaps - Education: Users need better documentation about search limitations and available solutions - Funding: Investment in search infrastructure improvements could benefit the entire WordPress ecosystem ### 🧠 Learning Styles Assessment Reading level estimate: Grade 12 #### 👀 Visual learners - Present: Clear problem examples and solution comparisons - Improve: Add database architecture diagrams, search result comparison screenshots #### 🔊 Aural learners - Present: Video presentation with clear narration - Improve: Provide audio summary of key technical concepts #### 🗣️ Verbal learners - Present: Technical terms explained with context and examples - Improve: Add glossary of MySQL and WordPress search terminology #### 🧩 Kinesthetic learners - Present: Practical problem demonstration with real search examples - Improve: Add step-by-step REGEX_REPLACE implementation guide #### 🧠 Logical learners - Present: Clear cause-and-effect relationship between block editor and search pollution - Improve: Add decision tree for choosing search solutions #### 🧑‍🤝‍🧑 Social learners - Present: References to community discussions and developer input - Improve: Add links to relevant WordPress Slack channels or forums #### 🧘‍♀️ Solitary learners - Present: Comprehensive problem analysis and potential solutions - Improve: Add self-assessment checklist for search issues #### 🌿 Naturalistic learners - Present: Evolution of WordPress search problems over time - Improve: Connect search performance to overall site ecosystem health ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure and technical explanations - Improve: Add more descriptive alt text for any code examples #### Keyboard navigation - Present: Standard video controls accessible via keyboard - Improve: Ensure any interactive elements are fully keyboard accessible #### Contrast and readability - Present: Clear verbal presentation - Improve: Ensure any visual aids meet contrast requirements #### Text resizing and reflow - Present: Video format adapts to different screen sizes - Improve: Provide transcript for better text resizing support #### Media and captions - Present: Audio content available - Improve: Add closed captions and transcript for accessibility #### Forms and interactions - Present: No complex interactions required - Improve: N/A for video content ### 🌍 DEIB Assessment #### Cultural breadth - Present: German developer perspective on global WordPress issues - Improve: Include examples from different regions and languages #### Inclusive language - Present: Technical language appropriate for developer audience - Improve: Define more technical terms for broader accessibility #### Neurodiversity support - Present: Clear problem-solution structure - Improve: Add visual breaks and section summaries #### Representation - Present: Technical focus with specific developer credited - Improve: Include diverse perspectives on search user needs #### Psychological safety - Present: Respectful critique of WordPress limitations - Improve: Frame issues as opportunities rather than failures ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear cause-effect relationship between block editor HTML comments and search pollution - Inductive strength: Uses specific examples to demonstrate broader search problems - Causal reasoning: Correctly identifies technical root causes of search issues - Analogical strength: Good comparison between simple and complex search solutions #### Fallacies - Appeal to authority: Relies heavily on core developer quotes without exploring alternative perspectives - False dichotomy: Presents choice mainly between status quo and Elasticsearch without exploring middle options thoroughly #### Tone of voice - Overall tone: Professional, technical, constructively critical with specific examples and proposed solutions ### 🛡️ DARVO Analysis - Deny: No denial patterns present - Attack: No attacks on accusers present - Reverse victim/offender: No victim/offender reversals present - Emotional cues and rhetoric: Constructive technical criticism focused on solutions rather than blame ## 🎯 Conclusion WordPress's native search functionality needs significant improvement to meet modern user expectations. The block editor's HTML comment storage creates pollution that makes basic search terms unusable, while the lack of filtering options forces users toward complex solutions or custom code. The proposed REGEX_REPLACE solution offers a practical middle ground for sites running modern database versions, though WordPress core should ultimately provide basic search configuration options. This represents a broader need to revisit foundational features that were deprioritized during the block editor focus phases. Site owners should evaluate existing search plugins or consider custom implementations while advocating for core improvements through WordPress development channels. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad ### [H1] WordPress native search lacks essential filtering capabilities > [!quote] [Nobody loves the native search](https://www.youtube.com/watch?v=UQ-X3wPoxuQ) by [[Thörn Lüsteal]] via [[YouTube]]. Published 2024. Accessed 2024-12 - Document Tags: [[feature]] [[wordpress]] [[development]] - Summary: Nobody loves the native search - URL: https://www.youtube.com/watch?v=UQ-X3wPoxuQ so let's go to Thomas L Steel around more or less Thomas is a WordPress freelancer from hurg Germany and we will talk about why nobody nobody likes the uh Native search of Wordpress wonder why welcome to Thomas so um I'm toron not Thomas oh that's no problem so I hope this yeah it works works uh it works for me works for you Serge is is it working for you for me it not really works well and I asked this uh question on on social media in the last days and uh got so many replies and uh the replies um where uh you can't filter for an author date uh tags or categories you can't exclude a page um or a post uh you can't limit um the search to to just search one custom post type or you can't exclude one um and you can't set up the search for um uh searching for custom Fields uh without coding so the search has its limits so what does the search do it searches the title The excerpt and the content with a very very simple uh full t SQL query and it shows the results ordered by relevance and it's a very simple sort of relevance so um but what if you search for something that is part of a block name so let me tell you a little story I uh searched the the block from a um a colleague in Germany um Bernard cow I know he um has blocked about the syntax highlighter U brush for Apache Snippets and I I know it I just need to find this blog post so I searched for syntax highlighter and I was wondering I got so many results and they had the even they don't have the search I term in them and then I got it I got every blog post with a code snippet in it but why does this happen this is because the blog editor Saves The Blog name and some settings in the uh HTML comment in the Raw content so every post with the syntax highlighter blog um was found through the SQL query in the Raw content but we do not want this common to be searched and uh ignoring the comment is not enough because uh there are often some CSS classes also and we need to get rid of them at least for the search so we want to search for the content but not for the markup I mean this is not a minor issue um for some edge cases with every new blog and every new setting we are polluting The Surge more and more and there are words now in WordPress which are essentially useless uh for the search you can't really search for paragraph or image or background you will get every every blog post or nearly every blog post so what is the opinion from the leading developers in this case of course um I reported this issue un unfortunately this is a known issue in WordPress because if you search for table uh you get results including the table tag MySQL string searching isn't capable of dealing with this kind of contextual passing and if you require 100% accurate search results the best option is to use a dedicated search engine like elastic search says Gary panderas so another comment I got was one opportunity is um for search backends to render the blogs fully before sending post content into the index so we need to build up a know index and I don't have any great ideas how to resolve this with mySQL I'd love to hear of a solution if someone has one says Daniel bber so Solutions please I think elastic search setting up elastic search is not what the the typical user um or client I have uh would do so and um building an own index in a new table is uh too much to do for WordPress maybe so I tried to find a simpler solution something in between those two solutions and I found one with the MySQL 8.0.4 and Mar IB 10.0.7 onwards there is a new function called regex replace now we can replace text um by a regular expressions in an SQL query for example we could remove every HTML command before we do the search yes this will of course um have some import performance impact but for smaller sites this could be acceptable a dedicated index will be better uh and there are plenty of plug-in solutions for it Unfortunately they still um often do not solve the problem with those false positives so um let's have a look um for the challenges uh what about file names and alternative text if we just remove REM the comment and maybe the CSS classes um then we will have a decent search uh result and file names and alternative text will still be searchable short codes could be also removed at least for the frontend search um we could um do this just for um those databases which supports it so we could have this as an Progressive enhancement um and It's tricky to remove the complete markup because uh we have none encoded characters sometimes the greater than lower than character sometimes it's not real reliable um uh that these are encoded correctly in the database there's a whole years old trick ticket about it um and some page Builders are um using Json to save uh data and maybe we need more adjustments uh to help those uh edge cases to have a decent search so um my conclusion is uh we have those focus on the four faes and the four phases are a necessary uh way to to focus the work on the most important uh part uh we want to do at one time but now after some years I think we need to look look at those things that we have left behind and the search one of the most basic advantages of a CMS is missing much uh needed features um and is not easy to extend without coding um I will skip this part um sometimes uh the philosophy of decisions not options uh need to be adjusted important features like SEO sitemaps or the search um would be much better with at least some basic settings so um if I have three fre wishes I would love to have more search features in core I would love to have um at least some basic search settings and um the part I skipped is about um translation strings which are um not containing text but uh settings settings um that I used to switch things on and off or um um set things up and uh those uh translations are missing um safeguards uh so you can translate them wrongly and break things so um if you want to help me get this fixed please talk to me um I'm zodiac 1978 on social media um I have collected more information and some links on this uh page um uh yeah and thank you for being here and he my talk than thanks Thon a gift for you thank you for being here, Nobody loves the native search, YouTube, 01ka7et731g3vmhbjjexjrb1zy.