How We Made Our Plugin 30x Faster  🔗 **Source:** [How We Made Our Plugin 30x Faster](https://twitter.com/rossmorsali/status/2092655210338361521/?rw_tt_thread=True) > [!note]+ Document Notes > tags: [wordpress-performance, faceted-search, bitmap-indexes, ai-assisted-development, benchmarking] > author: Ross Morsali > source: X (@rossmorsali) > reference: https://twitter.com/rossmorsali/status/2092655210338361521 > date created: 2026-09-12 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: How we made our plugin 30x faster > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References (in-page) - The article links only to the @SearchAndFilter account (checked against the stored HTML). It links no benchmark suite, code, dataset, research sources, or release notes. - One embedded image: a line chart titled by alt text "How we made our plugin 30x faster," plotting "Total workload" for "Historical" vs "Current" at 2k–40k posts/products. ## 📝 Summary - **What happened.** On 2026-08-26 Ross Morsali published an X article on rewriting the indexer of Search & Filter, a WordPress faceted search plugin by Code Amp. The rewrite took about three months. The author credits AI "deep research" for finding techniques and agent goal loops for micro-optimizations, then picked approaches by benchmarking. - **The technique.** Filtering combines large sets of post IDs. The new indexer represents those sets as bitmaps (inspired by Roaring Bitmaps, which the author says most WordPress hosts can't run), splits indexing into three table designs by filter type (range buckets, choice-field inverted indexes, a parent map for WooCommerce variations), and adds a native search index with BM25 scoring, typo tolerance, and a separate autocomplete index. - **The numbers.** Uncached frontend queries: 102 ms → 23.2 ms at 2,000 posts (4.4x) and 1.44 s → 50.7 ms at 40,000 posts (28.4x). Both ratios check out. The title's "30x" rounds up 28.4x. - **Why it matters.** Faceted filtering on large WooCommerce and content sites is a common WordPress performance bottleneck. The post shows a way to find data structures outside the usual "post ID mapped to field value" index while staying inside PHP and MySQL. - **Risks and tradeoffs.** The benchmark method isn't published: no hardware, PHP or MySQL versions, filter configuration, run counts, or which "previous version." The chart's values don't match the text's. The benchmarks were written "using AI," with no mention of review. The company's January 2026 release post claimed "10x," and the article doesn't reconcile the two figures. - **Bottom line.** A useful account of a technique change (bitmaps plus type-specific tables) and a method (research breadth-first, then benchmark everything), with performance claims that can't be reproduced from what's published. ## 📖 Main Content - **The core problem.** A faceted search plugin repeatedly intersects and unions large arrays of post IDs, once for results and again for each facet's option counts. Combining arrays in PHP is the cost the rewrite attacks. - **Step 1, micro-optimizations.** An agent running a `/goal` loop found many small wins that "did add up, but nothing out of this world." The post doesn't name the agent tool. - **Step 2, research breadth-first.** The author asked AI deep research what large-scale systems use for this, telling it not to rule anything out, then looked for what could run inside WordPress. - **Bitmaps.** A bitmap stores set membership as one bit per ID, so combining two filters becomes a bitwise operation over compact data instead of an array operation over thousands of integers. Roaring Bitmaps add compression; roaringbitmap.org lists them in Apache Lucene, Elasticsearch, Druid, Spark, and ClickHouse. - **PHP constraint.** The author says Roaring Bitmaps "are not available on most WordPress hosts," so the plugin uses bitmaps in PHP with "alternative compression techniques," which the post doesn't name. - **Type-specific tables.** Range fields use bitmap buckets; choice fields use a bitmap inverted index; WooCommerce variations use a parent map (nested document pattern). - **Facet counts.** The author says facet counts are the most demanding part, and that combining per-filter bitmaps beats array approaches there. - **Search.** A new native search index adds BM25 scoring, fuzzy and typo-tolerant matching, and a separate autocomplete index. - **Step 3, benchmark everything.** For each plausible option the team built benchmarks (written with AI) across PHP versions and server configurations, then chose the fastest options that work in restricted environments. - **Caching caveat.** All figures are uncached. With Redis or the plugin's database caching, repeat hits on the same filter combination get faster after the first request. - **Closing advice.** Combine deep research (for options outside the model's default knowledge) with self-improving agent loops (for iteration). **Terms** - *Faceted search:* filtering with multiple independent criteria, each showing how many results each option would return. - *Bitmap (bitset):* a sequence of bits where position N marks whether item N is in the set. - *Roaring Bitmap:* a compressed bitmap format that switches container types by data density. - *Inverted index:* a map from each field value to the set of items that have it. - *BM25:* a standard relevance-ranking function for text search. **Uncertainties and contested claims** - **Benchmark conditions** are unstated, so the 4.4x and 28.4x figures can't be reproduced or compared. - **Chart vs text.** Read by eye from the image, the chart's "Historical" line is about 250 ms at 2k and about 4.45 s at 40k, while the text says 102 ms and 1.44 s. The chart's "Total workload" likely aggregates more than one query, but the post doesn't say. - **10x vs 28.4x.** The Search & Filter 3.2.0 release post (2026-01-09) described the rewritten indexer as "10x faster (and more in some circumstances)" and promised "our upcoming performance article… (with benchmarks!)." The X article matches that promise but doesn't say whether the gap reflects later work or a different measurement. - **Distribution.** The free Search & Filter listing on WordPress.org is still version 1.2.18 (Code Amp, 50,000+ active installs, updated 2026-08-22), with a changelog notice that "Version 3 is coming soon." The rewritten v3 indexer ships from searchandfilter.com, not that listing. ## 🗒️ Notes - Published 2026-08-26 as an X article; 896 words. - Rewrite of the Search & Filter indexer "from the ground up"; about 3 months. - Faceted search backbone: huge post ID arrays combined repeatedly per filter. - Micro-optimizations via an agent `/goal` loop: cumulative gains, none large. - WordPress plugins are limited to PHP plus MySQL or MariaDB. - Most WordPress filter plugins map post IDs to field values in an index. - Research question: what do large-scale systems (the post's example: "the Amazons of the world") use, and can any of it run in WordPress? - Win 1: bitmaps (Roaring-inspired); implemented in PHP with alternative compression. - Example: a filter matching 20,000 posts means 20,000 IDs in a traditional index, versus one compact binary structure. - Win 2: three subsystems by filter type: range buckets, choice-field inverted index, WooCommerce variation parent map. - Facet option counts are the heaviest workload; bitmap combination is faster than PHP arrays there. - Win 3: native advanced search with fuzzy matching, BM25, autocomplete, and a separate autocomplete index. - Selection: AI-written benchmark suites across PHP versions and server configs; fastest option that runs in restricted environments. - Deep research gives more options than agent loops, which rely on built-in knowledge. - Data (uncached): 2,000 posts, 102 ms → 23.2 ms (4.4x); 40,000 posts, 1.44 s → 50.7 ms (28.4x). - Caching (Redis or the plugin's database cache) lowers repeat-hit times. - Recommendation: combine deep research with self-improving agent loops. - Copy issues: one sentence repeats ("So that was around 3 months of research and building, end to end."); typos "could have take" and "unmanagable." ## 📚 References - [How we made our plugin 30x faster — Ross Morsali on X](https://twitter.com/rossmorsali/status/2092655210338361521) (primary source) - [Search & Filter Version 3.2.0 Release](https://searchandfilter.com/version-3-2-0-release/) (2026-01-09; "10x" indexer claim, bitmap and bucket tables, BM25, promised benchmark article) - [Search & Filter on WordPress.org](https://wordpress.org/plugins/search-filter/) (free listing, version 1.2.18 as of 2026-09-12) - [Roaring Bitmaps](https://roaringbitmap.org/) (format and adopters) - [PECL](https://pecl.php.net/) — no `roaring` package (HTTP 404 for `/package/roaring`, checked 2026-09-12); Packagist search for "roaring bitmap" returns 3 packages, the largest with 39 downloads ## 🗣️ Paraphrase - **Intro.** The team rebuilt the Search & Filter indexer over three months, using AI research and agent loops. - **The problem.** Filtering means combining big lists of post IDs many ways, and doing that fast is the whole challenge. - **Micro-optimizations.** An agent loop hunting for speedups found many small wins but nothing dramatic. - **Research.** Because WordPress plugins are stuck with PHP and a relational database, and competitors all index the same way, the author asked AI what large-scale systems do and told it to consider everything. - **Win 1.** Large systems use compressed bitmaps. Each post ID becomes a bit, so a 20,000-post match is one compact structure instead of 20,000 integers, cutting storage, memory, and data movement. Roaring Bitmaps aren't on most hosts, so the team built a PHP equivalent. - **Win 2.** Different filters need different storage, so the indexer split into range buckets, choice-field inverted indexes, and a parent map for WooCommerce variations. Bitmaps make facet counts, the hardest part, much faster than PHP arrays. - **Win 3.** The rewrite also added long-missing native search: typo tolerance, BM25 ranking, and a separate autocomplete index. - **Choosing.** The team benchmarked each candidate across PHP versions and server setups and kept the fastest that works on restricted hosting. Research finds more options than an agent loop, which relies on what the model already knows. - **Data.** Uncached, 2,000-post queries went from 102 ms to 23.2 ms, and 40,000-post queries from 1.44 s to 50.7 ms. Caching improves repeat queries further. - **Advice.** When optimizing a plugin with AI, pair deep research with agent loops. ## 📰 Newsletter **Bits beat arrays: how Search & Filter got 28x faster.** Ross Morsali's write-up on rebuilding the Search & Filter indexer is a rare look inside WordPress faceted search. The team swapped PHP arrays of post IDs for bitmaps (the same idea behind the Roaring Bitmaps used in Lucene and Elasticsearch, rebuilt in plain PHP because hosts don't ship the extension), split the index by filter type, and added BM25 search. Uncached queries on a 40,000-post site dropped from 1.44 seconds to 50.7 milliseconds. The method is the part to borrow: use AI deep research to widen your options past what any agent already "knows," then write benchmarks and let the numbers pick. One caution before you quote the headline number: the benchmark setup isn't published, the chart and text disagree, and January's release notes said 10x. 👉 [Read the thread](https://twitter.com/rossmorsali/status/2092655210338361521) · [Search & Filter 3.2.0 release notes](https://searchandfilter.com/version-3-2-0-release/) ## ✅ Action Items #### DevRel - Developer advocacy: a talk or demo on "research breadth-first, benchmark to choose" for plugin performance work, using this rewrite as the case study. - Education: a short workshop on bitsets and inverted indexes for WordPress developers who only know `WP_Query` and `tax_query`. - Developer experience: ask plugin authors publishing speedup claims to include a reproducible benchmark harness (dataset generator, PHP and MySQL versions, run counts). - Developer marketing: Not applicable to this source — no campaign content. - Community management: invite Search & Filter to post the benchmark method in reply so the numbers can be discussed on evidence. - Internal/external/cross-project: share the userland-PHP bitmap approach with other faceted search plugin maintainers. #### OSPO - Policy & Governance: set a rule that AI-written benchmarks get human review of what they measure before results are published. - License Compliance: if the PHP bitmap code derives from Roaring's published algorithms or code, confirm license compatibility (Roaring's reference implementations use permissive licenses; verify the specific repo). - Security Management: Not applicable to this source. - Software Inventory: track which PHP extensions a plugin assumes (APCu, Redis) versus supports optionally. - Developer Support: provide a template benchmark README: environment, dataset, method, variance. - Community Engagement: Not applicable to this source. - Strategic Alignment: Not applicable to this source. #### Community - Maintainers: publish benchmark method with any performance claim, and reconcile headline numbers across posts (10x in January, 28.4x in August). - Contributors: a reproducible faceted-search benchmark dataset (for example, a WooCommerce product generator at 2k–40k products) would help every plugin in the category. - Sponsors: Not applicable to this source. - Ecosystem: compare approaches across faceted search plugins on one shared dataset. #### WordPress - Core: Not applicable to this source — plugin-level indexing, no core change proposed. - Core Performance: study whether bitset-based set operations could speed up large `tax_query` and meta query combinations, and document the plugin's approach as a case study. - Core AI: collect this as an example of AI-assisted research plus human-chosen benchmarks for performance work. - **Core Program Team:** Not applicable to this source. - Mobile: Not applicable to this source. - Playground: a Playground blueprint with a 40,000-post generated dataset would let anyone rerun filter benchmarks in the browser (Playground timings won't match servers, but relative comparisons can). - Community: Not applicable to this source. - Training: a Learn WordPress lesson on indexing strategies for custom filtering (post meta vs taxonomy vs custom tables vs bitmaps). - Marketing: Not applicable to this source. - Support: Not applicable to this source. - Design: Not applicable to this source. - Documentation: a developer handbook page on when to build custom index tables instead of relying on `postmeta` queries. - Photos: Not applicable to this source. - TV: a WordPress.tv session on the rewrite, with the benchmark method on screen. - Polyglots: typo-tolerant and fuzzy search behave differently across languages; the 3.2.0 post lists stemming in 14 languages, worth testing for non-Latin scripts. - Accessibility: the article's chart has alt text that only repeats the title; describe the data. - Meta: Not applicable to this source. - CLI: a WP-CLI benchmark command pattern (generate N posts, run M filter combinations, report p50/p95) would standardize claims like these. - Hosting: document which PHP extensions (APCu, Redis, igbinary) are commonly available, so plugin authors know what they can assume. - Tide: Not applicable to this source. - Openverse: Not applicable to this source. - Themes: Not applicable to this source. - Plugins: encourage performance claims in plugin listings to link a method. #### The WPOCC - Programs: fund or host a shared benchmark dataset and harness for plugin performance claims. - Projects: a documentation sprint on custom index tables and set operations in WordPress. - Funding experiments: Not applicable to this source. #### FAIR - Technical independence: Not applicable to this source. - Policies and covenants: Not applicable to this source. - Productized governance artifacts: a "performance claim" disclosure template (environment, dataset, method, variance) could sit alongside other plugin metadata. ## 🧪 Hypotheses 1. On a published dataset, the new indexer's uncached p50 facet query time at 40,000 posts falls within 20% of the reported 50.7 ms on mainstream shared hosting (PHP 8.2+, MySQL 8). 2. The speedup ratio grows with post count because bitmap combination cost scales with ID range density while array combination scales with matched set size. 3. The chart's "Total workload" sums several queries per page load (results plus each facet's counts), which explains why it exceeds the text's single-query figures. 4. The 10x (January) and 28.4x (August) figures come from different workloads or versions, not from contradictory results. 5. Deep research surfaces at least one candidate technique that an agent `/goal` loop without research doesn't propose, on the same optimization task. 6. Userland PHP bitmaps lose most of their advantage on sparse filters (few matches over a large ID range) compared with sorted-array intersection. ## 📊 Methodology - **H1–H3:** Generate WooCommerce datasets at 2k, 10k, 20k, 30k, and 40k products with fixed attribute distributions. Install the previous and current plugin versions in identical containers (pinned PHP and MySQL). Run a fixed set of filter combinations 30 times each with object caching disabled. Record p50 and p95 per query and per page load, and count SQL queries per page to test H3. - **H4:** Request the benchmark definitions behind both published figures; rerun both on the same dataset. - **H5:** Give two agent sessions the same slow function and budget. One gets a deep research brief first; one runs a goal loop only. Compare the distinct techniques each proposes and the best benchmark result each reaches. - **H6:** Vary filter selectivity from 0.1% to 50% of posts at fixed dataset size; compare bitmap vs array intersection timings. ## ❓ Questions - What hardware, PHP version, MySQL or MariaDB version, dataset shape, and filter set produced 102 ms → 23.2 ms and 1.44 s → 50.7 ms? - Which "previous version" is the baseline: pre-3.2.0, or 3.2.0 itself? - What does the chart's "Total workload" measure, and why does it differ from the text's figures? - How do the January "10x" and August "28.4x" claims relate? - Which "alternative compression techniques" does the PHP bitmap implementation use (for example, run-length encoding, packed strings, or GMP)? - How are bitmaps stored in MySQL (BLOB columns, chunked rows), and what does an index update cost when a post changes? - Who reviewed the AI-written benchmarks for what they measure? - When will the v3 indexer reach the WordPress.org listing, still at 1.2.18? ## 🔮 Future Research - A shared, published benchmark for WordPress faceted search plugins on identical datasets and environments. - Userland PHP bitset performance across PHP 8.x versions and JIT settings. - Write amplification: how bitmap indexes affect save-time and bulk-import cost on large stores. - Whether a PHP extension for Roaring Bitmaps could reach enough hosts to matter, and what plugins would gain. - Measured comparisons of AI deep research versus goal-loop-only agents for performance engineering. ## 💡 Implications - **Practice:** plugin authors facing PHP and MySQL limits can borrow data structures from search engines instead of tuning the same index design. - **Policy:** performance claims without methods train readers to distrust all numbers; a disclosure norm would help the category. - **Security:** Not applicable to this source. - **Education:** set operations and index design are learnable skills most WordPress developers never meet through `WP_Query`. - **Funding models:** a commercial plugin funded three months of rewrite work; a shared benchmark harness is the kind of public good no single vendor has an incentive to maintain. ## 🧠 Learning Styles Assessment - **Reading level:** Flesch-Kincaid grade 11.5; Flesch reading ease 55.6 (computed on the stored text: 896 words, 37 sentences, 24.2 words per sentence). #### 👀 Visual learners - Present: one line chart of Historical vs Current workload across 2k–40k posts. - Improve: label the chart's measure and units; add a diagram of three index subsystems; show a bitmap AND operation next to an array intersection. #### 🔊 Aural learners - Present: none. - Improve: a short narrated walkthrough of the rewrite. #### 🗣️ Verbal learners - Present: a plain explanation of bitmaps with the 20,000-post example. - Improve: define inverted index, BM25, and parent map where they first appear. #### 🧩 Kinesthetic learners - Present: none; no code or harness. - Improve: a runnable PHP snippet comparing `array_intersect` with a bitwise combination on generated IDs. #### 🧠 Logical learners - Present: a three-step method; before/after numbers at two dataset sizes. - Improve: a table of all five dataset sizes with p50 and p95, plus the benchmark environment. #### 🧑‍🤝‍🧑 Social learners - Present: an X article open to replies. - Improve: invite other faceted search maintainers to run the same benchmark. #### 🧘‍♀️ Solitary learners - Present: self-contained; about 3.9 minutes at 230 words per minute. - Improve: a checklist for applying "research, then benchmark" to one slow plugin query. #### 🌿 Naturalistic learners - Present: none. - Improve: frame the gain as less server work per request, and so less energy per page load. ## 📊 Accessibility Assessment #### Screen reader semantics - Present: three H2 sections and one real list (the three subsystems). - Improve: "Win 1," "Win #2," and "Win #3" are bold paragraphs, not headings, so screen reader users can't jump to them (WCAG 1.3.1). #### Keyboard navigation - Present: platform-controlled (X). - Improve: not tested. #### Contrast and readability - Present: grade 11.5 reading level; 24.2 words per sentence. - Improve: shorter sentences in the research section; the chart's light gridlines and axis labels weren't measured for contrast (WCAG 1.4.11 applies to the plotted lines). #### Text resizing and reflow - Present: platform-controlled. - Improve: not tested. #### Media and captions - Present: one chart image. - Improve: its alt text only repeats the title. Describe the two series and their end values, or give the data as text (WCAG 1.1.1). The two series differ by color (orange and blue) with a legend; direct labels would help readers who can't distinguish them (WCAG 1.4.1). #### Forms and interactions - Present: none. - Improve: Not applicable to this source. #### Standards reference - Gaps found: 1.1.1 (chart alt text), 1.3.1 (bold pseudo-headings). To check: 1.4.1, 1.4.11. ## 🌍 DEIB Assessment #### Cultural breadth - Present: British spellings ("optimisations," "colour," "orientated") in a global developer venue. - Improve: none needed. #### Inclusive language - Present: gender-neutral throughout. - Improve: "the Amazons of the world" and "elephant in the room" are idioms that translate poorly; name the systems instead (Lucene, Elasticsearch). #### Neurodiversity support - Present: numbered steps and named "wins" give a predictable structure. - Improve: real sub-headings and a summary of the numbers at the top. #### Representation - Present: first-person team account; no other people named. - Improve: credit team members or the research sources used. #### Psychological safety - Present: candid about small early gains ("nothing out of this world"). - Improve: none needed. ## ✅ Logical Validity #### Strong reasoning - **Mechanism before result.** The post explains why bitmaps help (compact representation, cheaper combination) before claiming a speedup, so the claim has a causal account. - **Constraint-aware selection.** Rejecting Roaring because hosts lack it, and choosing "the fastest options that could work in restricted environments," ties design to the real deployment target. - **Arithmetic holds.** 102 / 23.2 = 4.4 and 1,440 / 50.7 = 28.4, matching the stated ratios. - **Caveat stated.** Numbers are labeled uncached, and the post notes caching changes repeat-hit results. #### Fallacies - No formal fallacies found. Three weaknesses: - **Unreproducible evidence.** Location: the data section ("could have taken 1.44 seconds vs new: 50.7 ms"). Weakness: no environment, dataset, method, or variance. "Could have taken" also hedges whether the baseline was measured. Fix: publish the harness and a results table. - **Headline rounding and conflicting figures.** Location: title "30x" vs text "28.4x"; chart values vs text values; January's "10x." Weakness: three numbers for one change with no reconciliation. Fix: state which measurement each figure comes from. - **Unexamined AI method claim.** Location: "we built a suite of benchmarks (using AI of course)." Weakness: AI-written benchmarks can measure the wrong thing, and the post doesn't say how they were checked. The closing recommendation generalizes one team's result to all plugin optimization. Fix: describe the review step, and scope the advice to similar data-structure problems. #### Tone of voice - Informal, enthusiastic, first-person ("the gains get even wilder," "cheesy, but true"), with product-marketing framing around a technical account. ## 🛡️ DARVO Analysis - Not applicable to this source — a technical write-up with no accusation, dispute, or conflict. ## 🎯 Conclusion - **Bottom line:** Search & Filter replaced PHP array set operations with bitmap-based, filter-type-specific indexes found through breadth-first AI research and chosen by benchmarks. The technique and the method are worth borrowing. The headline numbers aren't reproducible from what's published and disagree with the chart and January's release notes. - **Next steps for readers:** - If your plugin combines large ID sets, prototype a bitset combination in PHP and benchmark it against `array_intersect`. - When you use AI for performance work, widen options with research first, then let written benchmarks decide. - Before citing "30x," ask for the benchmark method. ## 🔄 Refactor Appendix — How we made our plugin 30x faster > Density: 2–5 Literature atoms (H3), 0–3 Permanent candidates (H4). > Defaults: 3 Literature, 1 Permanent. Resonance filter: if you can't > articulate in one line why a claim matters, drop it. > > Konik claim-header rules: present tense, declarative, ≤10 words when > possible, searchable, unique, mirrors annotation focus if present. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Atoms (H3) #### [H1] Bitmaps represent post ID sets more compactly than arrays > [!quote] ID:rw_01m2bfmw1wpaap9w8wsgt6s946 > represent large sets of post IDs much more efficiently than storing each ID individually. **Context & use**: - "Win 1 - Switching to Bitmaps." - The data-structure change behind the speedup; applies to any plugin combining large ID sets. **Tags**: #xref **Highlight**: [view highlight](https://read.readwise.io/read/01m2bfmw1wpaap9w8wsgt6s946) <!-- atomicity: 1 idea ✓ --> #### [H2] WordPress hosts can't run Roaring Bitmaps, so PHP approximates > [!quote] ID:rw_01m2bfmyd5eqwq8d29jjn791hw > Unfortunately Roaring Bitmaps are not available on most WordPress hosts. **Context & use**: - "Win 1" section. Consistent with PECL having no `roaring` package and Packagist's largest match at 39 downloads. - Hosting constraints decide which algorithms a plugin can ship. **Tags**: #xref **Highlight**: [view highlight](https://read.readwise.io/read/01m2bfmyd5eqwq8d29jjn791hw) <!-- atomicity: 1 idea ✓ --> #### [H3] Benchmarks pick the fastest option that runs on restricted hosts > [!quote] ID:rw_01m2bfn1htktnhczzdfga97308 > we went with the fastest options that could work in restricted environments. **Context & use**: - "3 - Choosing which research results to go with." - The selection rule: speed, filtered by what deploys. **Tags**: #xref #articleseed **Highlight**: [view highlight](https://read.readwise.io/read/01m2bfn1htktnhczzdfga97308) <!-- atomicity: 1 idea ✓ --> #### [H4] Uncached 40k-post queries drop from 1.44 s to 50.7 ms > [!quote] ID:rw_01m2bfn93xc61w8dqhfm8mmj36 > the previous version could have taken 1.44 seconds vs new: 50.7 ms **Context & use**: - Data section; 28.4x, the source of the rounded "30x" title. - Cite only with the caveat that the benchmark method is unpublished. **Tags**: #xref #share **Highlight**: [view highlight](https://read.readwise.io/read/01m2bfn93xc61w8dqhfm8mmj36) <!-- atomicity: 1 idea ✓ --> ### 🧠 Permanent Candidates (H4) #### [P1] Widen the option set before letting an agent optimize **Claim** (one sentence, Courtney's voice): An optimization loop can only improve among options it already knows, so research that widens the option set has to come first. **Evidence pointer**: [[#H3]] > [!quote] ID:rw_01m2bfn5pjxfxn1pt8ck3600t0 > they usually use their own baked in knowledge **Scope & conditions**: holds when the best solution sits outside the obvious design (here, bitmaps instead of the usual post ID index); matters less for tuning a design that's already right, where loop-style micro-optimization is enough. **Links**: - [[Exploration vs exploitation]] — because extends (research explores; goal loops exploit) - [[Credit routing for agent work]] — because applies-in deciding when to spend on deep research vs iteration <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - [F1] Ask Search & Filter for the benchmark environment and dataset behind 4.4x and 28.4x — https://twitter.com/rossmorsali/status/2092655210338361521 #fleeting #inbox - [F2] Which "alternative compression techniques" make PHP bitmaps work, and how are they stored in MySQL? #fleeting #inbox - [F3] Could a bitset approach speed up large tax_query combinations in my own plugins? #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Prototype a PHP bitset vs array_intersect benchmark on 40k generated post IDs — test the article's core mechanism #things - [ ] Watch searchandfilter.com for the benchmark method behind the 28.4x claim — reconcile with 3.2.0's 10x #things <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Tweets/How We Made Our Plugin 30x Faster.md|How We Made Our Plugin 30x Faster by Ross Morsali]] ## Highlights - The first big win was finding that large-scale systems often use Roaring Bitmaps, which use some clever compression and bitmap-based structures to represent large sets of post IDs much more efficiently than storing each ID individually. ([View Highlight](https://read.readwise.io/read/01m2bfmw1wpaap9w8wsgt6s946)) - Note: Core technique: represent filter result sets as compressed bitmaps instead of PHP arrays of post IDs. - Tags: atom/literature/h1 - Unfortunately Roaring Bitmaps are not available on most WordPress hosts. But we can still use bitmaps in PHP with some alternative compression techniques, which allowed us to achieve something similar in spirit. ([View Highlight](https://read.readwise.io/read/01m2bfmyd5eqwq8d29jjn791hw)) - Note: Constraint-driven design: the plugin can't rely on a native extension, so it approximates the technique in userland PHP. The compression method isn't named. - Tags: atom/literature/h2 - For each plausible option we built a suite of benchmarks (using AI of course), pitting the ideas against each other across different PHP versions and server configurations, and then we went with the fastest options that could work in restricted environments. ([View Highlight](https://read.readwise.io/read/01m2bfn1htktnhczzdfga97308)) - Note: Selection method: benchmark every candidate across PHP versions and server configs, then pick the fastest that runs in restricted hosting. The suite isn't published. - Tags: atom/literature/h1 - Self-improving agent loops are good, but they usually use their own baked in knowledge and I really like deep research mode because it gets you more options and things to try. ([View Highlight](https://read.readwise.io/read/01m2bfn5pjxfxn1pt8ck3600t0)) - Note: Agent loops optimize within the model's existing knowledge (local search); deep research widens the candidate set first (breadth), then benchmarks choose. - Tags: atom/zettel/z1 - A site with 40,000 posts, the previous version could have taken 1.44 seconds vs new: 50.7 ms... ([View Highlight](https://read.readwise.io/read/01m2bfn93xc61w8dqhfm8mmj36)) - Note: 1,440 / 50.7 = 28.4x, uncached. The title's "30x" rounds this up; "could have taken" and no stated hardware, PHP version, or filter set make it unreproducible as written. - Tags: atom/literature/h2