# Benchmark-Gated Adoption Lets You Ship Techniques You Don't Fully Understand ## Claim You can safely adopt an unfamiliar optimization technique when a passing benchmark proves it works, even before you fully understand the underlying theory. <!-- atomicity: 1 idea ✓ --> ## Scope & conditions - Holds when the technique reduces to a small, testable unit with a clear, measurable success criterion. - Breaks when behavior is emergent, non-deterministic, or can't be isolated into a benchmark — a passing benchmark on a narrow slice doesn't guarantee correct behavior on the whole system. - Works best paired with LLM-assisted research: the model surfaces candidate techniques from adjacent domains (large-scale search, e-commerce) faster than a solo developer could survey the literature themselves. ## Evidence Search & Filter plugin developer Ross Morsali used Claude's deep-research mode to surface bitmap-packing techniques from large-scale search systems (Elasticsearch, Algolia-adjacent literature) for WordPress faceted search, then built 5-10 benchmarks per candidate theory to validate the approach — without first mastering the compression theory behind it. The technique shipped in a January indexing rewrite, about 3-3.5 months from research to release. > [!quote] > You don't even necessarily really need to understand all the theory behind it. ## Links - [[3. Resources/Permanent Notes/Cap Deep-Research Depth at One or Two Levels]] — because applies-in; the companion discipline from the same source that keeps benchmark-driven exploration from running away. - [[3. Resources/MOCs/AI & the Future of Knowledge Work MOC]] — because applies-in; a concrete pattern for AI-assisted technical work by a solo or small-team developer. ## Tests / falsifiers - Track whether the shipped bitmap-indexing rewrite holds up under production load at scale it wasn't explicitly benchmarked against — a production failure the benchmarks didn't predict would weaken the claim. - Try the same method on a technique with fuzzier success criteria (e.g., a UX or architectural decision rather than a raw performance number) and check whether "benchmark passes" still substitutes for understanding, or whether the method only works for cleanly measurable problems. ## Maintenance - Last tended: 2026-08-16 - Next review: 2026-11-16 ## Source - [[3. Resources/Readwise/Articles/How to Use AI Deep Research to Improve WordPress Plugins]] — Steve Burge interviewing Ross Morsali, The PublishPress Podcast, on the deep-research-to-benchmark workflow behind Search & Filter's indexing rewrite.