Introducing WP-Bench: A WordPress AI Benchmark 🔗 **Source:** [Introducing WP-Bench: A WordPress AI Benchmark](https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/) > [!note]+ Document Notes > ```yaml > tags: [WordPress, AI, Benchmark] > author: WordPress AI > source: make.wordpress.org > reference: [https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/] > date created: 2026-01-18 00:00 > > --- > > # 🏷️ Title: Introducing WP-Bench: A WordPress AI Benchmark > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - Include links cited in or relevant to the article. ## 📝 Summary - What happened: The WordPress project introduced WP-Bench, an official benchmark that tests how well AI models understand modern WordPress concepts, APIs, and real-world coding tasks. - Why it matters: It fills a gap where most AI evals target generic programming, not WordPress’s unique architecture, hooks, and security patterns. - Key stakeholders: WordPress core and Core AI teams, plugin/theme developers, AI tool vendors (OpenAI, Anthropic, Google, etc.), and site owners relying on AI-assisted workflows. - Risks and tradeoffs: Early dataset is small, skewed toward WordPress 6.9+ and newer APIs (Abilities, Interactivity), which may bias scores; poorly designed tests could mislead model selection. - Bottom line: WP-Bench aims to become the standard, open benchmark and leaderboard for AI performance on WordPress, and it depends on community-contributed test cases and results. ## 📖 Main Content - The article introduces WP-Bench as “the official WordPress AI benchmark” that evaluates how well language models actually understand WordPress development—from core APIs and coding standards to plugin architecture and security best practices. - It argues that, since WordPress powers over 40% of the web, AI systems that generate or review WordPress code must be specifically evaluated on WordPress tasks, not just generic coding challenges. - WP-Bench measures capabilities along two axes: - Knowledge: multiple-choice questions on WordPress concepts, APIs, hooks, security patterns, and coding standards, with special focus on newer additions like the Abilities API and Interactivity API. - Execution: code generation tasks automatically graded by a real WordPress runtime using static analysis and runtime assertions, so WordPress itself effectively acts as the grader. - The project aims to support: - Developers choosing which AI models and assistants to use for WordPress work. - AI labs, by providing a standard benchmark to run during pre-release evaluation, encouraging them to improve WordPress-specific understanding. - An open, community-visible leaderboard tracking model performance on WordPress tasks. - The article is transparent about current limitations: - The test suite is relatively small and needs broader coverage across APIs and patterns. - It currently leans toward WordPress 6.9-era features (e.g., Abilities API, Interactivity API), which may disadvantage models trained before those features existed. - Early runs show high scores on older patterns, suggesting some questions are too easy and don’t provide strong signal. - WP-Bench is positioned as part of the broader “AI Building Blocks for WordPress,” alongside efforts like the Abilities API and MCP Adapter, giving the ecosystem a standardized way to compare and integrate the best-performing models. - The article closes with a call for community involvement: adding test cases, running benchmarks, improving grading logic, and contributing results to a public leaderboard. It invites collaboration from AI labs and points readers to the #core-ai channel for coordination. ### 🗒️ Notes - WP-Bench is introduced as a targeted benchmark for assessing AI understanding of WordPress development, filling a gap where existing benchmarks focus on general-purpose programming. - Key evaluation dimensions: - Knowledge: multiple-choice questions covering concepts, APIs, hooks, security, and coding standards, especially newer APIs like Abilities and Interactivity. - Execution: generated code is run in a sandboxed WordPress environment with static checks and runtime assertions, giving practical, correctness-oriented feedback. - Importance and goals: - WordPress’s large footprint (40%+ of the web) justifies a dedicated benchmark. - Helps plugin authors, agencies, and product teams choose AI tools backed by measurable WordPress performance. - Intends to become a standard that AI labs include in their evaluation pipelines, making WordPress support a first-order concern. - A planned public leaderboard should offer transparent comparisons between models and inform how WordPress engages with AI vendors. - Current limitations: - Small dataset: needs more breadth across APIs, hooks, and real-world development patterns. - Version bias: current focus on WordPress 6.9-era features helps probe weaknesses but can bias results against older-trained models. - Benchmark saturation: older concepts often yield very high scores, providing weak signal; there is an ongoing need to identify genuinely challenging cases. - Relationship to other initiatives: - WP-Bench “complements the other AI Building Blocks for WordPress” by aligning evaluation with infrastructure like the Abilities API and MCP Adapter. - Serves as a feedback loop to ensure that as core AI tooling evolves, it is tested against realistic, difficult WordPress scenarios. - Community call-to-action: - Add test cases for tricky patterns that often trip up developers and, by extension, AI models. - Run benchmarks with the models you use in production or experimentation and share the findings. - Help improve grading logic to make evaluation more robust and nuanced. - Submit results for inclusion in a public leaderboard and collaborate via the #core-ai channel. - AI lab stakeholders are explicitly invited to integrate WP-Bench into their internal evaluation processes. - Governance and recognition: - The post credits @jason_the_adams for leading WP-Bench development, highlighting specific contributor leadership within the WordPress AI initiative. ## 📚 References - Introducing WP-Bench: A WordPress AI Benchmark – make.wordpress.org (article you’re reading) - WordPress 6.9 release documentation (for Abilities API and Interactivity API details) - WordPress Developer Resources – Code Reference and Plugin Handbook (primary sources for APIs, hooks, and coding standards) ## 🗣️ Paraphrase - Paragraph A paraphrase: WP-Bench is a new official benchmark designed to test how well AI language models understand WordPress, including its APIs, architectural patterns, and security practices. - Paragraph B paraphrase: Because WordPress powers a huge share of the web and most AI evaluations ignore its specifics, WP-Bench aims to guide both developers and AI companies by providing transparent, WordPress-focused performance data through knowledge questions, execution tests, and an open leaderboard. ## 📰 Newsletter The WordPress AI team has launched **WP-Bench**, a dedicated benchmark for evaluating how well language models understand real-world WordPress development. Instead of relying on generic coding tests, WP-Bench uses WordPress itself as the grader, combining multiple-choice knowledge checks with sandboxed code execution. This matters for anyone building AI-assisted plugins, themes, or developer tools. A standardized benchmark helps you choose models that actually know their way around hooks, the Abilities API, and the Interactivity API—rather than just vanilla PHP. You can get involved by contributing tricky test cases, improving grading logic, or submitting your own model runs. Join the conversation in **#core-ai** on Make WordPress Slack and help shape how the ecosystem measures and improves AI support for WordPress. ## ✅ Action Items #### DevRel - Promote WP-Bench in talks, blog posts, and demos; show side-by-side model comparisons on WordPress-specific tasks. - Develop tutorials and workshops on using WP-Bench locally and interpreting results (e.g., for agencies and plugin shops). - Add quickstart examples and sample configs to make running WP-Bench against popular models trivial. - Build a content funnel (posts, videos, office hours) that helps developers move from curiosity to regular WP-Bench usage. - Encourage community feedback loops on confusing tests and share common failure patterns with the Core AI team. #### OSPO - Document contribution policies for WP-Bench test additions and changes, ensuring clear review and acceptance criteria. - Clarify licensing for test suites and evaluation logic so AI labs can safely integrate WP-Bench. - Integrate WP-Bench into open source risk and security review discussions, especially where AI-generated code is involved. - Track WP-Bench as part of strategic metrics on WordPress–AI ecosystem health for executive reporting. #### Community - Maintain a curated list of “good first” test-case contributions to onboard new contributors. - Schedule contributor sprints around expanding WP-Bench coverage (APIs, hooks, security edge cases). - Share real-world “bug stories” where AI-generated WordPress code failed and convert them into benchmark tasks. - For sponsors and agencies, demonstrate impact by publishing how WP-Bench-guided model choices improved quality or speed. #### WordPress - Core: Identify core APIs and patterns under-tested by WP-Bench and file issues or proposals for new test coverage. - Core Performance: Investigate whether WP-Bench can incorporate performance-sensitive tasks (e.g., avoiding slow queries or poor caching patterns). - Core AI: Treat WP-Bench as a primary evaluation tool; align responsible AI guardrails and pilot projects with benchmark coverage. - **Core Program Team:** Include WP-Bench in cross-team planning, ensuring test coverage reflects upcoming core features and changes. - Mobile: Explore whether WP-Bench can be extended or mirrored for mobile-specific WordPress integrations and SDKs. - Playground: Provide pre-configured Playground environments where contributors can design and debug WP-Bench tasks in the browser. - Community: Add WP-Bench contributor sessions to Meetup and WordCamp formats; offer slide decks and organizer kits. - Training: Create Learn.WordPress courses on AI & WordPress, including labs built around WP-Bench tasks. - Marketing: Craft narratives showcasing WordPress as a leader in open AI evaluation for a major web platform. - Support: Use WP-Bench failure examples to enrich support docs with AI-specific caveats and best practices. - Design: Ensure any public leaderboard or dashboard is clear, comparable, and avoids misleading visual communication. - Documentation: Cross-link DevHub/HelpHub pages with related WP-Bench test domains (e.g., hooks, security). - Photos/TV/Openverse: Not central here; optionally produce short explainer videos on WP-Bench and host them on WordPress.tv. - Polyglots: Plan for future internationalization of docs and possibly translated descriptions of test cases. - Accessibility: Review WP-Bench UI (leaderboards, docs) against WCAG; ensure accessible code samples and charts. - Meta: Host the benchmark’s public leaderboard and docs with robust, well-documented APIs for external tooling. - CLI: Consider a wp-cli command to run or summarize WP-Bench results locally. - Hosting: Collaborate on recommended environments for running WP-Bench; offer presets for popular hosts. - Tide: Explore synergy between Tide’s plugin scoring and WP-Bench’s AI evaluation, especially for automated code checks. - Themes/Plugins: Encourage maintainers to validate their AI-assisted workflows against WP-Bench and share findings. #### The WPOCC - Support WP-Bench-related contributor time via microgrants or fellowships, especially for test design and grading improvements. - Fund dedicated documentation and training sprints focused on AI and WP-Bench. - Pilot matched-funding experiments with AI vendors interested in co-developing WordPress-specific evals. ### FAIR - Ensure WP-Bench remains technically independent and open, with reproducible test suites and transparent scoring. - Publish governance documents describing how tests are added, revised, or deprecated, and how conflicts are handled. - Provide reusable “governance artifacts” (charters, contribution policies) other open-source AI benchmarks can adopt. ## 🧪 Hypotheses - Models that perform well on WP-Bench execution tasks will also produce fewer WordPress-related bugs in real-world projects. - Adding more tests around new APIs (Abilities, Interactivity) will widen the performance gap between frontier models and older models. - Involving the WordPress community in authoring test cases leads to more realistic and challenging benchmarks than centrally designed tests alone. - Public leaderboards will influence AI vendors to optimize for WordPress-specific capabilities over time. - WP-Bench scores will correlate with developer satisfaction when using AI coding assistants in WordPress projects. ### 📊 Methodology - Collect WP-Bench scores across multiple models and versions (e.g., GPT variants, Claude, Gemini). - Gather real-world incident data from agencies and plugin authors on AI-generated WordPress code (bugs, security issues). - Run longitudinal studies comparing teams that choose models based on WP-Bench results vs. teams that do not. - Analyze item difficulty and discrimination for each test case to refine and rebalance the benchmark. - Survey developers on perceived usefulness of model recommendations before and after exposure to WP-Bench-based rankings. ### ❓ Questions - Which WordPress APIs and patterns are currently most difficult for AI models to handle correctly? - How often do high WP-Bench scores translate into measurable productivity or quality gains for developers? - What governance model best balances rapid test evolution with benchmark stability? - How can WP-Bench remain relevant as WordPress and AI models both evolve rapidly? - Should WP-Bench eventually incorporate performance, accessibility, and security sub-scores? ### 🔮 Future Research - Expand coverage to multi-site, complex plugin interactions, and advanced security scenarios. - Investigate benchmarks for AI-assisted content creation and site configuration, not just code. - Explore variants of WP-Bench tailored for agencies, product companies, or educators. - Study the impact of WP-Bench-driven competition between AI vendors on the broader open web ecosystem. ### 💡 Implications - Practice: Developers gain a concrete tool to choose safer and more capable AI assistants for WordPress work. - Policy: WordPress can set expectations with AI vendors around WordPress support and transparency of performance. - Security: Better benchmarks could reduce insecure patterns generated by AI for WordPress sites. - Education: WP-Bench tasks can inform teaching materials on both WordPress and responsible AI-assisted development. - Funding: Benchmarks like WP-Bench provide concrete metrics to justify investment in AI–WordPress integration. ### 🧠 Learning Styles Assessment - Reading level estimate: The article reads around upper high school to early college level (roughly Flesch-Kincaid Grade 10–12), with moderate technical jargon. #### 👀 Visual learners - Present: Clear headings and bullet lists; conceptual contrast between knowledge vs execution dimensions. - Improve: Add diagrams of WP-Bench architecture, sample grading flows, and example leaderboard snapshots. #### 🔊 Aural learners - Present: No explicit audio, but text is short and could be easily narrated. - Improve: Provide a 2–5 minute audio summary or podcast-style explainer of WP-Bench. #### 🗣️ Verbal learners - Present: Definitions via context (benchmark, knowledge vs execution tasks); clear explanations. - Improve: Add more analogies (e.g., “WordPress driving test for AI”) and a small glossary for newer APIs. #### 🧩 Kinesthetic learners - Present: Calls to action (add test cases, run benchmarks) offer practical engagement. - Improve: Step-by-step “run WP-Bench locally” and “write your first test case” guides. #### 🧠 Logical learners - Present: Clear structure: problem → solution → methodology → limitations → call to action. - Improve: Include tables of early scores, explicit metrics, and criteria for “good” vs “bad” performance. #### 🧑‍🤝‍🧑 Social learners - Present: Emphasis on community contributions and #core-ai collaboration. - Improve: Offer suggested discussion prompts for meetups or study groups around AI & WordPress. #### 🧘‍♀️ Solitary learners - Present: Self-contained article, easy to read alone and reflect on. - Improve: Add reflection questions (“How do you choose AI tools today?”) and checklists. #### 🌿 Naturalistic learners - Present: No explicit environment or sustainability tie-ins. - Improve: Draw analogies between ecosystems (WordPress + AI) and natural systems, or note energy considerations of model evaluation. ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Proper headings and simple structure typical of Make blogs. - Improve: Ensure all images in any future WP-Bench UI/leaderboard include descriptive alt text and correct language attributes. #### Keyboard navigation - Present: Standard WordPress.org templates generally support keyboard navigation. - Improve: If/when a WP-Bench UI exists, verify focus states, skip links, and logical tab order. #### Contrast and readability - Present: Base WordPress.org styles typically meet contrast guidelines. - Improve: Double-check leaderboard charts or badges for color contrast and font size. #### Text resizing and reflow - Present: Responsive design likely handles zoom and mobile. - Improve: Test at 200%+ zoom, ensure no horizontal scrolling for main content, and honor prefers-reduced-motion. #### Media and captions - Present: Text-only article; no media to caption. - Improve: Provide captions and transcripts for any future explainer videos or demos. #### Forms and interactions - Present: None in the article itself. - Improve: For any submission forms (results, test cases), ensure labeled fields, clear error messages, and accessible controls. #### Standards reference - Map open issues in future WP-Bench UI to WCAG 2.2 AA criteria (e.g., focus appearance, drag-and-drop alternatives, error prevention). ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global infrastructure (WordPress) context; neutral language. - Improve: Explicitly recognize global contributor base and plan multi-lingual docs for WP-Bench. #### Inclusive language - Present: Neutral, technical language; minimal jargon. - Improve: Add more plain-language summaries and avoid assuming deep prior knowledge of newer APIs. #### Neurodiversity support - Present: Chunked content with headings and bullets. - Improve: Maintain consistent patterns across future WP-Bench docs, use icon+text labels, and avoid dense, unbroken paragraphs. #### Representation - Present: Focuses on roles (developers, AI labs) rather than individuals. - Improve: When adding visuals or stories, highlight diverse contributors and use inclusive imagery. #### Psychological safety - Present: Encouraging, non-blaming tone; invites collaboration. - Improve: For community spaces around WP-Bench, surface community guidelines and moderation expectations. ### ✅ Logical Validity #### Strong reasoning - The argument from WordPress’s share of the web to the need for a specialized benchmark is a strong causal/needs-based argument. - The two-dimensional design (knowledge + execution) logically supports the claim that WP-Bench measures both understanding and practical capability. - Transparent discussion of limitations strengthens inductive credibility and avoids overclaiming. #### Fallacies - No clear logical fallacies stand out; the article avoids sweeping claims like “WP-Bench solves AI and WordPress” and acknowledges constraints. #### Tone of voice - Tone is pragmatic, collaborative, and forward-looking—technical but accessible. - It balances enthusiasm (“official WordPress AI benchmark”) with realism about current dataset size and version bias. ### 🛡️ DARVO Analysis - Deny: No denial or minimization of issues; limitations are directly acknowledged. - Attack: No attacks on critics or alternative approaches; the post is invitational rather than adversarial. - Reverse victim/offender: No attempt to portray WordPress or AI vendors as victims; instead, it frames everyone as collaborators solving a shared problem. ## 🎯 Conclusion WP-Bench is an early but important step toward systematically measuring how well AI models understand and generate WordPress code. By combining knowledge questions with runtime-graded execution tasks, it offers a concrete way for developers and AI vendors to evaluate and improve WordPress-specific capabilities. The benchmark will only become truly robust if the community contributes test cases, grading improvements, and model results—so the practical next step is to get involved via #core-ai, run WP-Bench against the models you use, and help iteratively refine the test suite. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad - Purpose: Stage Konik CEE units that become per-source Literature notes. - Split level: H3. - Include verbatim quotes under 25 words and a [view highlight] link. ### [H1] 5–10-word present-tense claim > [!quote] [Introducing WP-Bench: A WordPress AI Benchmark](https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/) by [[WordPress AI]] via [[make.wordpress.org]]. Published 2026-01-14. Accessed 2026-01-18. > “WP-Bench evaluates how well AI models understand WordPress development.” Two lines: context and use. Cite data if relevant. [view highlight](https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/#H1) #xref #articleseed ID:rw_[MISSING] ### [H2] 5–10-word present-tense claim > [!quote] [Introducing WP-Bench: A WordPress AI Benchmark](https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/) > “WP-Bench measures both knowledge and execution.” Two lines: context and use. [view highlight](https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/#H2) #xref ID:rw_[MISSING] --- ### 🔖 Atomic Split Pad (Zettel Candidates) - Purpose: Promote reusable claims into Atomic notes. - Split level: H4. - Each item: one idea, present tense, specific, testable. #### [Z1] WP-Bench evaluates AI on WordPress code - Scope: Language models used for WordPress development. - Pointer: Main WP-Bench introduction section. - Links: [[WordPress-AI]] [[AI-Evaluation]] #### [Z2] WP-Bench uses knowledge and execution tasks - Scope: Benchmark design for AI on WordPress. - Pointer: “How It Works” section. - Links: [[AI-Benchmarks]] [[WordPress-Developer-Experience]] --- ### ✏️ Fleeting Queue (Extract by Selection) - Purpose: Capture quick actions or sparks for Fleeting notes. - Extraction: Select one line → “Refactor: Extract selection to new note (content only).” - Keep each line short and self-contained. - [F1] Draft a short guide: “How to run WP-Bench locally” — context/https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/ #fleeting #inbox - [F2] Propose a WordCamp session on WP-Bench and AI code assistants — context/https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/ #fleeting #inbox - [F3] Identify three tricky WordPress patterns and turn them into WP-Bench test cases — context/https://make.wordpress.org/ai/2026/01/14/introducing-wp-bench-a-wordpress-ai-benchmark/ #fleeting #inbox --- <!-- deep-read:sections end -->