Introducing WP-Bench: A WordPress AI Benchmark  How well do language models actually understand WordPress? To answer this, we’re introducing [WP-Bench](https://github.com/WordPress/wp-bench) – the official WordPress AI benchmark. WP-Bench evaluates how well AI models understand WordPress development, from core APIs and coding standards to plugin architecture and security best practices. #### Why WP-Bench Matters WordPress powers over 40% of the web, yet AI models are typically evaluated on general programming tasks. WP-Bench fills this gap by measuring WordPress-specific capabilities. **Understanding today’s models.** Whether you’re building AI-powered plugins or using coding assistants, knowing which models excel at WordPress helps you make better tooling decisions. **Shaping tomorrow’s models.** We want WP-Bench to become a standard benchmark that AI labs use when developing new models. When providers like OpenAI, Anthropic, and Google run pre-release evaluations, we want WordPress performance on their radar – not as an afterthought. This creates incentive to optimize for the millions of developers and site owners who depend on WordPress. **Building an open source leaderboard.** We’re working toward a public leaderboard tracking model performance on WordPress tasks. This will provide transparent results for the community, inform how the WordPress project engages with AI providers, and help developers choose the right tools for their projects. #### How It Works WP-Bench measures AI capabilities across two dimensions: * **Knowledge** – Multiple-choice questions testing WordPress concepts, APIs, hooks, security patterns, and coding standards, with an emphasis on modern additions like Abilities API and Interactivity API. * **Execution** – Code generation tasks graded by a real WordPress runtime with static analysis and runtime assertions. The benchmark uses WordPress itself as the grader, running generated code in a sandboxed environment. This ensures we measure both theoretical understanding and practical ability to produce working, standards-compliant code. #### Current State & Known Limitations WP-Bench is an early release, and we’re being transparent about where it needs work: * **Dataset size** – The current test suite is relatively small. We need more test cases across WordPress APIs and patterns to make this a comprehensive benchmark. * **Version coverage** – The benchmark currently skews toward WordPress 6.9 features like the Abilities API and Interactivity API. This is partly intentional (newer APIs are where models genuinely struggle) but also creates a bias since these features post-date most models’ training data. We need more coverage of established WordPress patterns to create a balanced evaluation. * **Benchmark saturation** – Early testing showed models scoring very high on older WordPress concepts, which means those questions aren’t providing strong signal. The challenge is finding problems that are genuinely difficult, not just new. These limitations are exactly why we’re releasing now rather than waiting. We know that the WordPress community is uniquely positioned to help build a robust, representative benchmark. #### Quick Start | | | | --- | --- | | 123456789 | `# Install``python3` `-``m venv .venv && source .venv``/``bin``/``activate``pip install` `-``e .``/``python``# Start the WordPress runtime``cd runtime && npm install && npm start``# Run the benchmark``cd .. && wp``-``bench run` `-``-``config wp``-``bench.example.yaml` | Configure your model provider API keys in a `.env` file, and results are written to `output/results.json`. The harness supports running multiple models in a single pass for easy comparison. #### Supporting the AI Building Blocks WP-Bench complements the other [AI Building Blocks for WordPress](https://make.wordpress.org/ai/2025/07/17/ai-building-blocks/) by measuring how well AI models work with WordPress. As we build out the Abilities API, MCP Adapter, and other infrastructure, a standardized benchmark helps ensure these tools integrate with the best available models. #### Get Involved WP-Bench needs your help. The benchmark is only as good as its test cases, and the WordPress community has decades of collective knowledge about what makes WordPress development challenging. **Ways to contribute:** * **Add test cases** – Know a tricky WordPress pattern that trips up developers? It probably trips up AI too. We need coverage across more APIs, hooks, and real-world scenarios. * **Run benchmarks** – Test models you’re using and share your findings. * **Improve grading logic** – Help make the evaluation more rigorous. * **Submit results** – Contribute to the public leaderboard. If you work at an AI lab, we’d love to collaborate on integrating WP-Bench into your evaluation pipeline. **Resources:** Our goal is for WP-Bench to become the standard evaluation AI providers use when releasing new models – creating a virtuous cycle where WordPress performance improves with each generation. Join us in [#core-ai](https://wordpress.slack.com/archives/C08TJ8BPULS) to discuss, share results, and help shape the future of AI in WordPress. *Props to [@jason\_the\_adams](https://profiles.wordpress.org/jason_the_adams/) for leading development on WP-Bench.*, Introducing WP-Bench: A WordPress AI Benchmark, WordPress AI, 01kez82qjeydhyzmb1tx3g5ez2.