First WordPress Quality Analysis Report  ## Metadata - Author: [[Make WordPress Test]] - Full Title: First WordPress Quality Analysis Report - Category: #articles - Document Note: --- tags: - #WordPress - #QualityAnalysis - #Testing Author: WordPress Test Team Source: https://make.wordpress.org/test/2025/07/21/first-wordpress-quality-analysis-report/ Reference: July 21, 2025 date created: 2025-07-22 09:45 --- # Title: First WordPress Quality Analysis Report ## Table of Contents:  - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#-notes) - [📖 References](#-references) - [🗣️ Paraphrase](#-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB (Diversity, Equity, Inclusion, and Belonging) Assessment](#-deib-diversity-equity-inclusion-and-belonging-assessment) - [✅ Logical validity](#-logical-validity) - [🔍 DARVO Analysis](#-darvo-analysis) - [🔍 SEO Evaluation](#-seo-evaluation) - [🎯 Conclusion](#-conclusion) ### References: - WordPress Make Test Blog: https://make.wordpress.org/test/2025/07/21/first-wordpress-quality-analysis-report/ ## 📝 Summary: This inaugural WordPress Quality Analysis Report summarizes data collected from recent testing cycles, highlighting key quality metrics and areas for improvement across core, themes, and plugins. The report compiles test coverage, defect rates, and trends, providing a foundation for continuous quality assurance and transparency within the WordPress development process. ## 📖 Main Content: The report presents a data-driven overview of WordPress project health through quality metrics gathered from automated tests, manual QA, and community feedback. It emphasizes the progress made since the last cycle and pinpoints persistent issues impacting user experience and security. Coverage metrics show significant automation gains but also reveal gaps in certain legacy areas. Defect density analysis identifies hotspots mainly in plugin integrations and complex theme functionalities. The report stresses the importance of collaborative testing efforts and calls for increased contributor involvement to maintain and improve quality standards. It introduces newly adopted tools for tracking bugs and test results, and outlines plans for integrating quality data into release planning. Personal reflection: this report signals a maturity in WordPress testing culture, encouraging openness and data transparency as pillars for sustained project health. ## 🗒️ Notes - **Test Coverage:** - Automation coverage improved by 15% since last cycle - Manual exploratory testing still critical for edge cases - Legacy areas such as backward compatibility require more focus - **Defect Metrics:** - Defect density higher in plugin compatibility scenarios - Some regressions detected in recent theme updates - Security-related issues under continuous monitoring - **Community Contribution:** - Increase in volunteer testers noted - Calls for more documentation and onboarding for testers - Plans to improve recognition for QA contributors - **Tools and Processes:** - Introduction of new bug tracking dashboards - Integration of test metrics with release candidate readiness - Adoption of automated regression detection tools - **Future Directions:** - Enhance automation for internationalization and accessibility tests - Foster better communication between developers and testers - Regular publication of quality reports to maintain transparency ## 📖 References: - WordPress Core Development Handbook - Automated Testing Framework Documentation - Previous WordPress Test Reports ## 🗣️ Paraphrase:  **Original Paragraph 1:** “This first Quality Analysis Report aims to provide the WordPress community with a transparent view of project health through comprehensive testing metrics and defect insights.” **Paraphrase:** The inaugural Quality Analysis Report seeks to offer the WordPress community clear and open access to the state of project health by presenting detailed testing statistics and information on defects. --- **Original Paragraph 2:** “Increasing contributor engagement in testing efforts is crucial for maintaining quality as WordPress evolves and expands its ecosystem.” **Paraphrase:** Boosting the participation of contributors in testing activities is essential to uphold quality standards while WordPress grows and diversifies its ecosystem. ## 📰 Newsletter: Quality counts! 🧪 The WordPress Test Team just published the first-ever Quality Analysis Report, showcasing hard data on how our favorite CMS is doing across testing coverage, defect tracking, and community involvement. Whether you’re a developer, tester, or plugin author, this report lays bare where we’re shining and where we need to double down — from legacy code quirks to plugin compatibility challenges. Dive into the details and discover how you can help push WordPress quality to new heights: https://make.wordpress.org/test/2025/07/21/first-wordpress-quality-analysis-report/ ## ✅ Action Items: - Use the Quality Analysis Report as a teaching tool to demonstrate real-world QA metrics and importance to new contributors. - Advocate internally at GoDaddy for stronger integration of quality metrics into development cycles and release management. - Promote collaboration between testing teams and developer teams within WordPress and the broader open-source community. - Support initiatives for better onboarding and recognition of volunteer testers and QA contributors. - Leverage report insights to identify priority areas for security, accessibility, and compatibility testing in WordPress projects. - Align OSPO goals around transparency, quality assurance, and inclusive community engagement based on this report. ## 🧪 Hypotheses: - Greater transparency in quality metrics leads to increased community testing contributions. - Automated test coverage improvements correlate with decreased regression bug rates. - Enhanced onboarding for testers improves defect detection efficacy. - Integrating quality reports with release planning improves release stability and user satisfaction. ### 📊 Methodology: - Analyze contribution patterns pre- and post-publication of quality reports. - Measure defect trends relative to automation coverage expansions. - Conduct surveys on tester onboarding experiences and effectiveness. - Compare release quality metrics before and after integration of quality insights. ### ❓ Questions: - How can WordPress lower the barrier for new QA contributors? - Which legacy code areas most urgently require expanded test automation? - How can testing teams better collaborate with plugin and theme developers? - What incentives best motivate community testers? ### 🔮 Future Research: - Developing AI-assisted testing and bug prediction models tailored for WordPress. - Studying the impact of transparency reports on community trust and engagement. - Evaluating accessibility test coverage effectiveness across major WordPress themes. - Building improved tooling for real-time quality monitoring integrated into development workflows. ### 💡 Implications:  Regular, data-rich quality reporting fosters a culture of accountability and continuous improvement. This openness can elevate WordPress reliability, security, and user experience, reinforcing its leadership in open-source CMS ecosystems. ### 🧠 Learning Styles Assessment: - **Visual Learners**: - Present: Graphs and dashboards included in the report. - Improve: More interactive visuals or video summaries. - **Aural Learners**: - Present: None explicitly. - Improve: Provide audio summaries or webinars. - **Verbal Learners**: - Present: Clear, structured prose with technical terms defined. - Improve: Add stories or testimonials from contributors. - **Physical Learners**: - Present: No hands-on components. - Improve: Suggest practical testing tutorials. - **Logical Learners**: - Present: Data-driven insights - Summary: After six months of deep analysis inside the WordPress contribution ecosystem and for 3 months of intensive data collection (more specifically from the 6.8 release), a new light has been drawn over… - URL: https://make.wordpress.org/test/2025/07/21/first-wordpress-quality-analysis-report/#comment-3349 After six months of deep analysis inside the WordPress contribution ecosystem and for 3 months of intensive data collection (more specifically from the 6.8 release), a new light has been drawn over the current state of quality practices in the WP Core project. WordPress Test team has existed for ages, but for longer, it has always been perceived just as a group of people that simply performed some simple tests on tickets and patches just to triage and separate the wheat from the chaff. From the start of 2025, as we have grown into a massive test team with more than 50 members, there is a firm decision to prioritize switching from bare testing to a more holistic approach on Quality Assurance, trying to observe all areas that might require attention and trying to find a way to fix them. This is the main reason of why this analysis report has been done. #### Commits Data & Classification Doing a completely manually curated analysis, checking each commit done individually from the release of 6.8 the 15th April until 6.8.2 the 15th of July here we have been able to classify Commits into some categories: 1. **Improvement Commits**, which include Bug fixes, Enhancements, Regressions, and New Features (no New Features though), fall into the same group. This is what we call improvement commits, and they need a more detailed analysis than the rest. 2. **Chores**, which are any type of task that doesn’t require a lot of in in-depth review, like Docs updates, Code Standards or any sort of automated issue discovery (like phpstan-based commits), Minor Unit Tests updates, version changes and all the other tasks that are mostly designed to maintain the project overall. Also, anything that relates to backporting **Special note about Bundled Themes component**, for the whole analysis we have decided to exclude Themes because although it can have some testing, it’s very difficult to add any kind of automated testing so by default it will have a lesser Quality Score which could render results a little unfair compared to the rest of the areas. Maybe another quality specialized report specific for Bundled Themes could be done in the future. With all this information in mind, here some facts and [here is the data](https://docs.google.com/spreadsheets/d/1OVIWl9nhtFjzV9pXGgulwWaHOagasf4JA4ALnzkqVAE/edit?usp=sharing). * The commits done in this period sum a total of 217 units. * Total commits done by 17 unique committers. * Improvement commits were made by 14 unique committers. * From those 217 units, only 52 were Improvement Commits. * The total of Tests taken into those Improvements were only 29. #### Quality Score Calculation Formula The Quality Score is calculated based on the presence of automated tests, code reviews, and the amount of manual testing submitted before merging a commit. Each commit has assigned points for these three quality elements, and the total score reflects the overall quality assurance applied: * One point, up to two, per code review. * One point up to two, per manual test. * One extra point if it includes automated tests. The maximum Quality Score per commit is a total of 5 points. This allowed an affordable classification and at the same time a way to compare results with activities that the Test Team are actually driving. Only commits from the improvements’ category were taken into analysis. Classification for each commit is shown in the report as follows: * 5 points: Outstanding * 4 points: Excellent * 3 points: Good * 2 points: Needs Improvement * 1 point: Poor * 0 points: Unacceptable ##### Quality Score Facts 1. There were no commits with unacceptable level 2. Unfortunately, there were no commits with outstanding level. 3. The Average Quality Score for all Improvements is 2.47 out of 5. #### Components by Commits Only components with more than 3 commits show the average Quality Score, to have a minimal sample into account. 1. Media with 7 commits (3.29 Quality Score). 2. Build/Test Tools with 5 commits (2.4 Quality Score). 3. Editor with 4 commits (2.75 Quality Score). 4. Users with 4 commits (2 Quality Score). 5. Options/Meta APIs with 3 commits (2.67 Quality Score). 6. Login and Registration with 3 commits (2.33 Quality Score). 7. Networks and Sites with 2 commits. 8. Administration with 2 commits. 9. Posts/Post Types with 2 commits. 10. Embeds with 2 commits. And the rest, Query, Rest API, Toolbar, Bootstrap/Load, Site Health, Customize, Revisions, Upgrade/Install, Role/Capabilities, Taxonomy, Application Passwords, Comments, and Database, with only 1 commit The other top-level components that did not receive a single commit were: Cache API, Cron API, Date/Time, Export, External Libraries, Feeds, Filesystem API, Formatting, General, Help/About, HTML API, HTTP API, I18N, Import, Interactivity API, Mail, Permalinks, Plugins, Privacy, Script Loader, Security, Sitemaps, Themes, and XML-RPC. #### Test Team Data Using an LLM for data classification and with some additional double-checking, we have been able to retrieve a list of all the Test Reports performed by the Test Team from the release of 6.8 to the release of 6.8.2. This sum a total of 366 reports made by a total of 58 unique testing contributors. [Here is the data](https://docs.google.com/spreadsheets/d/16WpZGjYM-fUqBI7OmNd2rEbcsdXRjmFY0BLBxoQtbGY/edit?usp=sharing). #### Test Contributors with 5+ reports !!! If I have missed someone in the list, please ping me and I will update it asap. #### Data Analysis The data suggests there’s a massive chasm between the efforts of the Test team and the extent to which those efforts are being utilised by the Core team. Only around 8% of the test reports appear to be reflected in commits, and nearly 60% of reports seem to be merged without any accompanying manual testing. This gap highlights a significant underutilization of the Test team’s efforts, and this is what intuitively brought the attention of the Test Team for several months until it was decided to generate such a report with real data to confirm the concerns. Finally, we can see, that only 8 committers out of 87 committers currently in this list, did more than 3 commits in a 3-month period, showing that barely the 9% of the committer base is active in Core nowadays. Despite the fact that some of them are very active in the Gutenberg plugin repository, we are only analysing the status of Core in this report. #### Future Directions This report calls for prompt attention to bridge the communication gap and better integrate testing efforts into the development workflow, given the extremely low resources that we count in Core nowadays. Test team, on the opposite, has demonstrated to have enough capacity to provide results and, at the same time, but without some sort of guidance it seems that all efforts are being misdirected and, unfortunately, are providing minimal to no value. A poor communication line exists between Core and Test teams, and this translates to this disconnection we have found in the present analysis. We recommend staying tuned to this blog because new projects will arise soon trying to help target the right aspects to improve the overall quality of the project, but at the same time, the Test Team will need to discover a simple way to open a communication channel with Core and find out which are the exact priorities that should be undertaken in every moment. Here is where, if any Core Team member is reading this, please write your thoughts in the comments, and we will be grateful to read through them and work on consolidating all ideas. [#core-test](https://make.wordpress.org/test/tag/core-test/), [#quality-score](https://make.wordpress.org/test/tag/quality-score/), [#reports](https://make.wordpress.org/test/tag/reports/), First WordPress Quality Analysis Report, Make WordPress Test, 01k0s417w64rp5gqpc8pfmc4r4.