RFC: Automated Hosting Test Runner Service  ## Metadata - Author: [[Make WordPress Hosting]] - Full Title: RFC: Automated Hosting Test Runner Service - Category: #articles - Document Note: --- tags: - #wordpress - #hosting - #automation Author: Make WordPress Hosting Source: Make WordPress (article) Reference: https://make.wordpress.org/hosting/2025/06/25/rfc-automated-hosting-test-runner-service/ date created: 2025-08-23 00:00 --- # Title: RFC — Automated Hosting Test Runner Service ## Table of Contents: - [📚 References](#-references) - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#-notes) - [📖 References](#-references-1) - [🗣️ Paraphrase](#-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-deib-diversity-equity-inclusion-and-belonging-assessment) - [✅ Logical validity](#-logical-validity) - [🔍 SEO Evaluation](#-seo-evaluation) - [🎯 Conclusion](#-conclusion) ### References: - WordPress.org Hosting Team RFC: Automated Hosting Test Runner Service — https://make.wordpress.org/hosting/2025/06/25/rfc-automated-hosting-test-runner-service/ ## 📝 Summary: The RFC proposes an automated service to run WordPress Hosting Team’s test suite on hosts’ environments with minimal setup. Today, hosts and contributors must clone/configure the suite, prep environments, manage dependencies, and interpret results—creating a participation barrier. The service would accept DB and WordPress.org credentials, return a public SSH key to install, connect to the host, stage the suite, execute tests in a standardized way, and post results back to make.wordpress.org. Security considerations include ephemeral, session-scoped key pairs, IP allowlisting, and avoiding long-term credential storage. Benefits include lower barriers, consistent runs, scalable onboarding, and aggregated insights. The team requests feedback on approach, privacy/security, and early-access participation. ## 📖 Main Content: - Problem: - Running hosting tests requires manual setup, environment prep, and domain knowledge of dependencies and edge cases. - This complexity disproportionately impacts smaller hosts and non-technical contributors. - Proposed Solution and Workflow: - User provides minimal inputs: DB credentials and WordPress.org username plus app password/token. - Service returns a public SSH key; host adds it to their environment. - Service connects via that key, prepares and transfers required files, executes tests, and sends results to make.wordpress.org. - Emphasis on standardization and automation to reduce cognitive and operational load. - Security: - Use ephemeral, per-session SSH key pairs. - Prefer not to store credentials; if unavoidable, store securely and short-lived. - Restrict connections to specific, short-lived IPs. - Benefits: - Lower barrier to entry and reduced operational toil. - Standardized environment for more comparable/consistent runs. - Improved onboarding and broader participation. - Aggregated analytics for trend identification and prioritizing fixes. - Request for Feedback: - Validate whether this approach adequately lowers barriers. - Surface privacy/security concerns early. - Recruit hosts for early-access testing. ### 🗒️ Notes - Problem - Manual cloning/configuration is a bottleneck. - Environment setup and dependency management are error-prone. - Results interpretation is non-trivial without context. - Barrier especially affects small hosts and non-technical contributors. - Solution - Inputs: DB creds; WordPress.org user + app password/token. - Output: Public SSH key to install; triggers secure, automated run. - Steps: connect → prepare suite → move files → execute → ship results. - Security - No long-term credential storage. - Ephemeral per-session keys. - IP allowlisting and short-lived connections. - Benefits - Accessibility: Easier for new/small hosts. - Consistency: Standardized/validated environment. - Scale: Faster onboarding; more runs. - Insights: Aggregate and analyze results for trends. - Open Questions - Is the minimal-inputs model viable across diverse host setups? - How to handle atypical environments (custom PHP modules, read-only FS)? - What telemetry is privacy-preserving yet useful? ## 📖 References: - Automated Hosting Test Runner Service (RFC): https://make.wordpress.org/hosting/2025/06/25/rfc-automated-hosting-test-runner-service/ ## 🗣️ Paraphrase: - Original idea: “The Hosting Team has received feedback that the current process for running hosting tests is cumbersome… This RFC proposes the development of a service that simplifies and automates…” - Paraphrase: Community feedback indicates hosting tests are overly complex to set up and execute. This RFC suggests creating a managed service that automates preparation and execution to streamline participation. - Original workflow: Provide DB credentials and WordPress.org token; receive a public SSH key; service connects, stages suite, runs tests, and reports results. - Paraphrase: Users would submit minimal credentials, install a one-time SSH key, and the service would securely log in, deploy the testing tools, run them, and automatically publish the outcomes. ## 📰 Newsletter: This week in WordPress: learning never sleeps, and neither do our build pipelines. The Hosting Team proposes an Automated Hosting Test Runner Service to make running official tests a one-click affair—no more wrestling with dependencies or arcane configs. The service would accept minimal credentials, deliver an SSH key, connect to your environment, execute a standardized suite, and publish results back to the project. Why it matters: lower barriers for smaller hosts, consistency across runs, and better insights from aggregated data. For learners: brush up on PHP/JS fundamentals with hands-on testing workflows, and explore CI-friendly hosting validation. Community: early adopters are invited to kick the tires and share security/privacy feedback. New tools on our radar: ephemeral keys, IP allowlisting, and short-lived sessions are first-class citizens here. Read the RFC: Automated Hosting Test Runner Service on Make WordPress Hosting. Contribute your thoughts: comment on the post with edge cases, telemetry needs, and onboarding ideas. ## ✅ Action Items: Prompt you can reuse for future articles, aligned to your role and goals: - “Analyze how this article advances open-source advocacy in WordPress. Map its proposals to contributor onboarding, education pathways (PHP/JS/Git), and sustainability (funding/operations). Identify concrete opportunities for a Developer Advocate at GoDaddy to: - Facilitate contributor engagement (workshops, office hours, docs PRs). - Design curriculum and lab exercises for new contributors and hosts. - Advocate OSPO-aligned governance: transparency, metrics, security-by-design. - Propose cross-org collaboration models (e.g., early-access cohorts, data sharing). - Define inclusive practices and measurement for DEIB. - Outline how this work strengthens a future OSPO charter (risk, compliance, community health). - Produce an adoption plan (KPIs, success metrics, rollout stages) for ecosystem impact.” Alignment for this RFC: - Advocacy: lowers barriers, invites broader host participation. - Contributor engagement: standardization enables repeatable workshops and labs. - Education: great case study for CI/CD, security practices, and reproducible testing. - OSPO trajectory: governance over credentials, data minimization, and transparent reporting. ## 🧪 Hypotheses: - H1: Automating test setup will increase the number of participating hosts by at least 50% within six months. - H2: Standardized runs will reduce variance in test outcomes across hosts by ≥30%. - H3: Aggregated insights will surface top 3 recurring host configuration issues within the first quarter. - H4: Ephemeral credentials and IP allowlisting will reduce reported security incidents to near-zero. - H5: Lower friction will increase non-technical contribu - Summary: Summary The Hosting Team has received feedback that the current process for running hosting tests is cumbersome and requires significant setup beyond simply providing hosting space. This RFC propos… - URL: https://make.wordpress.org/hosting/2025/06/25/rfc-automated-hosting-test-runner-service/?__readwiseLocation= #### Summary The Hosting Team has received feedback that the current process for running hosting tests is cumbersome and requires significant setup beyond simply providing hosting space. This RFC proposes the development of a service that simplifies and automates the process of preparing and executing the hosting tests. #### Problem Currently, contributors and hosting providers who want to run the hosting tests must: * Manually clone and configure the test suite * Set up the proper environment * Understand all dependencies and edge cases * Execute and interpret the results themselves This level of complexity creates a barrier for participation, especially for smaller hosts or non-technical contributors. #### Proposed Solution Create an automated service that prepares and runs the hosting test suite on behalf of the user. The workflow would be as follows: 1. **User Input:** * Database credentials (host, name, user, password) * WordPress.org username and app password/token 2. **Response:** * The service returns a public SSH key to be added to the user’s hosting environment 3. **Execution:** * Once the public key is in place, the service connects to the hosting environment * Prepares the test suite * Moves all the required files to the host * Executes the tests * Sends the result to make.wordpress.org #### Security Considerations * All credentials must be securely transmitted and stored (if at all) * The public key approach ensures that credentials are not reused or stored long-term * A unique key-pair per session can further isolate the testing instance * Connections should be limited to specific IP addresses and short-lived #### Benefits * **Lower barrier to entry:** Users do not need to understand the full setup process * **Consistency:** Every test runs in a standard, validated environment * **Scalability:** Easier onboarding of new contributors and hosts * **Insights:** Results can be aggregated to better understand trends and common issues #### Next Steps 1. Discuss and approve the general concept 2. Create a prototype of the service 3. Review and iterate based on initial testing 4. Plan deployment and documentation for broader use #### Request for Feedback The Hosting Team would appreciate feedback on the following, especially from those hosts who raised concerns about the current setup: * Is this the right approach to lowering the barrier? * Are there any privacy or security concerns that must be addressed? * Would hosts be willing to test this service in an early access phase? Some content could not be imported from the original document. [View content ↗](https://widgets.wp.com/likes/?ver=14.9.1#blog_id=120824672&post_id=138321&origin=make.wordpress.org&obj_id=120824672-138321-68aa074caae00), RFC: Automated Hosting Test Runner Service, Make WordPress Hosting, 01k3c2s852d9nvzmz93kpf8x8c.