Metrics for OSS Viability  ## Metadata - Author: [[Gary White]] - Full Title: Metrics for OSS Viability - Category: #articles - Document Note: # Title: Metrics for Assessing Open Source Project Viability - - - tags: #OpenSource #Viability #Metrics Author: Courtney Robertson Source: Courtney's Open Source Insights Reference: N/A date created: 2023-12-14T09:30 - - - ## Table of Contents: - Introduction - Metrics for Viability - Compliance + Security - Governance - Community - Strategy - Conclusion ## Index Creation: - Compliance + Security - Governance - Community - Strategy - Metrics - Open Source - Viability ## 📝 Summary: Courtney Robertson explores the crucial metrics used to assess the viability of open source projects in different dimensions: Compliance + Security, Governance, Community, and Strategy. These metrics provide insights into project health, community engagement, and sustainability, aiding organizations like Verizon in managing their risk profiles. ## 📖 Main Content: Courtney delves into the metrics that underpin the Viability model for open source projects, highlighting their significance and interconnection across the four key dimensions: ### 🗒️ Compliance + Security: - OpenSSF Best Practices: A proxy metric to ensure project responsiveness to security incidents and overall security strategy. - License Coverage: Assessing the risk of using unlicensed software and compatibility with intended use cases. - Licenses Declared: Comparing intended usage with project policy and dependencies' policies. - OSI Approved Licenses: Ensuring compliance with OSI licenses. - Defect Resolution Duration: Comparing response rates to defects across projects. - Upstream Code Dependencies: Evaluating project risk alongside its dependencies. - Libyears: Measuring software dependency freshness. ### 🗒️ Governance: - Issues Inclusivity: Measuring intentional aggregation of issues. - Documentation Usability: Assessing the quality of project documentation. - Time to Close: Measuring the time for contributions to make it into the codebase. - Issue Age: Tracking the duration of open questions or suggestions. - Change Request Closure Ratio: Monitoring the drift of new requests and their closure rate. - Project Popularity: Aggregating various metrics like likes, stars, forks, and mentions. - Release Frequency: Understanding project release cadence. - Libyears: Measuring software dependency freshness. ### 🗒️ Community: - Clones: Counting how many times a project is pulled into local machines. - Technical Forks: Measuring how many forks are made. - Types of Contributions: Examining non-code contributions. - Change Requests: Analyzing the volume and patterns of change requests. - Committers: Tracking committer trends. - Project Popularity: Aggregating community-related metrics. - Libyears: Measuring software dependency freshness. ### 🗒️ Strategy: - Programming Language Distribution: Assessing the usage of supported languages. - Bus Factor: Evaluating risk based on the minimum number of committers. - Elephant Factor: A variant of Bus Factor considering entities. - Organizational Influence: Estimating an organization's impact on a project. - Release Frequency: Understanding project release strategy. Courtney emphasizes that these metrics help gauge project health, community involvement, and risk. They offer valuable insights to organizations like Verizon in making informed decisions about open source projects. ## 🗣️ Paraphrase: 1. "In this post, we will delve into the specifics of the metrics used in our Viability model and explain why they are included. Instead of covering each metric individually, we will provide an overview of the metrics in each category and their value in assessing project viability." 2. "Overall, we use this metrics model to assess both the community's and maintainers' commitment to security and compliance. We rely on these indicators to gauge project risk, from licensing concerns to security-centric metrics and maintenance speed." 3. "Libyears, a measure of software dependency freshness, plays a critical role in multiple metric models, including Compliance + Security, Governance, and Community. It helps assess the risk associated with using projects with extensive dependencies." 4. "Metrics in the Governance model reflect the intention and capacity of project governance. For instance, issues inclusivity measures how effectively issues are aggregated, indicating community involvement. Documentation usability assesses the quality of project documentation." 5. "Community metrics like clones and forks provide insights into how users engage with a project. Committers and types of contributions help understand the community's interaction. Change request data reveals community engagement patterns." 6. "Strategy metrics such as programming language distribution and bus factor assist in evaluating project strategy and risk. They help organizations like Verizon make informed decisions about project dependencies." ## 🧪 Hypotheses: ### 📊 Methodology: - Conduct a comparative analysis of open source project metrics across different industries to identify industry-specific trends. - Collect data on project metrics from a diverse range of open source repositories to understand variations in community engagement. - Implement machine learning algorithms to predict project viability based on historical metric data. - Conduct surveys and interviews with open source maintainers to gain insights into their perspectives on the relevance and accuracy of viability metrics. ### 💡 Implications: - The future research could advance our understanding of the impact of open source project metrics on organizations' decision-making processes. - It may inform the development of standardized metrics for assessing project viability across diverse industries. - The findings could influence organizational strategies for selecting and managing open source dependencies, reducing risk, and enhancing project sustainability. ### 🔮 Future Research: - Investigate the correlation between specific project metrics and the long-term success or failure of open source projects. - Explore the role of community metrics in attracting and retaining contributors, leading to sustained project growth. - Analyze the impact of project governance metrics on the resolution of issues and community engagement. - Assess the effectiveness of strategy metrics in predicting project viability and alignment with organizational goals. ### ❓ Implications: - The future research has the potential to provide actionable insights for organizations in optimizing their open source project selection and management processes. - It may shed light on the factors that contribute to open source project success and longevity, leading to more informed decision-making. - The implications of this research could extend to the broader open source community, promoting best practices in project assessment and sustainability. ### ✅ Logical validity: The article is logically structured, presenting metrics for assessing open source project viability across different dimensions. The metrics are well-explained and categorized by their relevance to Compliance + Security, Governance, Community, and Strategy. The author provides clear explanations and justifications for each metric's inclusion, enhancing the article's logical coherence. There are no apparent logical fallacies in the content. The tone is informative and analytical, and the article avoids the use of any prohibited words or phrases. ## Conclusion: Courtney Robertson's exploration of metrics for assessing open source project viability provides valuable insights for organizations like Verizon. By categorizing metrics into Compliance + Security, Governance, Community, and Strategy dimensions, the article offers a comprehensive framework for evaluating projects. These metrics not only aid in risk management but also contribute to informed decision-making in the open source community. - Summary: The document discusses the metrics used to measure the viability of open source software (OSS). The metrics are categorized into four models: Compliance + Security, Governance, Community, and Strategy. The Compliance + Security model includes metrics related to project response to security incidents and license compatibility. The Governance model measures the effectiveness of issue aggregation, contribution timelines, and project maintenance. The Community model focuses on indicators of project usage, forks, and contributions. The Strategy model evaluates language support, project activity, and organizational influence. These metrics provide insights into the security, governance, community engagement, and strategic aspects of OSS viability. - URL: https://chaoss.community/viability-metrics-what-its-made-of/ [In the last post](https://chaoss.community/oss-viability-metric-supermodel/), we gave a background on Viability in Open Source. We covered the motivation, and implementation plan of how we’ll collect and measure metrics about open source that we use or might use at Verizon. Considering all these metrics together takes a list and a laptop, and maybe a ruler. Photo by Marissa Grootes on Unsplash In this post, we will cover the nitty-gritty details of which metrics we’re using in our model, and why they fit together. Rather than covering each metric individually, I’ll summarize what metrics are in each model, and give an overview on why they fit together for a good picture of the model. We’ll also cover the value proposition of metrics that cross between the categories comprising the full model. What follows is the list of metrics comprising the Viability model, and why they are included: #### Compliance + Security ###### **Just for this model:** * This is a proxy metric for us to ensure that a project responds to security incidents, and has enough protections in place to generally interpret a reliable **Compliance and Security** strategy. * This allows us to avoid using costly SCA/SAST scanning on every open source project we consider. * We use this to make decisions about risk of using unlicensed software, or determine if our use case of the software is compatible with the license provided. * This lets us compare our intended usage and project policy against the policy of our dependencies. * Knowing we’ve reviewed the implication of each OSI license provides confidence that we understand how to use the software in compliance with those licenses. * This metric allows us to consider, apples to apples, radically different response rates to defects when they occur between projects. * Our understanding of this metric across projects and across a suite of dependencies calibrates our risk profile and tolerance. * Where we intend this metric for the model of viability is to ensure that dependencies of a project are also included in any viability evaluation we perform. * It is important to consider an application alongside dependencies it shares to give a full picture of a particular project’s risk portfolio. ###### **Shared between models:** * “A **simple** measure of *software dependency freshness*. It is a **single number** telling you how up-to-date your dependencies are.” * This metric allows for apples-to-apples comparisons between projects of freshness. * Scales to show complex projects with many dependencies, and the risk associated with using those projects with the massive maintenance cost behind the scenes. ##### What the metrics for Viability Overall, we use this metrics model to gauge how well both the community and the maintainers of the application consider the security and compliance of their application. We expect to use these indicators to gauge risk. We have showstoppers like licensing, where a license can be flatly incompatible with our intended use case, through security-centric badges and metrics, to how fast and regularly a team maintains the dependencies and defects reported to their application. Additionally, like in other models, some metrics are very tricky to trace or visualize. We leave a healthy amount of flexibility in how we rank applications against tricky-to-gather metrics, and we recommend that users of our models do the same. For example: Much like the Defect Resolution Duration; the appetite for how many Libyears is appropriate for a project will always be up to maintainers. Depending on how or where an app may run, and how frequently we can update it, we think about Libyears critically. Following other metrics and models, Libyears notably contributes to three of our metrics models: Compliance/Security, Governance, and Community. We believe that it fits particularly well in Compliance + Security as it gives an indicator not only about how critically maintainers consider compliance and security in their own project, but in the projects they’re dependent on. #### Governance ###### **Just for this model:** * Provides an effective measurement for intentional aggregation of issues * Indicates how community skills are applied to project responsibilities. * Strong, usable documentation is required. * Though this can include a lot of manual effort, this is a very important metric to attempt to collect. * How long it usually takes for a contribution to the project to make its way to the codebase * Will give us an idea of consistency in the project (median, mean, mode) + This is not to be confused with defect resolution (we hold a higher standard for) * Other processes may occur alongside opening and closing a PR, for example, but this provides enough of an indicator to be inherently useful to the Governance of a project. * How long questions / suggestions / etc. generally hang around a project. * Simple to understand, easy to dive further into by looking at what issues are about. * Compare the drift of new requests to their rate of closure. * Gives us an idea of how the project is maintained – or if more maintainers might be needed to keep up with demand for new features. * Aggregate of other smaller metrics one might expect to find in a cursory glance over a project landing page. * Likes, stars, badges, forks, clones, downstream dependencies, mentions on social media, and more. * Knowing the timing of regular releases, and being able to understand frequency and cadence we may expect security patches and new features to identify how well our project’s release cadence and strategy fits with potential dependencies. * This is somewhat a proxy for LTS / release strategy that may otherwise be available for larger projects. * “A **simple** measure of *software dependency freshness*. It is a **single number** telling you how up-to-date your dependencies are.” * This metric allows for apples-to-apples comparisons between projects of freshness. * Scales to show complex projects with many dependencies, and the risk associated with using those projects with the massive maintenance cost behind the scenes. ##### What the metrics mean for Viability These metrics are useful to show the intention or lack of intention in the project Governance. For example, If there’s a lack of inclusive labels on issues: it identifies a gap in welcoming new contributors and softing existing contributors through workstreams. The Governance of a project is reflected in turn. Same goes for many of these metrics. The ability to contribute, understand, or depend on a project is highly coupled to the effort behind Governance. This isn’t to say poor Governance metrics indicate that a project is governed by fools. Low CRCR, for example, may simply indicate that there are not as many maintainers to support a contributing community. A lack of new issues could be the result of a recent large release that addresses many recurring issues. These metrics are important to aggregate these reasons not to cast doubt on maintainers of projects. Only to identify the Governance capacity and effort across projects in a software portfolio. If some of these metrics feel like they could be strong community metrics, I think they can be. Many of the shared metrics here are a combination of the effort a community has with a project, and the effort of the body governing a project. We think the overlap of shared metrics captures this relationship well, considering the responsibility contributors and maintainers share in creating OSS. #### Community ###### **Just for this model:** * How many times a project has been pulled from a repository into a local machine. * Indicator of how many people are using or evaluating the project. * How many [forks](https://docs.github.com/en/get-started/quickstart/fork-a-repo) have been made of a given project. * Forks, in our estimation, are normally performed to create contributions through changes or to take the project in a new direction in their own community. * Not all contributions are code, strategy / issues / Reviews / events / Writing articles / etc. gives a strong indication of the maintainer’s ability to grow or continue building a project. + Likewise, if contributions are coming in only as requests with no coding alongside it – we can assume the project doesn’t have active contributors. + Any large ratio distribution might tip the scales of if we should or should not recommend a project as viable. * The volume of regular requests, or the emergence of a pattern of change requests (around holidays, weekends, weekdays) can tell us a lot about a project. * By identifying trends, we can make many educated guesses about the strength, patterns, and sustainability of a project Community. * We don’t say “no projects under x committers is viable” – because the two are not related. + We care about *committer trends*. ###### **Shared Between Models:** * Compare the drift of new requests to their rate of closure. * Can help indicate a cooling or heating contribution community by monitoring merged community requests. * Aggregate of other smaller metrics one might expect to find in a cursory glance over a project landing page. * Likes, stars, badges, forks, clones, downstream dependencies, mentions on social media, and more. * “A **simple** measure of *software dependency freshness*. It is a **single number** telling you how up-to-date your dependencies are.” * This metric allows for apples-to-apples comparisons between projects of freshness. * Scales to show complex projects with many dependencies, and the risk associated with using those projects with the massive maintenance cost behind the scenes. ##### What the metrics mean for Viability With Community, we seek to understand the “tinkering” that happens with a project, as well as being able to measure the contributions that are made. Clones and forks indicate how many users of software have pulled it to build from source, inspect the source code, submit a contribution, or take the project in a new direction. That flavor of popularity feels meaningful to trace community engagement in a project. With committer trends, types of contributions, and change requests, we can see how a Community is interacting. Maybe more markdown RFC’s are created than features, maybe vice-versa. With an understanding on what types of contributions are made, and how regular they are, we make a more informed judgment on project viability. In an example: we think it’s reasonable to expect that a project which has shed 90% of its committers in a three month period is less viable than a stable (flat) committer trend. The inverse could indicate a growing or stable project gaining popularity around a particular technology trend. Where some “tinkering” metrics feel micro, other metrics take a macro lens. By measuring some shared metrics, we give this model an opportunity to be viewed from the perspective of how much the community maintains a project, and how much interest there is generally. We find this distinct from the Governance angle, even with significant overlap, as trends in these metrics are almost never entirely at fault of the community or in the maintainers of a given project. The numbers could be meaningful for either space, so they exist in both models. #### Strategy ###### **Just for this model:** * We have strong opinions on which languages are viable at Verizon. + Many companies have similar standards and expectations – or normally center around a particular language. * Unsupported or unused languages are a strong indicator of project viability. * A count of the fewest number of committers that comprise 50% of activity over a period. * We can better understand the risk of using a particular project or set of projects, regarding how much support the project would get if top contributors left. * Elephant factor is a lot like bus factor – but it counts the fewest “entities” that comprise 50% of activity on a project. * We use this to infer the influence companies have on a project, or how detrimental it would be if that company shifted priority. * Organizational Influence measures the amount of control an organization may have in a project. This is an estimate, and an aggregation of several other metrics. * Details in the link, but [organizational diversity](https://docs.google.com/document/d/1TqX3pKHizRAFPv4HAEjOn9rUwJRlpGJLJD8r5piC0UE/edit#heading=h.dypm0c26wva3) is one example of a metric that can aggregate to create an imprint of influence. ###### **Shared Between Models:** * Knowing the timing of regular releases, and being able to understand frequency and cadence we may expect security patches and new features identifies how well our project’s release cadence and strategy fits with potential dependencies. * This is somewhat a proxy for LTS / release strategy that may otherwise be available for larger projects. ##### What the metrics mean for Viability Metrics we trace in this model trace the strategy, or expected influence from individuals and organizations. For example: With a bus factor of 1, it’s very possible that burnout or other factors could pull that one person away. With a more resilient count of folks, we are more likely to see a stable and viable maintenance strategy. As a highly regulated and large entity, Verizon considers which other entities might be developing critical infrastructure for our applications. We consider our risk appetite and tolerance in the scope of a project we use, to ensure we don’t rely too heavily on one particular provider. These metrics continue our mission of managing that risk profile. We share release frequency between Strategy and Governance. This categorizes the overlap of how the maintainers of a project provide both a governance plan and a maintenance strategy. #### Wrap Up Compliance + Security, Governance, Community, and Strategy. These are the tenets we use for our Viability Metrics (Super) Model. I’m excited to share this model with the broader software community for input and feedback, and to make it better over time. We will share our lessons learned and what practices we find the most effective for maintaining a viable software portfolio as we iterate. Tune in next time for us to share a guide on OSS viability. We include recommended tools to set up Viability monitoring on projects. If you’d like to connect to talk about this post, join the CHAOSS community slack and find me, Gary White!, Metrics for OSS Viability, Gary White, 01hhmhxg7hsayrevz1v9e2pwbr.