Episode 131 Repo-Health - A Tool Built on CHAOSS Metrics

Episode 131: Repo-Health - A Tool Built on CHAOSS Metrics ![rw-book-cover](https://img.youtube.com/vi/k7xlCxImq00/maxresdefault.jpg) 🔗 **Source:** [Episode 131: Repo-Health - A Tool Built on CHAOSS Metrics](https://www.youtube.com/watch?v=k7xlCxImq00) > [!note]+ Document Notes > tags: #open-source #metrics #community-health #repository-analysis #developer-tools > author: CHAOSS Community > source: YouTube > reference: https://www.youtube.com/watch?v=k7xlCxImq00 > date created: 2024-12-19 15:30 > > # 🏷️ Title: Episode 131: Repo-Health - a tool built on CHAOSS metrics >  ## 🧰 Action Inbox - [H1] Explore Repo-Health tool for project evaluation — first impression metric for contributors #open-source - [H2] Consider balance between AI automation and human interaction in OSS — community engagement priority #community-health - [H3] Investigate four-pillar framework for repository assessment — activity, maintenance, community, documentation #metrics ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [CHAOSS Project](https://chaoss.community) - [Repo-Health Tool](https://github.com/elshadsm/repo-health) ## 📝 Summary - **What happened**: Elshad, a software developer from Azerbaijan, created Repo-Health, a tool that implements CHAOSS metrics to provide an overall health score for open source repositories - **Why it matters**: Addresses the need for quick first-impression assessment of repository health for new contributors, bridging the gap between comprehensive metrics and practical decision-making - **Key stakeholders**: Open source contributors, maintainers, students learning about OSS, developers evaluating projects - **Risks and tradeoffs**: Single health score risks oversimplification (Goodhart's law), but provides needed accessibility for newcomers vs. CHAOSS's multi-metric approach - **Bottom line**: Tool demonstrates practical application of CHAOSS metrics for specific use case of repository first-impression assessment ## 📖 Main Content Repo-Health represents a practical implementation of CHAOSS (Community Health Analytics in Open Source Software) metrics, designed to give contributors and maintainers a quick assessment of repository health. The tool organizes metrics into four main pillars: activity (30% weight), maintenance (weighted for response times), community (popularity indicators), and documentation (AI-assisted evaluation). The project addresses a specific gap in the open source ecosystem - while CHAOSS provides comprehensive individual metrics for deep community analysis, newcomers often need a simple first impression to decide whether to invest time investigating a project further. This creates a complementary relationship rather than competition with existing CHAOSS tools like Augur. Key technical features include commit recency analysis, issue response time tracking, popularity scoring using logarithmic scaling for stars/forks, and LLM-based documentation quality assessment (contributing 20% to overall score with sanitized data inputs). The tool emerged from Elshad's teaching experience, where he observed students struggling with poor documentation and lack of communication in open source projects. This pedagogical background influenced the emphasis on documentation quality and maintainer responsiveness as key health indicators. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/k7xlCxImq00?t=0) — [H1] Introduction and panel setup - Key idea: CHAOSS Cast podcast focusing on measuring open source community health - Evidence: Panel includes Alice (FreeBSD Foundation), Adrian (Augur project), Georg (CHAOSS), guest Elshad #### ⏱️ [03:45](https://youtu.be/k7xlCxImq00?t=225) — [H2] Elshad's background and motivation - Key idea: Azerbaijan-based developer passionate about open source contribution - Evidence: "Contributing to open source projects is my passion and I love contributing what I do" #### ⏱️ [05:12](https://youtu.be/k7xlCxImq00?t=312) — [H3] Discovery of CHAOSS metrics - Key idea: Found CHAOSS while researching contribution metrics and project health - Evidence: "I felt like CHAOSS metrics have already done that and have already researched about it" #### ⏱️ [07:30](https://youtu.be/k7xlCxImq00?t=450) — [H1] Four-pillar framework explanation - Key idea: Activity (30%), maintenance, community, documentation structure for health assessment - Evidence: "I created a section with four pillars by getting inspired by CHAOSS metrics" #### ⏱️ [09:45](https://youtu.be/k7xlCxImq00?t=585) — [H2] Activity metrics focus - Key idea: Commit recency and uniqueness as primary health indicators - Evidence: "What's the commit recency? What's the commit's uniqueness? And based upon that especially I gave 30% to this section" #### ⏱️ [11:20](https://youtu.be/k7xlCxImq00?t=680) — [H3] Maintenance pillar details - Key idea: Issue response time and open issue ratio indicate project maintenance quality - Evidence: "What is the issue response time? How much time contributors need to wait to see the response from maintainers?" #### ⏱️ [12:50](https://youtu.be/k7xlCxImq00?t=770) — [H2] Community vs popularity distinction - Key idea: Georg challenges stars/forks as community metrics vs true people engagement - Evidence: "When I personally think of community I think of the people...Stars and forks in my perception is more popularity" #### ⏱️ [15:30](https://youtu.be/k7xlCxImq00?t=930) — [H1] Documentation and LLM integration - Key idea: Teaching experience drives documentation focus with AI-assisted evaluation - Evidence: "I integrated LLM judge here...feeding LLM, I would say. So, Elshad, you've kindly explained" #### ⏱️ [18:45](https://youtu.be/k7xlCxImq00?t=1125) — [H3] Tool purpose and user experience - Key idea: Repo-Health provides first impression for contributors to avoid low-quality contributions - Evidence: "Not contributing by writing their names into the read me because...during Hacktoberfest a lot of PRs are being generated with LLMs" #### ⏱️ [22:10](https://youtu.be/k7xlCxImq00?t=1330) — [H2] Single metric vs multiple metrics debate - Key idea: Tension between CHAOSS multi-metric approach and user need for simple scores - Evidence: "Why is there not a single CHAOSS metric that is just community health single number?" #### ⏱️ [26:30](https://youtu.be/k7xlCxImq00?t=1590) — [H1] Goodhart's law considerations - Key idea: Risk of metrics becoming targets that lose effectiveness - Evidence: "If a metric becomes your goal, it ceases to be a good metric" #### ⏱️ [29:15](https://youtu.be/k7xlCxImq00?t=1755) — [H3] Contextual metrics discussion - Key idea: Alice explains different metrics serve different goals and contexts - Evidence: "It's very contextual which metrics are useful and of what they're for" #### ⏱️ [33:45](https://youtu.be/k7xlCxImq00?t=2025) — [H2] First contact vs embedded community perspectives - Key idea: Repo-Health serves newcomer needs while CHAOSS serves embedded community analysis - Evidence: "First contact moment...coming much more at that sort of first contact" #### ⏱️ [37:20](https://youtu.be/k7xlCxImq00?t=2240) — [H1] AI impact on open source collaboration - Key idea: LLM-generated contributions changing dynamics of human interaction in OSS - Evidence: "There's a big conflict here there's a big dilemma...without human review it's kind of very difficult" #### ⏱️ [42:10](https://youtu.be/k7xlCxImq00?t=2530) — [H3] Future development and collaboration - Key idea: Plans to refine metrics and potentially integrate with existing CHAOSS tools - Evidence: "I would add more metrics and I would reduce some features...over engineered that feature" ## 📚 References - [CHAOSS Community](https://chaoss.community/) - [CHAOSS Podcast](https://podcast.chaoss.community/) - [FreeBSD Foundation](https://freebsdfoundation.org/) - [Augur Project](https://github.com/chaoss/augur) - [Goal-Question-Metric Framework](https://en.wikipedia.org/wiki/GQM) ## 🗣️ Paraphrase - Repo-Health translates comprehensive CHAOSS metrics into an accessible first-impression tool for repository evaluation - The tool balances quantitative metrics (activity, maintenance, popularity) with qualitative assessment (documentation quality via AI) - Project addresses specific need for newcomer-friendly repository assessment while complementing rather than replacing deep community health analysis ## 📰 Newsletter **New Tool Spotlight: Repo-Health Brings CHAOSS Metrics to Repository First Impressions** The WordPress community continues to benefit from broader open source health research. A new tool called Repo-Health demonstrates how comprehensive community metrics can be made accessible for everyday contributor decisions. Created by developer Elshad from Azerbaijan, Repo-Health implements CHAOSS (Community Health Analytics in Open Source Software) metrics through a four-pillar framework: activity (30% weight), maintenance quality, community engagement, and documentation standards. The tool provides an overall health score to help contributors quickly assess whether a repository merits deeper investigation. This approach addresses a common challenge in WordPress contribution - how do newcomers evaluate the health of plugins, themes, or core areas before investing time? The tool's emphasis on issue response times and documentation quality directly applies to WordPress project evaluation. **Key takeaway**: While comprehensive metrics serve maintainer teams, simplified health indicators help newcomers make informed contribution decisions. **WordPress Application**: Consider how this framework might help evaluate plugin sustainability, theme support quality, or contributing team responsiveness across WordPress.org projects. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer experience: Create repository health assessment guidelines for WordPress projects - Education: Develop workshops on evaluating project health before contributing - Community management: Study correlation between health metrics and contributor retention #### OSPO - Software Inventory: Integrate repository health scoring into dependency evaluation processes - Community Engagement: Evaluate upstream project health before funding or partnership decisions - Strategic Alignment: Establish health metrics KPIs for sponsored projects #### Community - Contributors: Implement first-impression assessment tools for newcomer onboarding - Maintainers: Monitor health metrics to improve project approachability - Sponsors: Use health indicators to guide funding allocation decisions #### WordPress - Community: Adapt four-pillar framework for WordPress project evaluation standards - Training: Create lesson plans teaching project health assessment skills - Documentation: Establish health-focused documentation standards across WordPress.org - Plugins: Implement health scoring for plugin directory sustainability indicators - Themes: Apply maintenance metrics to theme review and directory processes #### The WPCC - Programs: Fund development of WordPress-specific project health assessment tools - Projects: Support governance clinics incorporating health metrics evaluation #### FAIR - Technical independence: Establish repository health as criterion for technical sustainability assessment ## 🧪 Hypotheses - Repository health scores correlate with contributor retention and satisfaction rates - Four-pillar framework (activity, maintenance, community, documentation) predicts project longevity better than individual metrics - AI-assisted documentation evaluation provides reliable quality assessment when combined with quantitative metrics - First-impression tools reduce barriers to open source contribution by lowering evaluation overhead ### 📊 Methodology - Longitudinal study tracking contributor engagement across repositories with different health scores - Comparative analysis of four-pillar scores against traditional popularity metrics (stars, forks) - Validation study of LLM documentation assessment against human expert evaluation - A/B testing newcomer contribution rates with and without health score availability ### ❓ Questions - How do health scores change contributor behavior and project selection decisions? - What weight distribution across the four pillars optimizes newcomer success prediction? - Can health metrics identify projects at risk of abandonment before traditional indicators? ### 🔮 Future Research - Integration pathway between Repo-Health approach and existing CHAOSS tool ecosystem - WordPress-specific health metrics incorporating plugin/theme review processes - Longitudinal impact study of health-score-guided contribution decisions ### 💡 Implications - Practice: First-impression tools can reduce contributor onboarding friction while maintaining quality standards - Policy: Health metrics may inform funding and partnership decisions for open source sustainability - Education: Structured health assessment teaches newcomers project evaluation skills beyond surface indicators - Community: Balancing accessibility tools with comprehensive analysis serves different stakeholder needs ### 🧠 Learning Styles Assessment Reading level estimate: 12-14 (Graduate level due to technical terminology) #### 👀 Visual learners - Present: Four-pillar framework diagram, metric weight visualization - Improve: Repository health dashboard mockups, metric correlation charts #### 🔊 Aural learners - Present: 45-minute podcast discussion with examples - Improve: Audio summary of key health assessment principles #### 🗣️ Verbal learners - Present: Clear definitions of activity, maintenance, community, documentation pillars - Improve: Comparative analogies between health assessment and medical diagnosis #### 🧩 Kinesthetic learners - Present: Step-by-step tool usage walkthrough - Improve: Hands-on repository evaluation workshop with real examples #### 🧠 Logical learners - Present: Weighted scoring algorithm, metric selection rationale - Improve: Decision tree for repository selection, correlation analysis #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format with multiple perspectives - Improve: Community discussion guide, peer evaluation exercises #### 🧘‍♀️ Solitary learners - Present: Individual tool exploration and metric interpretation - Improve: Self-assessment checklist, reflection prompts #### 🌿 Naturalistic learners - Present: Organic evolution of open source ecosystem health - Improve: Health metrics as ecosystem sustainability indicators ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive link text - Improve: ARIA labels for metric visualizations, alt text for framework diagrams #### Keyboard navigation - Present: Standard podcast player controls - Improve: Keyboard shortcuts for timestamp navigation #### Contrast and readability - Present: Standard text presentation - Improve: High-contrast mode for metric dashboards #### Text resizing and reflow - Present: Responsive transcript display - Improve: Scalable metric visualization layouts #### Media and captions - Present: Full transcript available - Improve: Time-synced captions, audio descriptions for visual elements #### Forms and interactions - Present: Basic podcast interface - Improve: Accessible health score calculator interface ### 🌍 DEIB Assessment #### Cultural breadth - Present: International perspective from Azerbaijan developer - Improve: Multi-region case studies, localized health metric examples #### Inclusive language - Present: Technical terminology with explanations - Improve: Plain language summaries, non-English project examples #### Neurodiversity support - Present: Structured four-pillar framework - Improve: Multiple presentation formats, visual organization aids #### Representation - Present: Diverse geographic and project perspectives - Improve: Varied contributor background examples #### Psychological safety - Present: Respectful technical discussion - Improve: Explicit welcome for newcomer questions ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear framework derivation from CHAOSS metrics principles - Inductive strength: Evidence from teaching experience supports documentation emphasis - Causal reasoning: Activity metrics linked to project sustainability outcomes - Analogical strength: Health assessment compared to medical diagnosis approach #### Fallacies - False dichotomy: Single score vs. multiple metrics presented as either/or rather than complementary - Appeal to authority: CHAOSS metrics assumed optimal without critical evaluation #### Tone of voice - Overall tone: Collaborative, technically focused, respectful of different --- ## Full Transcript [00:05] [music] [00:09] >> Welcome back to Chaos Cast, the Chaos [00:13] Community podcast where we share use [00:15] cases and experiences with measuring [00:18] open source community health. [00:20] >> [music] [00:22] >> On the panel today, we have Alice. Hi [00:26] there. I'm Alice Soweby. I'm a freelance [00:29] technical program manager and currently [00:31] supporting the FreeBSD Foundation. I'm [00:34] active in several open source projects [00:36] including producing this podcast with [00:37] Chaos and editing the Osprey book with [00:40] the To Do Group. I'm based in the UK in [00:42] a small village near Bath. [00:45] Awesome. Great to have you with us, [00:47] Alice. And Adrian. [00:49] Hi, I'm Adrian. Just joined Chaos [00:52] relatively recently and have now I've [00:53] been mostly doing work with the Augur [00:56] project to helping maintain [00:59] that tool for [01:00] bringing various metrics and helping [01:02] communities measure how healthy they [01:04] are. [01:05] Ah, welcome. And myself, Georg Link. Hi [01:08] everyone. Great to be back with you for [01:12] another episode of Chaos Cast. I am [01:15] super excited to be here today. We have [01:18] a guest, Elshad. I had come across his [01:22] work. He's published a blog post [01:25] implementing Chaos metrics. And so I'm [01:28] really excited to hear more about your [01:31] story, Elshad. And how about we start [01:34] with an introduction? So for the [01:36] listeners [01:38] of our community that they can get to [01:40] know you. [01:41] Hi, my name is Elshad. I'm a software [01:44] developer, an open source contributor. [01:47] Actually, contributing to open source [01:49] projects is my passion and I love [01:52] contributing what I do. And [01:55] I built Repo Health by getting inspired [01:58] by Chaos metrics. And today I'm really [02:03] glad that I'm in this podcast that I'll [02:05] discuss about how I came up to this [02:07] point and integrated Chaos metrics into [02:10] my project. Yeah, welcome. And what part [02:14] of the world are you from? I am from [02:17] Azerbaijan. Awesome, welcome. So so glad [02:20] to have you on the podcast. And what was [02:23] the journey for you to discover the [02:26] Chaos project and the Chaos metrics? [02:29] What kind of sent you down this rabbit [02:32] hole? When I started, I was doing [02:34] decision metrics. And I just wanted to [02:37] Google my thoughts that what I'm [02:40] thinking about contributions, [02:43] all documentation and maintenance. And I [02:47] felt like Chaos metrics have already [02:50] done that and have already researched [02:52] about it. And I just thought that [02:54] actually I can represent my ideas by [02:56] using Chaos metrics and show in a [03:00] different use case that I can help [03:03] contributors and at the same time [03:05] maintainers. And they can use my tool [03:08] and come up to some point. Is it okay if [03:11] me to jump in with a a question, Elshad? [03:14] How long have you been looking at and [03:17] trying to understand repo health? And [03:20] like what was your first impression of [03:23] of [snorts] the the Chaos project when [03:25] you found it? [03:27] So when I found Chaos metrics, I kind of [03:31] started thinking about that this project [03:33] is taking care of contributors. And it's [03:37] basically helping me how to avoid some [03:41] projects and what kind of metrics I can [03:43] check in one project if I want to [03:46] contribute. And if I want to see the [03:49] whole purpose of the project, I would [03:51] say, and the health overall health of [03:54] the project. That's how I came through, [03:56] actually. And while doing research I did [04:00] decisions based on different metrics. [04:03] And I real in my decision metrics at the [04:06] end of the day I chose using Chaos [04:08] metrics. Were there particular [04:11] like specific metrics from the Chaos [04:14] site that you found especially useful or [04:17] others that you found less useful? Like [04:19] I imagine you probably weren't [04:20] implementing all of the Chaos metrics [04:21] cuz there's so so many of them. [04:23] There are so many metrics. But what I [04:26] really focused on is activity, I would [04:29] say. What's the commit recency? What's [04:32] the commit's uniqueness? And based upon [04:36] that especially I gave 30% [04:39] to this section. I would say that I [04:41] created a section with four pillars by [04:45] getting inspired by Chaos metrics. And I [04:49] would say I tried to represent in a way [04:51] that [04:52] was being structured in my mind. Also, I [04:56] focused on maintenance part. What is the [04:59] issue response time? How much time [05:02] contributors need to wait to see the [05:05] response from maintainers? And what's [05:07] the open issue ratio? And what's the [05:10] project age? I think these show the [05:14] overall structure of the project. Also, [05:17] community. So is how popular is the [05:21] project is I think is very important. [05:24] And here I mainly focused on log scale. [05:28] I decided to make a decision such as [05:31] there's a difference between 10 and 100 [05:33] stars compared to 1,000 and 1,000 10 [05:37] stars. And especially in documentation [05:41] part I just checked out whether the [05:43] project has nice readme, [05:46] contributing.md, [05:48] and all of the necessary details that [05:50] contributors need when they contribute. [05:53] Because [05:54] I kind of relate to my own personal [05:58] life. When I was teaching students, I [06:00] realized that lack of communication [06:03] is leading people towards wrong [06:07] directions and they're making wrong [06:09] decisions. That's why I focused on [06:12] documentation. That's pretty much it, I [06:14] would say. Yeah, I like how you broke [06:17] that out into the four pillars as you [06:20] call them. One, activity. Two, [06:22] maintenance. Three, community. Four, [06:24] documentation. And this is a pattern I [06:28] think consistent with how I've been [06:30] thinking about project health. Activity, [06:34] frequency of commits, recency of updates [06:36] you have in here. Is there activity in [06:38] the project? Is it alive? Is something [06:41] happening? Because without that activity [06:43] how is the software going to survive the [06:46] evolution of technology? A stale project [06:48] is a dead project. And so we we need [06:52] that activity. The maintenance also [06:55] makes sense what I'm seeing here with [06:58] issue response time, open issue ratio. [07:02] Like how active, not just active, but [07:05] how quickly are the responses from the [07:07] maintainers? How long does it take for [07:10] something that comes up to be fixed? And [07:13] then community, you took stars and [07:16] forks. This is where I have a different [07:18] opinion. I'm curious to maybe reason [07:21] this out. When I personally think of [07:24] community I think of the people. And the [07:28] people who are active, the unique [07:31] authors, which in your model you've put [07:33] into activity and you probably have a [07:35] good reason for that. But then also the [07:38] users, the people who are opening issues [07:42] and commenting and so on. That's what [07:44] what I think of community. Stars and [07:46] forks in in my perception is more [07:50] popularity or awareness that this [07:54] project exists. So [07:56] maybe we can maybe you can explain your [07:59] thoughts with how you divided those [08:01] metrics up and named them. [08:03] So when I focused on this matrix I [08:07] mainly focused on activity part because [08:10] I thought that that's the main section. [08:12] Because Git history tells us a lot about [08:14] the project. And we can see the commit [08:17] messages and even get some ideas from [08:20] there that how important that software [08:23] it is. And also that's why I gave 30% to [08:27] this section. And then in maintenance [08:31] part I just thought about the issue [08:34] response time and overall ratio the [08:37] issue open issue ratio. So I would say [08:42] for me I just thought about these four [08:45] sections. And shortly community wise, I [08:48] gave 20% to this section. [08:51] Because [08:52] I think I can improve this section. [08:55] Right now it mainly focuses on the [08:57] popularity of the project. But it would [09:00] be better to give more details and to [09:02] use more metrics, I would say. And I'm [09:04] planning to work on that. And my [09:07] software is in the early stages. I And [09:11] I'm trying to evolve this project as [09:13] much as I can. And the documentation as [09:17] I told, it comes from my experience. And [09:20] when I was studying in college, I saw [09:23] that difficulty from college students. [09:26] And I realized that it would be better [09:27] to integrate. And I would say I [09:31] integrated [09:33] LLM judge here. Because at some point I [09:36] realized that it would be better to feed [09:39] this compact sanitized data into the LLM [09:43] and get some reasoning out of there. But [09:47] of course not focusing on LLM because as [09:50] we already know LLM is not [09:52] deterministic. [09:54] And it cannot give us the real data. but [09:57] but I kind of delegate 20% [10:01] to LLM [10:03] with sanitized data by feeding LLM, I [10:06] would say. So, [10:08] Elshad, you've kindly explained a little [10:10] bit about how you were thinking about [10:12] metrics and what value each one had and [10:15] how you would combine those values [10:16] together to understand the health of a [10:19] repo. I don't know if we've made it very [10:22] clear yet to the listener if anyone [10:24] hasn't read your blog [clears throat] [10:25] yet, which they may not have done, they [10:27] may be coming to this podcast as the [10:29] first time they're hearing about this. [10:31] Would you like to explain at a high [10:34] level what the tool is that you've made [10:37] and what the experience of using the [10:39] tool is, why somebody might want to go [10:41] there and and try it out and what they [10:43] would would be able to get from it? [10:45] Yeah, thanks for asking. I would say [10:47] repo health is for contributors and [10:50] maintainers to see what to do and what [10:53] not to do in one project and it kind of [10:57] gives overall perspective and overall [10:59] health of the project to contributors [11:03] and they can see how's the activeness in [11:05] the project and what kind of things they [11:08] can avoid. If I give an example, I would [11:11] say not contributing by writing their [11:13] names into the read me because as we [11:16] already know in hack during [11:18] Hacktoberfest a lot of PRs are being [11:21] generated with LLMs and some people are [11:24] trying to contribute without putting [11:26] effort and they can see and they can [11:28] avoid some decisions and they can see [11:31] the positive structure, positive [11:33] commits. I mean, I have a section which [11:35] is called banger commits that they can [11:37] see [11:38] what kind of things they can do to [11:40] improve the project and on top of that, [11:43] they can see the overall score of the [11:45] project based on the real metrics and [11:49] LLM judge. [11:50] You said you you had like you were [11:51] computing these metrics into like an [11:53] overall score, like [11:55] what kind of led you to wanting to have [11:57] an overall score and present it that [11:59] way? I would say when users research [12:02] about the project, they kind of want to [12:04] see in which stage is this software it [12:07] is and with overall score, they can see [12:11] that and they can kind of continue to [12:13] see other metrics and that's basically [12:17] to give the overall point, overall [12:20] health of the project. That's kind of [12:22] interesting cuz I think within some of [12:24] the kind of chaos spaces, like I've [12:25] heard I've heard this kind of question [12:26] asked before on my chaos, like for [12:29] example where someone was kind of asking [12:32] about like, why is there not a single [12:33] chaos metric that is just community [12:36] health single number? And I think some [12:38] of the responses I've kind of heard or [12:41] I'm not sure exactly how this is [12:42] probably [12:42] me combining this from several things [12:45] I've seen, but it seems like the idea [12:47] behind [12:48] why chaos decided not to have an overall [12:51] single metric is is sort of the whole [12:53] idea of like the Goodhart's law, like if [12:55] a metric becomes your goal, it ceases to [12:57] be a good metric. And like I feel like [12:59] some of the attitude at least from some [13:02] of the chaos tools is more like, hey, [13:05] show multiple metrics. Like, hey, show [13:06] multiple graphs of okay, here's a graph [13:09] of commit activity, here's a graph of [13:10] this so that you can kind of look at the [13:13] more holistic view and see, oh, this [13:15] project is seemingly doing okay on on [13:18] most of these and maybe this is one [13:19] metric that's just had a weird spike or [13:22] did a weird thing. [13:24] So, it's just kind of interesting that [13:25] kind of you took that different decision [13:26] to do an overall metric compared to kind [13:28] of what chaos has done. [13:30] I would love to add something here [13:32] because [13:33] the last time somebody created an issue [13:35] in my project by giving different ideas [13:38] such as nowadays with the usage of LLM, [13:42] sometimes commits are going up and [13:44] documentation is getting ready, but [13:48] there is basically commit burst and I [13:50] realized that I can kind of add [13:53] different metrics and pivot my idea into [13:56] different section, maybe change the [13:59] representation of it and I'm thinking [14:02] about it actually because right now [14:04] there are several metrics in their [14:06] overall score but I think I can improve [14:09] it and I can bring other ideas as I told [14:13] for example sometimes [14:15] most of the commits are happening into [14:17] the read me and it doesn't show the real [14:20] activity in the project. Maybe I can [14:23] just map through the commits and see [14:25] what's happening and reduce the false [14:28] positive there. Of course, I didn't [14:31] complete the section because I would say [14:34] chaos metrics is doing really good job [14:37] by bringing all of this ideas [14:40] all together, but overall in open source [14:43] I think it's very complicated to bring [14:46] all ideas and bring only one nice [14:50] structure and showing that yes, that's [14:53] just the way it is. It's kind of not [14:55] possible [14:56] to show all leaders one way. So, that's [14:59] an interesting topic that from the very [15:03] beginning of chaos if come around and [15:07] Adrian, I think you summarized pretty [15:08] well why [15:11] the chaos project has not gone down this [15:14] route of creating one single [15:17] comprehensive metric. But Elshad, what [15:19] what you're saying is so true and I've [15:21] heard many times even before we started [15:23] chaos that hey, can we please have all [15:26] this noise, all this data [15:30] in one [15:31] single number, one place that we can [15:35] focus on making decisions, not on [15:37] analyzing data. One of the things that I [15:40] think actually as shortcoming [15:43] of the chaos project is that we never [15:47] went the step [15:49] from creating the raw individual atomic [15:53] metrics, whatever you want to call call [15:55] them. Yes, we've created metric models, [15:58] but we haven't created structure, an [16:02] algorithm, whatever to say here is a [16:04] health score for a project. What you're [16:07] saying that it shows there is a need to [16:10] do this and I've heard about this time [16:12] and time again. And so the struggle [16:16] in the chaos project has always been, do [16:17] we want to be that community that says [16:21] what does healthy look like or not look [16:23] like? [16:24] Or do we just want to be a resource for [16:27] someone like yourself to go and create [16:30] their own opinionated [16:32] metrics. So, so I'm kind of just sharing [16:35] a little bit of the history that has led [16:38] us to this point because I think it's [16:41] good [16:43] to have opinionated here's one way to [16:46] combine all these metrics to get one [16:48] view of project health [16:50] and maybe we can continue that work [16:53] somehow and if we could standardize on [16:56] that, I think we would as a open source [16:58] ecosystem as a community move forward. [17:01] I'd like to jump in on this if that's [17:02] okay. Actually, I think what's occurring [17:06] to me in listening to this this [17:07] conversation is that it's very [17:10] contextual which metrics are useful and [17:14] of what they're for. CHAOS often talks [17:17] about using a goal question metric [17:19] framework and goal question metric is [17:22] really the point of it is the the metric [17:25] is a tool towards achieving a goal and [17:28] that you would start with having a goal [17:31] you would work out if you have a [17:33] question that can be answered with [17:35] numbers [17:36] and then you look for a metric that will [17:38] help you to answer that question and [17:40] that you would use that metric over time [17:43] to understand something about whether [17:45] the work that you're doing or the [17:46] actions that you're taking are taking [17:49] you closer to achieving your goals or [17:51] not and then you can also evaluate the [17:53] metric when you do that as well as [17:54] evaluating that you know, your [17:56] activities. If I understand what Elshad [17:59] is is doing, he's looking at a very [18:01] specific situation or use case which is [18:05] when you first encounter a project, you [18:07] want an impression that gives you an [18:09] idea of do I want to spend time [18:12] investigating this project and [18:14] understanding what's inside it? Or is it [18:16] really unhealthy or perhaps its profile [18:19] doesn't really look the way I would want [18:21] the project to look if I engage with it. [18:23] And I actually think that Elshad's [18:26] work is a great example of how producing [18:29] an open source project like chaos with [18:31] the metrics that it surfaces allows then [18:34] other tools to be built on top that have [18:36] a much more specific context to the user [18:39] and it doesn't require that to be a part [18:42] of chaos for chaos to be useful in that [18:45] context. It's done its job if you like. [18:48] Most of the chaos metrics I think were [18:50] originally or have been used by people [18:53] who want to understand their own [18:54] communities that are already deeply [18:55] embedded in and improve their practices [18:58] and community health from the inside of [19:00] the community. Whereas Elshad is is [19:02] really talking about something that is [19:04] coming much more at um [19:07] that sort of first contact moment. And [19:10] I'd love to get your corrections on [19:13] on my comment there, Elshad, if you [19:15] would. I would say you nailed the point. [19:17] You articulated very better, [19:20] I would say. That was my first idea to [19:22] give the first impression about the [19:24] project to the user and they would see [19:27] kind of could they spend their time, [19:29] could they spend to investigate. That's [19:32] the overall structure, but after this [19:34] point, they can go deep and they can [19:36] search about the project. I feel like [19:39] some of the things Alshad was saying, [19:40] too, like kind of remind me of some of [19:42] my time back when I was a a student at [19:44] Rochester Institute Technology. I've [19:47] spent several years being the teaching [19:48] assistant for the humanitarian free and [19:50] open source software curriculum [19:53] that that university has and some of the [19:57] assignments that are kind of in that [19:58] class it's it's a it's very much an like [20:00] kind of an introduction to what open [20:02] source is and what communities are and a [20:05] lot of that. And basically I think one [20:07] of the assignments is [20:09] to have students teaching them to [20:11] evaluate for themselves like hey is this [20:13] the kind of project I would like to [20:14] contribute to right like have there been [20:16] commits somewhat recently like like who [20:19] are the maintainers like how active are [20:21] they like looking kind of what issues [20:24] and pull requests exist and like how [20:25] communicative they are if various things [20:27] like that there's [20:28] various kind of different aspects of the [20:31] assignment that help them evaluate that [20:33] but I think [20:34] it sounds like yeah like repo health is [20:36] sort of hitting on a very similar [20:38] purpose of [20:40] like people coming at a repository like [20:42] brand new like as I was saying like if [20:44] there's kind of a [20:45] complimentary aspect of of the chaos [20:48] metrics chaos metrics are maybe more for [20:52] like the people who are embedded in the [20:53] community already and are [20:55] like really trying to deeply understand [20:57] the health but there's also the need the [20:58] need for newcomers and it's like it's [21:00] kind of interesting tension between the [21:02] two perspectives where like they're both [21:04] correct and and it's not like there has [21:06] to be one or the other. I completely [21:09] agree that it's kind of different [21:11] analysis when it comes to usage of the [21:14] repo health and also what people did my [21:17] idea is the the in the last semester of [21:22] my study here in Canada I had a very [21:26] nice lesson introduction to open source [21:29] that when I started I kind of knew that [21:31] what I needed to do because of being a [21:33] teacher and getting through the problems [21:36] with my students but thanks to my [21:38] teacher I think he broadened my horizon [21:41] by pushing me out of comfort zone and I [21:44] started contributing to projects that I [21:47] would say are very deep compared to my [21:50] current skills and my teacher basically [21:53] pushed me through and he told me that [21:56] you can communicate with maintainers and [21:58] come up to some points and I kind of [22:01] remember once I contributed to one [22:03] project that was very difficult to me [22:05] but I just read the documentation and I [22:09] didn't have a lot of idea about the [22:11] project but I contributed thanks to the [22:14] nice documentation and nice [22:16] communication skills with maintainers [22:18] that's why also maintainer review is [22:21] very important in open source and [22:23] response time I really [22:26] took care of it and also [22:28] my classmates also experienced the same [22:30] thing when they contributed they were in [22:33] the loop kind of that they were [22:35] sometimes they wrestled with AI reviews [22:39] and maintainers didn't attend the [22:42] problems but there's a big conflict here [22:45] there's a big dilemma I would say [22:47] without human review it's kind of very [22:49] difficult to come up to the solution and [22:52] if they give us strong ideas we can come [22:56] up to the solution very quick. You're [22:58] touching on a point that about the way [23:02] open source engagement is changing with [23:06] AI and large language models getting [23:09] into that interaction we just had this [23:12] conversation yesterday in the AI [23:16] alignment working group which I want to [23:18] give a shout out here it's a chaos [23:22] working group about how AI is changing [23:26] open source or how to make sure that AI [23:31] follows our values and supports us [23:34] rather than gets in the way overloads us [23:36] with AI slob and PRs and projects now [23:41] closing their PRs so anyway there's a [23:43] lot of change right now and I appreciate [23:46] you seeing that and calling that out for [23:49] me when I think about this problem space [23:52] I return to the basic question of why do [23:58] we get together in an open source [24:00] community [24:02] are we trying to create something [24:04] together are we trying to share [24:07] knowledge are we trying to [24:10] stand up to [24:12] monopolies and create alternative [24:15] technology for the rest of us [24:18] and then the way that that we use AI in [24:21] that interaction with each other becomes [24:24] a different question than if we say hey [24:26] let's focus on the interactions and the [24:29] way we've collaborated is changing that [24:31] to me is not a helpful and concern [24:34] [clears throat] [24:35] evoking [24:36] conversation it's about getting back to [24:38] what we actually want to do as humans [24:40] beings and on this earth together when [24:43] we get together so [24:45] when I think about your question also [24:48] that you're answering with repo health [24:49] is what kind of humans do I want to hang [24:52] out with. [24:53] That's another perspective actually I [24:55] didn't think about that I would say [24:57] whatever I've done [24:59] through is that I've learned a lot from [25:00] people and I would say sometimes [25:03] projects just misunderstand this part [25:06] and by putting the AI in the review and [25:11] people are just struggling and wrestling [25:12] with AI reviews and there is no [25:15] interaction basically human is not in [25:17] the [25:18] review loop and which brings a lot of [25:22] frustration I would say and whatever [25:24] I've learned by just working on projects [25:27] with people with lack of knowledge on [25:29] different areas and discussing through [25:32] and coming up to some solutions with a [25:34] lack of knowledge because I think [25:37] intuition and communication are more [25:39] important than ever in open source [25:42] projects in any area of course technical [25:45] challenges are very important but if [25:47] there is no communication [25:49] why would I work on the project but also [25:52] that's my idea I I would say we are on [25:54] the same page. Yeah Georg I think the [25:57] that comment about like how it's it's [25:59] kind of like what kind of humans do you [26:00] want to work with is is actually really [26:02] interesting cuz I was actually [26:04] maybe independently coming to kind of [26:05] that idea [26:06] I got asked recently by another member [26:09] of the chaos community like what advice [26:10] I would have for people who are coming [26:12] into like open source and I think what I [26:15] ended up writing was something along [26:17] lines of like stay human like keep your [26:19] humanness right like yes many open [26:22] source projects the the front face of it [26:25] is like a git repo with code and it's [26:27] very technical and very the focus is [26:30] maybe sometimes on the code but the code [26:33] is not the community the code is sort of [26:35] just a system that the community has [26:37] built to achieve some kind of goal right [26:40] and so if your contributions to that [26:41] community are just focused on only the [26:44] code you're missing out on all the [26:46] context of why are we building this in [26:48] the first place who are we building this [26:50] for what goal are we trying to achieve [26:53] and all those human interactions and [26:54] human conversations and things behind [26:58] the scenes that may even help you make [27:01] your pull request faster or more [27:03] effectively if you're able to join the [27:05] community and get involved and learn [27:06] about the people first and then learn [27:10] about what they have on the road map [27:11] what's what's planned and fit your [27:12] contribution into something that they're [27:14] already working on rather than just oh [27:17] here's a random pull request for some [27:18] random thing I thought of or my LM told [27:21] me to do or whatever the case may be the [27:24] involvement you have with the community [27:25] is kind of what makes the difference [27:26] between like those two scenarios of [27:29] whether you sort of feel part of the [27:31] community or if you're just kind of [27:33] contributing code and unsure where [27:35] things are. That's really important idea [27:39] and that's the whole point that open [27:40] source is going through I would say for [27:43] example I reviewed one PR that was [27:46] created by whole courser and it was [27:50] mentioned in the pull request [27:52] description at least I respected that [27:55] it's mentioned in the PR description and [27:57] I gave my whole ideas and I gave review [28:00] and I tried to be nice as as much as I [28:03] can and just spread my thoughts about [28:07] the project and about the PR that is [28:10] being created by LLM and I gave the idea [28:14] that you should talk to me and if you [28:16] ask me questions on discord or wherever [28:20] it is [28:21] I would explain and we can complete this [28:24] issue together and sometimes I think [28:26] people misunderstand the point of the [28:29] open source projects and they just [28:31] directly jump on any issue and they are [28:34] coming oh okay that's my time I need to [28:36] contribute to this issue but that's not [28:40] how software is being built it's being [28:42] built with communication and with [28:44] technical skills by getting these two [28:47] skills I would say. What do you see as [28:49] the future of this repo health project I [28:51] think you called out some ideas in your [28:52] blog post like have you kind of been [28:54] working towards any of those do you [28:55] potentially see future collaborations [28:57] with sort of some of the chaos like [28:59] projects like augur grimor lab outside [29:01] of chaos where do you see this going in [29:04] the next maybe 6 to 12 months. I would [29:07] say I I would add more metrics [29:10] and I would reduce some features [29:13] because that's what happened to me when [29:15] I built this project I added a feature [29:18] and spend it and over engineered that [29:20] feature then I realized that that's not [29:22] the whole point of my project I just [29:25] kind of manipulated myself that oh [29:27] that's the feature that I wanted to do [29:28] because as a developer sometimes I kind [29:32] of love the abstraction and the whole [29:34] difficulty of the problem and I focus on [29:37] that and I realized that checking out [29:39] the dependencies is not the whole [29:41] purpose of this project that's why [29:43] nowadays I'm kind of consuming more data [29:46] and grasping more data to bring my own [29:50] thoughts and to bring my own experience [29:53] and represent in a way that could help [29:55] people and could help maintainers and [29:57] also contributors. There's a lot of [30:00] projects like I mentioned Augur that are [30:02] very much have similar goals of we want [30:04] to help the communities, we want to help [30:06] the people engage with open source and [30:08] and do you see any potential for kind of [30:12] working with other communities that are [30:14] also thinking about metrics? Potentially [30:16] like using Augur or using Gramour Lab [30:18] and pulling those that data into some of [30:21] your tools? That's another side of open [30:24] source that I love because I can use [30:27] other tools and integrate into my own [30:30] tool and which kind of helps me. Instead [30:34] of reinventing the wheel, it's better to [30:36] use some [30:38] already structured projects, I would [30:40] say. There was a also another project [30:42] that I used u n g h that really helped [30:45] me a lot when it comes to handling [30:48] GitHub related issues. This is fantastic [30:52] and we are coming up on the end of [30:54] another awesome conversation. I really [30:56] enjoyed this already and so Elshad for [31:01] our listeners who want to get in touch [31:03] with you, who want to follow your work, [31:05] where can they find you online? They can [31:07] check out my GitHub. I would say I'm [31:10] using GitHub as a social media. [31:12] Also, I've started a new activity with [31:16] my friend that by having a podcast on [31:19] YouTube channel and we are [31:21] weekly discussing about our progress. [31:24] Also, my friend's name is Daniel and we [31:27] are making podcast on Instagram. [31:29] Awesome. Yeah, we'll put the links to [31:32] your socials, GitHub, Instagram, YouTube [31:35] into the show notes. Anyone listening, [31:39] please look at the show notes. [31:41] And now we get to the last segment of [31:44] our episode, the value adds where we [31:47] share something that has brought value, [31:50] joy or meaning to our life recently. And [31:54] I can start us off. I am currently [31:59] really enjoying the in between time. My [32:03] work with Bitergia has is concluded and [32:07] maybe I [32:08] coming up on CHAOS board elections. And [32:13] so there's also [32:15] the end of my term as a board member. [32:17] This is my last episode that I'm [32:21] recording as an organizer for CHAOS Cast [32:24] and I am right now exploring what I want [32:28] to do next and I'm just really enjoying [32:32] looking at all the opportunities and all [32:34] the things that are out there and really [32:39] reorienting myself. One of the things [32:42] I'm exploring right now is all the new [32:44] technologies we have with AI and what [32:47] they allow us to do. [32:49] And so yeah, [32:50] being in between jobs, there is certain, [32:54] you know, stress that comes with finding [32:56] a new job, but there's also comes with a [32:58] lot of opportunity and I really enjoyed [33:03] this period of uncertainty and [33:05] exploration. So that's my high right [33:07] now. Who else has a value add for us [33:10] today? [33:11] I can go next. My value add is the [33:14] vernal equinox, which is [33:17] tomorrow or the day after potentially. I [33:19] haven't checked this year exactly which [33:20] day it lands on. And that we just been [33:23] getting so much more light where we are [33:25] in the UK. It's [33:27] further north than many people think [33:29] because we don't have a very cold [33:30] climate, but it is actually quite far [33:32] north and we get pretty short days in [33:35] the winter. [33:36] Not a lot of light. The sun can be quite [33:39] low and you don't get that kind of light [33:41] intensity. Now the sun is [33:44] getting a lot higher, the sunlight's a [33:46] lot, you know, stronger. We're getting [33:48] longer days. We're getting about 12 [33:50] hours of sunlight now and it's really [33:52] nice. It really you feel it [33:54] physiologically like the light makes a [33:56] big difference. So I'm very happy about [33:58] that. I can go next. Actually, my value [34:01] add is [34:02] essentially spending time with friends [34:03] and new experiences. So I last weekend I [34:05] had a chance to visit a very close [34:07] friend in person and we were at a local [34:10] open air sort of food market and I sort [34:13] of was like, "What is that over there?" [34:14] Like it kind of looks like bamboo, but [34:16] it's like way bigger and [34:17] I learned that that is in fact sugarcane [34:19] and so we ended up buying some and [34:21] taking it home and peeling it up like [34:23] taking the outer shell off and and [34:25] trying some and so I think that was a [34:27] really fun experience just to be able to [34:29] kind of hang out with friends, try that [34:31] and also get to learn what sugarcane [34:33] tastes like. [34:35] What I would love to say is [34:37] nowadays, [34:38] sometimes I just do coding without using [34:41] any high level tools, just opening [34:44] complete low level language and doing [34:47] what I really love and solving some [34:49] problems that is very short scoped and I [34:54] love being in this process and I kind of [34:56] miss this feeling for a long time [34:59] because of being too abstract in [35:02] software development. That's where I [35:04] find the joy and yeah, I'm exploring [35:08] kind of sometimes doing or reinventing [35:11] the wheel, I would say. Awesome. [35:14] Well, it is time to say thank you. [35:16] Thank you Elshad for being our guest [35:19] engaging with the CHAOS metrics. I [35:21] really really appreciate you sharing [35:23] your [35:24] experience by writing the blog post and [35:27] being willing to come on to the podcast. [35:30] Yeah, thank you for having me. That that [35:32] was really nice podcast, I would say. [35:35] And thank you Alice and Adrian for being [35:38] panelists today. [35:40] Yeah, thank you so much. Great to great [35:42] to be here and [35:43] happy to be on my first first CHAOS [35:45] Cast. [35:47] Yeah, thanks for inviting me. It was [35:48] fun. And thank you, dear listener, for [35:51] joining us today. [35:53] If you have a topic that you'd like us [35:54] to cover on the podcast, whether it's a [35:57] question you'd like answered, [35:59] suggestions for guests or panels or [36:01] feedback on what you've liked most so [36:03] far, we'd love to hear from you. Please [36:06] email [email protected] [music] [36:08] with your messages [36:10] and you could find that email address [36:11] also in the show notes. Again, thank you [36:14] for joining us today and we hope you [36:16] enjoyed this episode. Until next time, [36:19] your CHAOS Community.  📄 **Full text:** ![[Full Document Contents/Articles/Episode 131 Repo-Health - A Tool Built on CHAOSS Metrics|Episode 131: Repo-Health - A Tool Built on CHAOSS Metrics, YouTube]] ## Highlights - story, Elshad. And how about we start with an introduction? So for the listeners of our community that they can get to know you. Hi, my name is Elshad. I'm a software developer, an open source contributor. Actually, contributing to open source projects is my passion and I love contributing what I do. And I built Repo Health by getting inspired by Chaos metrics. And today I'm really ([View Highlight](https://read.readwise.io/read/01knwvvs4n3nh6f63sewd1047w)) - Tags: 👻 ai highlighted - glad that I'm in this podcast that I'll discuss about how I came up to this point and integrated Chaos metrics into my project. Yeah, welcome. And what part of the world are you from? I am from Azerbaijan. Awesome, welcome. So so glad to have you on the podcast. And what was the journey for you to discover the Chaos project and the Chaos metrics? What kind of sent you down this rabbit ([View Highlight](https://read.readwise.io/read/01knwvvs4r3pyy8c26jpaxy9t0)) - Tags: 👻 ai highlighted - started thinking about that this project is taking care of contributors. And it's basically helping me how to avoid some projects and what kind of metrics I can check in one project if I want to contribute. And if I want to see the whole purpose of the project, I would say, and the health overall health of the project. That's how I came through, actually. And while doing research I did ([View Highlight](https://read.readwise.io/read/01knwvvs53p49d0zma459t5rp1)) - Tags: 👻 ai highlighted - complete the section because I would say chaos metrics is doing really good job by bringing all of this ideas all together, but overall in open source I think it's very complicated to bring all ideas and bring only one nice structure and showing that yes, that's just the way it is. It's kind of not possible to show all leaders one way. So, that's an interesting topic that from the very ([View Highlight](https://read.readwise.io/read/01knwvvs6dcr4hrvtfdd1ntq7j)) - Tags: 👻 ai highlighted