Episode 76: CHAOSS Goals for 2024 and Beyond  ## Metadata - Author: [[CHAOSScast]] - Full Title: Episode 76: CHAOSS Goals for 2024 and Beyond - Category: #podcasts - URL: https://share.snipd.com/episode/65406161-6a46-4e61-8c70-8d2463939b8b ## Document Notes ## Highlights - **Guided Insights from Data Science** - CHAOSS is evolving from metric agnosticism to providing insight guidance through context-specific insight guides. - These guides help projects interpret metrics uniquely based on their community needs and improve over time. Transcript: Speaker 3 Have. And that is how do we provide guidance to use it. The goal is to use data driven insights to provide recommendations that help people generate new insights for their communities. And Don as our data scientist, what are your thoughts on this goal? Speaker 1 Historically the chaos project has really taken an agnostic approach to interpretation of metrics. So we historically have given you lots of metrics. We've given you some tools with the idea that then it was really up to the individual projects, ospos, whatever to determine how to use those metrics and how to interpret them in light Of their community because every community really is a little bit different and you do need to interpret the metrics based on what's going on within your project and community. What we found is that it's a lot of metrics and the software is complex in ways that make it really powerful but that can make it really hard for people who are relatively new to using metrics To actually generate those insights. So the whole goal of this data science effort, which I've been brought into the chaos project to drive, is really around helping people figure out what they need to know to generate those Insights. So we have a few specific things that are part of this new data science goal. One is collaborating with some of the context working groups that we have. So we have one for resource program offices, we have one for academia and universities, we have another one for scientific software. They all have very different needs. So this work is partly around understanding what those needs are as well as then helping people interpret the metrics in a way that makes sense for what they're trying to do as an organization Or as a project. Another piece of this that I've just started working on is to create a series of what we're calling insight guides to help others interpret their metrics and generate insights. So the idea behind these insight guides will be talking about a specific aspect of a community and which metrics you can use, how you can interpret those metrics and how you can take that Into making changes that will improve your community and then how do you monitor those changes over time. So we're starting to work on those insight guides so you'll see more about that in the coming probably first quarter of next year. And then the other part is kind of aligned with some of the other efforts that we did Nicole and Elizabeth talked about earlier which is building use cases with communities or other entities As examples of how to get insights for your data because I think one of the things that really helps people interpret data and know what to do with metrics is to see examples for how other People have done this. So they can look at an example of maybe a company or an organization or a project that is sort of similar to what they're doing and see how other people have interpreted their metrics and What they've learned from them. So I think having those use cases will also help people get more insights from their data. Speaker 5 One of the things that's going to be really interesting to me, Don, in the work that you're doing as director of data science is completely agree we've been agnostic towards the metrics For so long. And then if we go all the way to the other end of the spectrum, it would be giving like you're doing great, you're not doing great kind of advice and clearly we're not going that far because We would never be able to provide that type of advice. So it'd be really interesting to see the steps you talked about are kind of those first steps in moving us off being agnostic and just kind of how far we can go down that path. And I don't know the answer and I maybe you do. I don't know. So it'll be really interesting this process of discovery as to how far we can go down that path. Yeah. Speaker 1 And that's one of the things that I'm working on with this insight guides, which is why it's kind of taking me a while to make traction on ([Time 0:25:52](https://share.snipd.com/snip/1b448ea1-27e9-4dcd-98a8-2aea92e117b6)) - **Rethinking Vanity Metrics** - Traditional vanity metrics like stars and forks correlate strongly with actual software usage. - New data insights encourage rethinking these metrics' value in assessing open source project health. Transcript: Speaker 3 Metrics that we have. And Sophia and I just were talking about a little study we did looking at project health metrics from chaos and how they correlate with usage data from telemetry, which is very hard to Get in open source. Very few projects have telemetry data that show us exactly how many people are using the software and how. And being able to correlate that and see how these other metrics that are publicly available and that we can collect already easily on any open source project, correlates helps us rethink. And one of the things that fascinated me is how stars and forks as metrics. And we've been kind of dismissing them as vanity metrics. They are like, oh, they only tell us how well we do marketing for our project. They don't actually tell us anything about the community. Well, we saw that they were very strongly correlated with the actual usage of the software. And so I'm really excited to see more of these data driven insights and how we can rethink the metrics that we already defined. So with that, I'm super excited for goal ([Time 0:33:30](https://share.snipd.com/snip/1b00ca8a-65c7-4f51-ad13-729550554f22))