Episode 71: What's New in CHAOSS: Podcast Reboot Episode  ## Metadata - Author: [[CHAOSScast]] - Full Title: Episode 71: What's New in CHAOSS: Podcast Reboot Episode - Category: #podcasts - URL: https://share.snipd.com/episode/84f53222-ac27-46c5-8e47-26a170791514 ## Document Notes ## Highlights - **Context Groups Foster Specific Insights** - CHAOSS developed context groups to gather people concerned with specific projects or communities. - These groups help share health-related concerns and tailor CHAOSS metrics to various open source contexts. Transcript: Speaker 3 I'll go ahead and get started. Thanks, Gary. Chaos has been going on for six, seven years now for quite a while. And over a lot of that time, we've been developing metrics and metric models, which are collections of metrics, software, different initiatives. And I think these are the things that we're going to talk about a little today. So as an open source community grows, it can take on many different forms. And I think we're in a tough stage now where we're thinking about how the things that we've developed in the Chaos Project can really help people think about the health of the communities That they care about. So one of the things that we've done recently in the Chaos Project is develop, we call context groups. And these are groups of individuals, say, at corporate Ospos or at university open source program offices or as related to scientific software. These are people who are interested in the health of projects that they are interested in or they're interested in the health of projects that they're participating in. But they may not have the time or the energy to necessarily participate in the Chaos Project to develop those metrics or those metrics models or develop the tools that are specific to The questions that they have. So these context groups are meant to bring people together in these similar contexts, allow them to talk freely, encourage them to talk freely and openly about the health-related Concerns that they have. And then in the Chaos Project, we'll support the needs that they have with respect to open source community health. ([Time 0:02:56](https://share.snipd.com/snip/0cc60750-7043-444c-ab6d-a75a8833aaf3)) - **Make Metrics Searchable by Context** - CHAOSS reorganized its metrics to be searchable by context or topic rather than by working group. - Users can now more easily find relevant metrics using categories and tags on the CHAOSS website. Transcript: Speaker 2 It's not a metric that we care about. We are chaos. We should care about metrics. Speaker 1 I was just going to add gay or tear point earlier about the way we've kind of reorganized our groups. We've also reorganized the way we display our metrics on the website. So we used to have them listed by working group, but that kind of required someone looking for a metric to know in which group that metric was developed. And it wasn't super intuitive for those who are new to chaos. So we've reorganized them into more of a context or a topic category way and made everything searchable so that hopefully it's a little easier for folks to find what they need. So we have the categories and we also have each metric tagged as well with some topics that might show up on a search for someone that's searching for that kind of metric. ([Time 0:07:20](https://share.snipd.com/snip/ce063211-2837-4d11-813c-672bbd54cb2a)) - **Metric Models Reveal Deeper Insights** - Individual metrics alone often provide limited insight on project health. - CHAOSS metric models combine multiple metrics to reveal phenomena like community welcoming or developer responsiveness. Transcript: Speaker 1 I don't know if Matt, Sean, somebody wants to talk about that. Yeah, the metrics models have been around for about a year and a half now. And we were developing them because people want some kind of recipe for if I want to look at a phenomena in open source software health. Usually you're not looking at a single metric here, putting a group of metrics together. That's where our tools really come into play because that the tools that we have really enable us to put those metrics together for people and let them see the health of their projects In that through that lens. Speaker 3 Any single metric that we've referred to over time as a atomic metric may or may not be useful. So the one I always talk about is like age of an issue as a single metric by itself, the age of or the age of issues may not be very relevant to anybody. But as we bring these together in models, we can start asking questions like say around community welcoming this. So that would be an example of a metric model. And then we would assemble as Sean pointed out, maybe about a half a dozen metrics that might begin to give us insight about community welcomingness or developer response time would Be another example of a metric model. And so then we ask ourselves, what are the these atomic metrics or these single metrics that can be brought together in meaningful ways that can provide insight around say responsiveness Or welcomingness. And like any metric and any metric model, how you understand them is not going to be a perfect insight that you may be looking for. ([Time 0:08:19](https://share.snipd.com/snip/37644fff-6038-4313-b915-5d6963d0f8d6)) - **Data Science Initiative Guides Interpretation** - CHAOSS Data Science Initiative helps users interpret metrics thoughtfully rather than assuming absolutes. - It builds community, offers guidance on tools, and plans best practice guides for meaningful data use. Transcript: Speaker 1 Yeah, absolutely. So this is one of those areas that it is kind of like a context working group, but it's for data scientists or people who are interested in analyzing data. This position was really created within the chaos project. This new director of data science and then the data science community that I've been building around it is really designed to be a little bit more helpful for the people who use our software And metrics. Historically, we've been very careful to provide the tools, write the metrics, and then we'd let people do their own interpretations. We found was that it's really hard to do the interpretations and people need a little bit of help with that. We can't tell them if this metric goes up, it means this definitely, but we can give people some guidance for what to look at. That's one of the key areas that I'm focused on as part of this new data science group. We formed a whole working group around it. We have a Slack channel and we've gotten pretty good participation around it. There are a few things that I've started on for the data science initiatives. One of them is getting better about positioning our software. We have two tools. We have Agar, we have Girmar Lab, which one of them is better in certain situations. We're working on providing people with a little bit more guidance about which software tool might be better for their use cases. That's a work in progress. We've also been working hard to build this data science community. As I said, we've got the Slack channel, we've got the working group. We have a repository where we can work together on things. It's been really interesting to start to see that community coming together, which has been great. I've also been focused on some evangelism efforts. Conferences, how do we promote the work that we've been doing within chaos? As a part of that, we're getting ready to start some best practice guides, things that people can look to to provide a little more information about how they should interpret their metrics. That's another piece of the data science initiative. The final piece is really in understanding some of the challenges that people have had already. We'll use that data to figure out exactly what we're going to work on for maybe next year and the upcoming year. We distributed a survey. That was step one for starting to understand some of the challenges. The survey just closed yesterday. I've created a folder in the repository for the data science working group with all of the raw data. People can see what the data is and we can analyze it together as a community. That's a bit of an experiment. We'll see how that goes. We'll see if anyone else is interested in helping me analyze the data from that understanding challenges survey. ([Time 0:12:28](https://share.snipd.com/snip/124b455c-3e25-4ab8-ad81-8ce6925a13b9)) - **Interpret Metrics Within Context** - Metrics are not absolute indicators; context matters greatly in interpretation. - Focusing on trends over time is key to understanding project health and making improvements. Transcript: Speaker 3 Red, yellow, green. How can we be in there? Speaker 1 Interesting question. It feels to me like there's maybe some happy medium where, depending on the metric that we're looking at, it can mean a lot of different things, just depending on what the data looks like. I think that one way we can help people understand it is to talk about some of the things that it might mean. One of my pet peeves about metrics is that people tend to think of them as absolutes. This goes up, that means this, this goes down, that means bad. The reality is every single open source project is different and you really have to interpret every metric based on what's going on within your community. There are no absolutes, but I think what we can help people do is understand how to interpret the data in light of what's going on in their project. You may see a dip in July because you've got a whole bunch of European developers who are all on vacation. That's not bad thing. It's a good thing, right? People are refreshing. Not going down is not bad for the project. It's probably healthy for the project overall. I think that there are things that you can think about that help you decide whether something is good or bad or needs improvement. The other thing that I really try to get people to focus on, and it's something that I talk a lot about when I interpret metrics, is trends. ([Time 0:16:23](https://share.snipd.com/snip/c0d09657-64bc-4166-870c-a885ad18eae2)) - **Use Metric Models for Big Picture** - Use collections of metrics, or metric models, to get a broader picture of project health. - Start with best practice guides around metric models to help community understanding. Transcript: Speaker 1 Yeah, I think that's exactly it. And I think another piece of it is focusing a little bit on the metrics models or at least collections of metrics, because in a lot of cases that can help you understand the overall big Picture more than more than just looking at a single metric. So I think we'll probably start with some of the best practice guides around some of the metrics models to help people understand how they can use different metrics together to get a Better picture of their community. ([Time 0:18:25](https://share.snipd.com/snip/94c51a6d-bad4-49f1-850f-4bb5cc4e4d0f)) - **Help Communities Improve with Metrics** - CHAOSS can offer guidance on how to improve community health based on metric interpretation. - Although no universal recipes exist, CHAOSS helps projects ‘improvise in the kitchen’ of community management. Transcript: Speaker 1 Do you think that that's a role that chaos should play or maybe be the center for those conversations or is that way outside the scope as you see it? That's a really good question. I actually do tend to do a lot of that suggestions when I'm looking at interpreting the metrics. So I put together, we have a starter project health metrics model that I ran for a particular project for a particular person. So I can't share the details about that particular thing. But there were a number of places in that report that I put together for this project that were, this is happening and you should think about doing this as a result. So I do think that part of the interpretation is not just the interpretation, but what it means to improve it or some things you can look at for improving it. I think one of the nice side effects of people who've been working with chaos a while is that we understand how to evaluate different communities and interpret their metrics because We've had to over time. ([Time 0:19:10](https://share.snipd.com/snip/680010d6-10cf-4dfb-9b41-611a6c0b4f73)) - **Auger Scales Metrics Collection** - Auger software incorporates multiple data sources to scale metrics analysis across 100,000+ repositories. - They are releasing a GraphQL API to enable dynamic and flexible data access. Transcript: Speaker 2 We today on the software side. And Sean, maybe you can talk a bit about auger because when chaos started, we also started auger. I still remember the conversations about what should we name this thing? It coincided with chaos. Speaker 1 Auger got created. It was actually called GH data back in the day because it was looking at we were using the GH torrent data at the very beginning. And as it advised auger evolved, we incorporated Brian Warner's schwesad project. Brian Warner is former Linux Foundation leader and was working at Samsung at the time and had created this software called schwesad which mined Git repositories. And so we've rolled that into auger for all of our commit counting. And since then auger has evolved significantly, primarily focused being able to quickly explore new metrics or create new metrics. And over the last year, year and a half, auger has gotten really good at capturing large volumes of data. So the public instance of auger, which you can see at AI.chas.io has over 100,000 repositories in it right now. And you can visualize those red hat has created a project called OSS dash aspen slash eight of the number eight not. And OSS ashman slash eight not is at metrics M E T R I X dot chaos that I owe. And that is a dash plotly dashboard. So it gives folks a way to see metrics pretty quickly in a format that they can understand. And you hear this month, we're kicking off an API development project. The the auger core team has recently implemented and we're currently testing graph ql implementation of our API, which will enable folks to much more dynamically get any data that They want out of the auger instances that we have either publicly or ones that folks spin up. ([Time 0:21:57](https://share.snipd.com/snip/b16ee62e-e7dc-4893-8355-935a9e1c000b)) - **Grimor Lab Enhances Scalability, Identity** - Grimor Lab improved scalability with multi-tenancy and migrated from Elasticsearch to open search due to licensing. - Sorting Hat helps unify contributor profiles across platforms with a user-friendly interface. Transcript: Speaker 2 Yeah, I'm happy to share what we've been working on in grimor lab. There are two, maybe three big things. One that we've been working on for a while is multi tenancy and allowing the data collection, creating one common data pool, but then for different projects, different groups, different Departments to have their own dashboard on only see the data for the projects that they care about. And so scalability is something that has been in the works in grimor lab and optimizing the processes, the enrichment process. There's a lot of improvements that have been made and there's still more improvements that need to be made. So that's an ongoing work. Another big topic of the last year or two years really has been the migration to open search. So grimor lab was built on the elastic and kibana stack. And that was to not reinvent the wheel when it comes to visualizing and having a dashboard solution to explore the data that we have about open source project. And so grimor lab is really focused and honed in on collecting high quality data. But then elastic changed its license. And as a commitment to open source, we did not upgrade to the new elastic search version, but migrated to open search, which is still on an open source or as I approved license. And that migration is finally complete. So when you go to chaos.bitchurch.io, this is grimor lab in action and showing data about the chaos project. We finally have the new open search interface. I really like how it looks. It feels great. And then the other big update is the sorting hat. And this has been in the works again for years when sorting hat is the module within grimor lab for managing identities. So as people are participating in open source, they have different usernames on GitHub and GitLab and on mailing list, they use different email addresses and being able to merge all Of that into one profile, say, hey, this is all the same person. That's what sorting hat does in grimor lab. And we finally have a really beautiful, easy to use interface with drag and drop capability and a workbench where we can put an identity to use again later. And there's some recommendations for which identities might be the same person. So easy to merge. Anyway, there's a lot of improvement there. And I really enjoy using that new interface. ([Time 0:24:22](https://share.snipd.com/snip/452d230c-7d60-4938-ba8a-98c0d5422d7f)) - **Newcomers Energize CHAOSS Growth** - CHAOSS Slack community grew from 200 to over 1,600 participants in a few years. - Newcomers, especially from Africa, drive initiatives including badging, onboarding, and project management. Transcript: Speaker 1 Just for a point of context, about this time in 2021, we had roughly 200 folks in our Slack channel. And this year, right now we have over 1,600 folks in our Slack channel. So we've experienced massive growth in the last hours. And so it's been a great and also a little challenging to just to make sure everyone feels welcome and that they can find their way. So one of the things we've looked at doing is, first off, we wanted to create some local chapters to help people connect locally with each other. When we started with our Chaos Asia Pacific group, which was primarily in China, a few other folks around, but primarily in China. And that's great and it's still going on. It has slowed down, I will say, somewhat, but we also had then started a chapter in Africa. So we have Chaos Africa and that's led by the amazing Ruth, IKEA. And what's really interesting is that a lot of the folks who have come to the Chaos Africa community have also come to open source as newcomers as well. They are not familiar with open source at all. Some folks do have technical experience, just not in an open source setting. And others are brand new to the whole scene. So it's been really interesting journey for Ruth and her community there. It's been great as they've grown to almost 450 people in the Chaos Africa community, which is amazing. And a lot of those folks have actually ended up driving a lot of the initiatives within Chaos, which is also amazing. Even though they're newcomers, they're excited, they're enthusiastic and they have great, fantastic ideas. You'll see them, you know, driving initiatives like our program project management group. And they do a lot of work on our badging, which I'll talk about in a second. And they also do other things around like our Slack bot for newcomers and they're helping with onboarding. Like they are amazing folks. So we're super happy to have all of those folks joining Chaos. And that's actually kind of led us to look at a few other areas where open source is rather new. It's kind of a risk on Chaos as part to invest a lot of resources and folks who are new to open source. But it's a risk that is absolutely worth taking because those folks, they have such fresh ideas and they're enthusiastic. Like I said, and they have a lot of energy and they bring such a fresh perspective to our project. So we are super happy to be investing in those folks. One of the areas we're looking at growing is in Latin America. And we have a wonderful woman named Sally Yang, who is leading that initiative. And then the second group we are starting as well as in the Balkans, which is mostly southeastern Europe. That's led by another wonderful woman named Christie Progree. And so our hope is to really learn a lot about these regions and help them help Stella and Christie introduce open source in a good way through chaos. Or maybe they don't even, those folks don't even end up contributing to chaos. Maybe they go on to a different open source project, which is also completely fine. We're just happy to have them joining open source in general. So we have our new local chapters that we're starting. And like I said before, the folks in Chaos Africa have done a whole lot of work on our badging initiative. So for those who don't know, we have two badging initiatives. One is for events and one is for open source projects. And basically those initiatives are designed to help bring our DEI metrics into the real world and make a real impact. So our DEI event badging, we have 29 active badgers now. We've badged around 140 events. It's still continuing to grow. We're going to keep adding more metrics for event organizers to keep thinking about, just to make it a little harder for them to find, for them to achieve a bad. But a lot of those that work is done by folks in Chaos Africa. So really appreciative of their contributions there. And for those who also have not heard, we have been partnering with a group called All In to develop badging for open source projects, which is a little trickier just because of the scale Issue. And right now we have a lot of manual work that goes into reviewing and approving those event applications, but open source projects are a there's a lot more to those. So so we've had some folks working on doing some automation to help make that process a little more scalable. I will also say that because there is so much going on in Chaos onboarding and making sure that our newcomers have an exemplary experience is a challenge for us. We've got a few initiatives that we're trying out to see if we can make that experience better for folks. What's great is we can enlist the help of the newcomers in building those initiatives out because there's nobody better equipped to know what a newcomer needs than a newcomer themselves. So we're super happy that we can include them in those contributions. A lot of newcomers come to a project. They feel like they don't know anything. They don't know where to go. They don't know how to contribute to a project. So we're really happy that we can help bring those folks in and that they have the most experience of being a newcomer. So they're the perfect folks to actually contribute to this experience and this process. And we've implemented things like tour guides, which help folks have a one-on-one person to help them find their way based on what their skills and interests are. We are working on some onboarding courses, which will be kind of like a learning management system where they will be different modules people can read through or videos to watch that Help them learn about Chaos and what the different moving parts are here. We have newcomer Hangouts every week so newcomers can come meet each other, ask questions in a super informal low barrier. It's like a super chill meeting. You just come and hang out with us and ask whatever questions you want. We also have a newcomer Slack bot that we have implemented in our Slack, which again is another project that the Chaos Africa folks work on quite a bit. So they are helping us make that better for all newcomers, which is fantastic. And then the final thing I just wanted to mention was we have started this group for project managers. That's a skill that is so desperately needed in open source and often not really appreciated, I don't think, or included in the contribution list of possibilities. So I'm really happy to see that we do have some project managers here that really want to help us with all of our sub projects that are going on, the Badging bot, the Badging website, the Slack bot, an accessibility audit, just helping clean up repose and documentation. There's so much going on. So to be able to enlist the help of actual project managers that know how to do these things and how to distribute tasks and make sure that the project keeps going and that there are issues For newcomers to find and contribute to is really, really great. And I'm so excited that they're here and willing to contribute to our project. ([Time 0:33:35](https://share.snipd.com/snip/a63ab324-83fb-414b-92c9-3fcd6d6d4d85)) - **Empower Newcomers to Onboard** - Engage newcomers to design onboarding as they best understand newcomer needs. - Implement tour guides, onboarding courses, hangouts, and bots to lower barriers for contributors. Transcript: Speaker 1 We've got a few initiatives that we're trying out to see if we can make that experience better for folks. What's great is we can enlist the help of the newcomers in building those initiatives out because there's nobody better equipped to know what a newcomer needs than a newcomer themselves. So we're super happy that we can include them in those contributions. A lot of newcomers come to a project. They feel like they don't know anything. They don't know where to go. They don't know how to contribute to a project. So we're really happy that we can help bring those folks in and that they have the most experience of being a newcomer. So they're the perfect folks to actually contribute to this experience and this process. And we've implemented things like tour guides, which help folks have a one-on-one person to help them find their way based on what their skills and interests are. We are working on some onboarding courses, which will be kind of like a learning management system where they will be different modules people can read through or videos to watch that Help them learn about Chaos and what the different moving parts are here. We have newcomer Hangouts every week so newcomers can come meet each other, ask questions in a super informal low barrier. It's like a super chill meeting. You just come and hang out with us and ask whatever questions you want. We also have a newcomer Slack bot that we have implemented in our Slack, which again is another project that the Chaos Africa folks work on quite a bit. So they are helping us make that better for all newcomers, which is fantastic. ([Time 0:38:08](https://share.snipd.com/snip/0ddce0e0-dadd-48ce-9e29-1fc542022866)) - **DEI Reflection Boosted CHAOSS Growth** - Conducting a DEI reflection project was crucial for CHAOSS to uncover blind spots and grow inclusively. - Community diversity and ongoing learning remain central to CHAOSS culture. Transcript: Speaker 2 Is there something that other communities can also do? Speaker 1 That is a great question, Georg, and I will let others jump in as well. But I think that going through that process was absolutely crucial to chaos because I think we have blind spots. And I'm sure we probably still do. But as we grow and keep our community diverse as possible and include folks with fresh perspective, I hopefully we will not have as many of those. But going through that process with an Ashley, Stella, and Christie were on that team and now they're parts of chaos. So that's fantastic. We're super happy to have them be chaotic now. They are experts in their field. So yeah, we're really happy to have them around. But I think we learned a ton and Matt and Sean, you can also jump in. But I know I learned a ton from them just about how communities work. And you know, and I've been in OpenS we're so long time and I've been a community manager for a long time. And I'm not going to say that. I mean, there's always things to learn always, always. So I'm extremely happy that we went through that process. And we also did do our community survey, which did give us some insight onto how our community was feeling. And some of those recommendations were things like making sure the newcomer experience is good. That was a common theme of like, you know, keep that piece going. So that's exactly why we've done so much work in this area. And another thing was trying to make sure that everybody felt connected because that is one kind of drawback to having the local chapters is that it's easy for us to feel siloed at time. ([Time 0:42:23](https://share.snipd.com/snip/2bce1a80-2788-40f6-a805-3e65c3b8c028))