Episode 265: Sean Goggins on Sustainability Through CHAOSS  ## Metadata - Author: [[Sustain]] - Full Title: Episode 265: Sean Goggins on Sustainability Through CHAOSS - Category: #podcasts - URL: https://share.snipd.com/episode/41befc1d-23f8-4aae-956d-4c9af4fc60be ## Document Notes ## Highlights - **Distributed Leadership in Open Source** - Open source projects mostly use distributed leadership instead of single central leaders. - This model allows communities or organizations to govern projects collaboratively. Transcript: Richard Littauer Your work. Is that your main focus? And if so, what part of it? Open source is my primary research focus for sure. Sean Goggins And in my case, it goes back all the way to 2010, which is 15 years ago now, which is really hard to believe. It began when a new PhD student, Nora McDonald, who is now, I think, at James Madison University, came to my office and said, hey, have you heard of GitHub? I've really started looking into it. And there's all these interesting characteristics of it. And I'd like to do a study. And so we started doing some research together on open source, really exploring how different kinds of projects in the open source universe operationalize different forms of leadership. And we looked at highly centralized leadership projects that welcomed new people in, projects that sustained themselves by isolating a core set of contributors and accepting very Few contributions from the outside. And just trying to understand this through the lens of a lens we ended up selecting was something called distributed leadership, which I think exists in most of open source, where there's Really not often, not never, but not often a single individual who drives an entire project. Leadership through most successful open source projects is distributed across many people. It's governed by a community in many cases, if not a community in the cases of projects that have a very centralized model that don't accept a lot of contributions. There's usually an organization behind it or some kind of corporation that's really trying to drive a specific agenda with it. And often these projects are UI frameworks where they're very particular and they have to operate in a certain way and there's not a lot of room for creativity or learning the whole ([Time 0:02:44](https://share.snipd.com/snip/46e7ff97-57c6-408d-8eb8-70a2b01184e6)) - **Founding of CHAOSS Project** - Sean and Matt Germonprez co-founded CHAOSS after meeting and collaborating on open source data research. - They launched CHAOSS under the Linux Foundation in 2017 focusing on metrics for open source health. Transcript: Sean Goggins Distributed leadership with Nora McDonald led to chaos. Kind of interestingly, it was, I would call it a circuitous route. I was at Drexel University when I worked with Nora and I moved on from Drexel University in 2013 to come to the University of Missouri. I had this idea of I wanted to create a data science program for undergraduates. I ended up creating a master's degree instead. That's a separate long story. And getting involved in chaos was really Matt and I were collaborating. We met in 2012 at a workshop at the University of Maryland and began working on a project that ended up getting funded by NSF, and it was called the Open Collaboration Data Exchange. OCDX is the now dormant organization on GitHub. And our idea was, in the context of research, there's a replication gap in the conduct of research. And our aim was to define not only the data that is behind a paper, but also all of the software and the versions of the software that were used to produce a particular scientific result. Recognizing that if I used version 2.1 of something, I could possibly get a different result than if I used version 2.7, in that keeping track of all of these versions and what ended up In a paper was an important piece of replication. And so we ended up having a series of workshops and created a series of repositories to try to articulate just examples of how a researcher might catalog that full pipeline, including The data and the software. And in the course of that work, we were both obviously also working in open source to different extents. And Matt and Georg were actually at the community summit for the Linux Foundation, which has also been called the Maintainer Summit. And I think the Leadership Summit at various times, it's had a number of different names. I think it's called the Leadership Summit now. And it was a Linux Foundation event. It still exists. And there was a Birds of Feather session pertaining to metrics that I think was Georg's idea. And the Birds of Feather session got a just tremendous response. And I can still recall, it was about this time of year, and I was on spring break. And Matt called me and said, we just had this workshop with Georg. And people were so interested in it. It kind of builds on the things that we were doing with the OCDX, but more of a software part of it, really less concerned with the data, more concerned with the software. And we got really excited. And I think within a few months, we had a project. I think we were the 59th project in the Linux Foundation, which right at this stage, I think is over a thousand projects. So we were early on in the Linux Foundation's project blooming and it kind of came together. We formed ([Time 0:04:50](https://share.snipd.com/snip/bf0cbb5e-42cd-4aa1-9346-afbd4c9c4420)) - **Grant Enabled CHAOSS Focus** - A Sloan Foundation grant allowed Sean and Matt to dedicate significant research time to CHAOSS. - They initially focused on agreeing on standard metrics to measure open source project health consistently. Transcript: Richard Littauer Involvement are you planning everything are you the note taker are you the ideator how do you see yourself as being part of chaos while also having this other full-time role? That's changed over time. Sean Goggins Later in 2017, Matt and I got a one-year startup grant from the Sloan Foundation and just helped to support chaos. That enabled Matt and I to spend some of our research time on the chaos project. So as a professor, the way they usually characterize it on the tenure track is you have 40% of your time for teaching, 40% of your time for research, and 20% of your time for service. And chaos became about 30% of my overall time in that research bucket primarily. So the Sloan Foundation enabled Matt and I to engage fully in the creation of the project. And early on, we shepherded a lot of things and tried to create guardrails so that meetings didn't go completely in random directions. Tried to keep people focused on the, at the first, the generation of just a series of standard metrics, because the gap in open source at the very beginning was that everyone wanted to Understand health and sustainability. Many people recognized that metrics were part of how we got there. But we also knew that, for example, measuring a commit, depending on whose software you used or who you talked to, what you count as a commit could be different. Do you look at the micro-level commits? Do you look at whether you have squashed commits or not? How you look at lines of code? Some people counted all the lines. Some people looked at lines added and lines removed. Some people dispensed with all the things that might be white space. So there were many different ways of measuring these common open source constructs. ([Time 0:08:46](https://share.snipd.com/snip/ce17aa14-467a-4191-9acd-85ea28cc22b4)) - **Teaching Open Source Workflow** - Sean teaches students using CHAOSS and Augur to understand open source collaboration and workflows. - He emphasizes pull requests and Git/GitHub collaboration as key teaching elements. Transcript: Richard Littauer Coming out of it. Now, what you haven't mentioned yet is teaching. So you obviously have this data science master's that you run and you also teach people about engineering in general. Do you ever teach your students about open source in particular or about any of the chaos metrics? Sean Goggins When I was teaching software engineering, which I haven't in about two years now, because it's really kind of a side course in a computer science curriculum. So it's in many cases, like algorithms, there's a three-stage procession through the algorithms courses, and one builds on the other. Software engineering is really the first course that a student in our CS program and in most CS program will take that has an ethic where collaboration is not cheating. So in most CS curriculum, students are expected to do the work on their own. When they work together, that's called cheating. And I was the one who introduced them to this notion that you could work with people and share code and build it together. And that's not cheating. That's kind of the point and what the rest of your life is mostly going to be like. Going back 12, 13 years, when I first started teaching things, I would have to introduce students to Git and GitHub. They weren't familiar with it at all. So it would really begin with, this is Git, this is GitHub. Now my students come in, they've already got GitHub accounts, they're familiar with Git, and so it becomes a process of showing them how to use it to collaborate. So I don't have to start with the bare bones, I start with, you've all used Git, but you've probably never done a pull request, and almost nobody has actually done a pull request into a Project. And so we go through the different ways that you can collaborate using github or gitlab we go through the differences between branches and forks and how you facilitate collaboration And what kinds of collaboration are supported with each approach and so i actively used auger as kind of an example toy project in a sense where they'd go through they'd learn how to install It they'd learn how to operate it. That also familiarized the students with how do you engage with open source? How do you participate in a project in a way that makes sense? Because for students, open source is a little bit of an invisible black box. And so I was kind of peeling back the layers of the onion so that they could see how it worked, not just technically, but also socially. So they could get feedback from pull request reviews, understand what that looked like. And then I would kind of sell it to them ([Time 0:15:10](https://share.snipd.com/snip/0249aafd-b78c-47d7-8cbd-018a2fa6e610)) - **Newcomers Drive Sustainability** - Newcomers have an outsized impact on open source project sustainability. - Acceptance of a newcomer's first pull request strongly predicts their continued contributions. Transcript: Richard Littauer What chaos community has focused on, understanding community socialization, understanding communities of open source, and then trying to identify how we can learn about them. Do the metrics help make projects more sustainable? Sean Goggins I think they do. I think they do because when you're starting to contribute to open source, you probably pick a project. The project that you pick, it has an influence on whether or not you stick with it. And we know from research that we've done, research I've done with one of the students I had a long time ago, who's now a rock star, far bigger than me, Kelly Blinko at Auckland University Of New Zealand. She is way more accomplished than I am at writing about open source and software engineering. One of the things in one of the papers that we did, like, I think in 2016, is we looked at how do people figure out what project to start with? And turns out people follow the contributors or the maintainers who have the most stars on their project. The stars matter for how people pick a project. And then we know from another piece of research that we did that whether or not your first pull request is accepted has a significant influence on whether or not you ever do a second pull Request for that project. So if your first pull request is accepted, then you're likely to contribute a second or a third. And so you sustain yourself through that. You know, the projects can build a maintainer core by finding ways to accept pull requests. We know from work that James Howison at the University of Texas has done that small pull requests are significantly more likely to be accepted than giant pull requests. So the likelihood of a pull request being accepted varies inversely with the number of files and lines of code that are changed. And that's because the bigger the pull request, the more things in it, the more work is pushed onto the maintainer to try to analyze, code review, figure out what the effects of the changes That you made would be. And maintainers, as we know from other research, they face balancing pressures of knowing how much time to spend with it. So we know all these things. When we think about sustainable communities, newcomers have a phenomenally huge effect. I would say there are what I would call leading indicators and trailing indicators. And I think a leading indicator for community health, this is my opinion. I've got some research coming out ([Time 0:20:46](https://share.snipd.com/snip/2f099870-174e-4ddc-afb8-adb6feb4c1dc)) - **CHAOSS's Welcoming Community** - CHAOSS community is notably welcoming and inclusive, avoiding conflicts seen in other projects. - This positive environment helps foster diversity and sustained participation. Transcript: Richard Littauer And let me tell you, they are not all as welcoming as chaos. So it's really awesome. I guess a question that I haven't asked a lot of chaos people before is where has chaos not succeeded? Where is it not overwhelmed your expectations? What's been difficult? ([Time 0:25:14](https://share.snipd.com/snip/03742c06-8903-4266-a5f5-4cba699c6017)) - **Growing Importance of CHAOSS Software** - CHAOSS software usage has grown as organizations demand metrics storytelling for open source investments. - Software tools like Augur help translate metrics into actionable insights for sustainability. Transcript: Sean Goggins What people want from the project is not pure software. I think over the last three or four years, the software has become more important because people have wanted to produce metrics at scale and you need software to do that. So I think Grimoire Lab's seen a boom in interest in the products that they sell through Baturgia. And I know that there've been a growing number of contributors to Augur and users of Augur, again, because organizations, they need these metrics and because they need the data to tell A story because most C-level or director level executives at companies aren't going to sponsor something unless they understand the return on that investment. They're not going to put a software engineer into a project if they don't understand the return on investment. And you need metrics to tell those stories. And sometimes in organizations like Indeed, where Dwayne O'Brien created some really great metrics for storytelling inside that organization, you have an appreciation and a value Of what that storytelling gives you. And it goes on. So it's not like a one-time thing. In other cases, Augur has been used in a number of, I can't say names, but we've been used to sort of analyze the open source assets of companies that are going through mergers and acquisitions Or considering mergers and acquisitions. And so that becomes a one-time analysis that people need to see in order to evaluate the suitability of a mate, if you will. And so sometimes it's sustained and sometimes it's not. So two different use cases, both valid uses of chaos. And I think because the metrics themselves are defined and how we used them, how they're to be used in different ways has matured, especially through the work of Don Foster and Elizabeth Barron. We have more interest in the software because that's where people come to chaos kind of knowing what the metrics are now. And they're able to figure that out because we tell that story well. And then they often just want to jump to, all right, let me see those metrics for my projects. And that's where the software comes in. Richard Littauer How much do you ([Time 0:25:43](https://share.snipd.com/snip/d4c1de3d-bf8a-41ad-a810-ff1e0f1150ee)) - **Contextualize Metrics for Improvement** - Interpret open source metrics in context of the project's goals and roles rather than labeling health with colors. - Focus on understanding project boundaries and areas for actionable improvement, not just summaries. Transcript: Richard Littauer So if I'm just a generic project and I come to chaos and I say, hey, I want to be a better project, you could throw me a lot of metrics, but then I'm just looking at a dashboard. How do you help them interpret them? Sean Goggins How do we be a better project? I think it's the same kind of experience you might expect if you went to a family reunion and opened with, okay, how can we be a better family? It's almost really, you don't want to approach it that way. You want to approach it like, here's where we think we are and here's what we think we need. Are we right? It's like they're in the dark and they just want to know where the walls are. So we help them find the walls that exist for that room. And those focuses are very different from project to project. And even within projects from year to year, what people want to understand is different. And the one thing that we've not done, but we've been asked to do a lot is to sort of give folks a dashboard, like this project's red, this project's yellow, this project's green. And we haven't made those value judgments because if you think about it within a family, I've got five living siblings and I might give each of them a color, but I might not tell them what Color I'd give them. Like my brother, who's a Trumper, is probably a red. But I'm not going to go to the family reunion and dwell on that, right? I'm going to go to the family reunion knowing, okay, this is the boundaries of where this person is loved and productive in the family. And I don't need to put a color on them. They can just figure it out. It's that way with open source projects where depending on your perspective and your role, you might have different specific things that you would focus on or change. And sometimes a project is aligned around what those things are. And sometimes there are just different corners of a particular project that want to understand different pieces of it, if that makes sense. Richard Littauer It does make sense. It does feel like there's an opportunity for chaos ([Time 0:28:01](https://share.snipd.com/snip/9417e7ce-6192-45d9-903d-b1dde8280669)) - **Supporting Scientific Open Source** - Scientific open source projects often lack formal support structures like foundations. - Mentoring and shared organizational support, such as from Chan Zuckerberg Initiative, help sustain these projects. Transcript: Sean Goggins A bit more than elsewhere right now is to begin to just understand what you need to do to sustain the project because those kinds of open source projects, the scientific ones, have a more Cyclical level of contribution that often varies according to the funding that they get. And we try to, and let me give you an example, like in the corporatized open source community and the Linux foundation, everybody understands that having a Linux foundation there means That I don't have to deal with lawyers to get my governing board structure set up because they have a template for that. It means that I don't have to figure out how to pay for Zoom because I have a central account for that. It means that if I get small amounts of money that can be used by the community, I have an accounting system at the LF for that. So as a corporatized project, there's all this administrative and legal work that occurs through the Linux Foundation that removes ancillary work from the perspective of open source Contributors and maintainers and just makes it templated. In scientific open source, there's none of that. In fact, most scientists don't want to be part of a foundation. They don't see the value in that kind of help. And in many cases, not all, building a community isn't something they're interested in. They just want to kind of keep the scientific software that they built for their purposes going far enough into the future so that they can continue to use it to produce papers, which Are really the coin in the realm there. And so in those cases, we've built mentoring programs and we're continuing to build mentoring programs. We've collaborated a good deal with the Chan Zuckerberg Institute over the years, probably going back six years, because they sponsor a lot of scientific open source projects that Are essential for the conduct of science, but they don't have any other support structure or funding stream. And so they've helped to sort of shore up core things. And I'm talking about like SciPy and NumPy and visualization libraries like Boca. CZI has done really good work just kind of shoring up the peers in these pieces of scientific open source with some funds and a shared organization where they can talk with other scientific Richard Littauer Open ([Time 0:31:12](https://share.snipd.com/snip/a956e15a-e5db-424e-aeda-1eabf216eb05)) - **Future of CHAOSS Data Sharing** - CHAOSS is focusing on expanding data sharing infrastructure to make open source data more ethically shareable and accessible. - They aim to build mentoring programs and ease data constraints to support wider research. Transcript: Sean Goggins We have a project board that kind of spells out what we're doing. We also have a website I'll send you a link for. Matt and Dawn and Elizabeth and Raya Zain, who's a new collaborator of ours. We're working on building mentoring programs for scientific open source. That's a big part of where we're going. Just today, Matt and I kind of decided on kind of what we want to accomplish in the next four years through chaos, which is really, I think, the use of Augur and Grimor Lab data and the sort Of building data sharing stream that satisfies, because there's so many people who want to make sense of these projects right now on the projects within foundations, in the academic Research community that still have a hard time getting at the data. So we want to create an environment that is less constrained than the one that we currently operate in. And it's really constrained by the way that and I work with the university is we have to have something called an IRB, Institutional Reboot Review Board. It's really important from an ethics perspective. It protects individually identifiable information. It protects individuals, et cetera. And obviously, for sharing open source data, there's work that needs to be done to make it more shareable so that more people can look at it and examine it and make sense of it. And I think that's kind of where we're going is this data infrastructure, ([Time 0:35:20](https://share.snipd.com/snip/b8bd5fd2-392f-40be-bbd2-07727ea55818))