Episode 73: Reflections on the LF OSS Maintainer Report  ## Metadata - Author: [[CHAOSScast]] - Full Title: Episode 73: Reflections on the LF OSS Maintainer Report - Category: #podcasts - URL: https://share.snipd.com/episode/6d8b29c0-949c-4983-9e42-720d3342057c ## Document Notes ## Highlights - **Qualitative Depth Over Sample Size** - Thirty-two in-depth interviews can yield convergence on key topics despite being a small sample size. - Qualitative interviews provide rich insights distinct from large-scale surveys. Transcript: Speaker 4 I can start as practitioner and not academic, I was really curious to understand more about the methodology from the perspective of people here. What struck out to me was that there were only 32 interviews. And so I'm curious to know, is that enough interviews in order to get like the general best practices and guidelines that we'd see further on in the research? And not only were they just 32 interviews, maybe it's not just maybe 32 is significant number, but also it was such a broad range of projects and types of communities. So while they were all considered critical infrastructure projects, according to the census, that's how they were surfaced, it was hard for me to understand and gauge the validity Of the findings without knowing if this was like common practice in general research. But anybody who has a degree here or a ResRed academic paper can answer. Speaker 3 I have the other thoughts on this. They did 32 interviews out of 200 of these critical infrastructure projects that they identified. So 32 out of 200 is actually not bad, and these are all pretty important projects. And I can say that when one of the things that you notice when you do this, so these were in depth qualitative interviews that were an hour apiece. And what you find is that when you talk to enough people who are doing, you know, in a similar role, you do start to get convergence on a lot of topics. So when I did my PhD research on the Linux kernel, I talked to 16 kernel developers. And by the end of those 16 interviews, they were really like the last few, they were saying all of the same things that the other people had said. And I wasn't getting a lot of new information. So I think it's important to think about this as separate from like if you were doing a survey, there was just like, you know, fill out this survey, you'd get, you know, expect hundreds Of thousands of responses. But with these type of in-depth interviews, you get so much information that it does feel like the best practices probably would have converged. I don't know if Sibia has some additional perspective on that. Speaker 1 I have a couple of comments. I'm coming at this more from a market research background versus an academic research background. But I would say also 32 is quite a lot for an interview project versus a quantitative project where again, as Don pointed out, you're wanting to be in the hundreds of thousands. But often when you're thinking about it from a sampling perspective, you're trying to get a quote on representative samples ([Time 0:03:13](https://share.snipd.com/snip/e023af3a-7048-4de1-a2a1-9334b87a4c3e)) - **Caution Using Percentages In Small Samples** - Percentages based on a small interview sample (32) can be misleading. - Interview findings are better kept qualitative rather than presented quantitatively with small data sets. Transcript: Speaker 1 Perspective on that. I have a couple of comments. I'm coming at this more from a market research background versus an academic research background. But I would say also 32 is quite a lot for an interview project versus a quantitative project where again, as Don pointed out, you're wanting to be in the hundreds of thousands. But often when you're thinking about it from a sampling perspective, you're trying to get a quote on representative samples that who you talk to is going to represent the majority of Individuals that are present in the community. But there's sort of an inherent challenge there with open source because we don't really know the extent of it. And so I think the fact that they were focusing it again on those top 200 critical projects, we're already looking at a subset of the broader open source community versus a general swath. So that's something that I wanted to point out upfront. And I think their hand selection of trying to increase more diversity and representation of both projects type, I think was sort of their effort to try to get a wider perspective or more Perspectives on different types of experiences that could be had again in the smaller subset of the extended open source community. So I generally like their approach for that. I think what was a little bit confusing for me was that in some of the language of the findings and the way that the findings were discussed, because they had this 32 number in market research, 30, it's sort of a very, that's not a very conservative lower bound, but usually say 50 respondents in a sub-segment. But sometimes you'll get as low as 30 below that you can't really express things and percentages or groupings anymore in a way that I think because they are over 30, the researchers felt A little bit more comfortable talking about the distribution of respondents a bit more quantitatively versus keeping it all qualitative. And I feel like that was a little bit more diversion from some other interview based products that I've seen where you try to keep the discussion of the responses more specific to the Fact that these are individual speaking versus generalizations of a population. So I think that was a little bit, I want to say misleading, it was just a little bit more inconsistent with how I've seen some of these types of findings presented. But I think it was because they had number of 32 that allowed them to start thinking a little bit more quantitatively about it. I think if they were ([Time 0:05:31](https://share.snipd.com/snip/cf0cd5da-43e1-4476-a038-c66a396147b4)) - **Findings Serve As Experiences Inventory** - Some findings based on very few participants risk overgeneralization. - The report serves more as an inventory of maintainers' experiences than making broad theoretical claims. Transcript: Speaker 2 For a qualitative research study. 32 interviews, fantastic. I have done research projects with fewer interviews. In addition to my concern about the percentages that we have such a small sample size to even try to generalize in percentages, there was another concern with the best practices that Were pulled out because sometimes they acknowledge that only one person had this thing or they said almost everyone was paid fully to work on the open source project as maintainers And yet they made statements about funding for maintainers that are not paid to do so on a full-time basis. Some of the findings have a very small number of interview participants that had said something and in an academic venue that would probably be rejected as having sufficient data to Make claims on this. But here I don't think we are trying to make theoretical claims. Here it's an inventory of here are ([Time 0:08:29](https://share.snipd.com/snip/92837a8d-81be-45ae-ae11-6f297baa14b3)) - **Funding Section Raises Complex Questions** - The funding and satisfaction findings conflicted with common maintainer experiences. - Being employed full-time yet feeling unsupported reveals complex tensions in open source work. Transcript: Speaker 4 If I can do a follow-up, I mean Dawn said that a lot of the work following did resonate with her and while I feel like step around say documentation and governance and diversity did resonate, The funding section really and the kind of satisfaction section did not resonate for me. And I feel like it was a very confusing section that the majority of the maintainers that they interviewed all but one were full-time employed during open source work yet did not feel Like sufficiently supported with the projects they maintained. Min-me question exactly who they're speaking to as well as like what type of questions they're asking. I didn't ([Time 0:09:49](https://share.snipd.com/snip/255b4cff-9682-46cb-8dd1-e706dc76a014)) - **Funding Not Primary Motivation Insight** - People contribute to open source for motivations beyond pay. - Funding projects changes community dynamics and should be carefully considered. Transcript: Speaker 1 I did to a certain extent. I feel like we're jumping a little ahead in the questions like planned-all right. No, it is fine. I think we're going there anyway so let's go there. I think that again that based on how they were sampling this there was always going to be shortcomings and what we could extrapolate to in terms of experience in the broader community. I think a lot of the things that they did surface did resonate with a lot, at least my own experience working with maintainers across several types of projects but I did also struggle With the funding section the most and I think a part of that did to your point was a factor that are just more than enough people coming from that experience of not being funded to do this Work but I'm also kind of flagged that as my own question for further research and understanding different types of funding models because I think what's coming up is the conversations Around how we were going to grow or increase funding or increase participation by potentially sponsoring more contributors or bringing in contributions at companies and I think In terms of how this might connect to other research there has been a lot of investigation of motivation. They mentioned that the LF has done their own research into understanding contributors and contributor motivations. I know it has participated in a number of studies looking at contributor motivation. I've also worked on some studies working best at getting contributor motivation and they're all coming down to the same thing which is that people are not here to be paid. In terms of at least the studies that I've been a part of and read people aren't being motivated by that they're here for different reasons and this particular paper discusses that as Well but in the funding section they're sort of this like well we need to get paid we need to have make sure people are supported and part of that is encouraging more funding and more paid Roles in the community and to me that's a little bit opposed to a lot of understanding how these things come together and that people are not here because they're being paid so then if You pay people to be here how is that going to affect the composition and culture around the project. That's sort of an area that's I don't say unknown it's a big question for me and for other researchers ([Time 0:10:45](https://share.snipd.com/snip/9fff9c1d-1535-4fc9-9c27-19059ea1784c)) - **Technical Bias in Maintainer Samples** - Maintainers often come from a technical coder background, overlooking non-technical contributors. - This reflects inclusivity challenges in recognizing diverse maintainer roles. Transcript: Speaker 5 Anita. All right so I think the research does highlight what the definition of the maintainer however lots of people transition from contributors to maintainer most of these do not heal from Academic backgrounds do not have the scientific backgrounds but eventually there are communities that have people that are answering or doing the role of a maintainer from none of These technical backgrounds but the I noticed that the research does not like point out much or maybe it's data or the interviewees that are participating in this but I noticed that There's not much mention of non-technical contributors or persons who have this non-scientific skills or roles in communities and I wondered if that was like intentional or if there's A different angle to which maintainers come or maintainers can be you know put in place so if you don't have like the technical skills or experiences does that make you just a contributor That is doing the work of a metina but not a metina? Speaker 4 But did you notice I think to this point did you notice that they call this group of people super coders I don't know if again if that is typical in this in the space as well but it already Implies like that you're coming from technical background towards maintaining a project so I thought that was an interesting observation. Speaker 3 Yeah I feel like the interview selection was biased towards coders and then there are loads of maintainers who maintain other parts of the project. Speaker 5 Yeah it kind of highlights the inclusivity issue with non-technical contributors, persons who do not have these experiences with coding particularly. Speaker 4 I ([Time 0:13:38](https://share.snipd.com/snip/f26c1b00-846f-4b4e-8fe6-d295984bc64a)) - **Non-Code Contributions Underrecognized** - Research tends to focus on code contributions due to data availability. - Non-code contributions and maintainers often get underappreciated in research and community recognition. Transcript: Speaker 1 I will definitely plus one that Anita and any other comments from Don definitely felt like that it was only looking at that subset and I will say that I've contributed to that sort of bias And in prior analysis but a lot of it is because if you're doing research like this sometimes you're the other sources that you have are all very code-centric and code activity-centric And so if you're looking at trying to understand what's happening in a community and that's the sample you the data that you have then you want your research and interviews to also kind Of try to extrapolate that experience recognizing that your central data set is skewing toward technical contributions because that's the data set working with say logs around github. So I guess in my own research I've tried to find ways to either acknowledge that bias or try to expand it in other ways but I did feel like because of this being interviews and not looking At code contributions that they had an opportunity to be a little bit more inclusive in their own understanding of what it means to be a contributor the fact that they acknowledge that They're only counted contributors as code contributors and maintainers they said took on other types of tasks that were broader and beyond code that I felt that was a bit misrepresentative As well in the sense that saying you're only a contributor if you contribute code and then maintainers do more than code contribution and I didn't think that was acknowledging a huge Amount of individuals and work that's happening and in fact that came up later when they acknowledge how some projects need more than code contribution but they are commenting more On supporting infrastructure in the funding section versus saying that these could be other modes of contribution and how you're fostering a more inclusive contribution and that Does come through a little bit I guess they talk about outreach but in the diversity section but not in the encouragement of non-code contribution so it came up in other sections but I completely agree with that and thank you ([Time 0:15:23](https://share.snipd.com/snip/adedf704-1609-470b-9bac-8796d765f153)) - **Paid Maintainers Often Feel Unsupported** - Many maintainers are paid full-time by employers but feel a lack of support and recognition internally. - This results from difficulties in justifying open source work in traditional company structures. Transcript: Speaker 1 Out Anita. Speaker 3 Okay Mark how about how about you what did you find interesting in the methodology section? Speaker 2 One thing that stuck out to me and this goes back to the percentages that with 32 in the sample size it's not really helpful to have percentages but what what stuck out to me is that they Said 62 percent of interviewers were employed to work full-time on their project but then only 38 so about half said they feel high degree of support from their employer and 39 percent Reported that they felt open source work was highly valued in their organization that just makes me wonder about an organization paying someone to do the work but then those people Who are being paid to do so feel they don't have the support and open source is not being valued I just found that really intriguing how those numbers pan out where that comes from. Speaker 3 From just from working at companies who pay open source ([Time 0:17:18](https://share.snipd.com/snip/90a72c8c-414d-4da3-ae5f-551d781d1c39)) - **Documentation Needs Diverse Approaches** - Documentation challenges stem from engineers writing for other engineers, not all contributors. - Documentation styles must cater to different learning styles to be effective and inclusive. Transcript: Speaker 1 Community. Speaker 2 Thank you for for explaining a little bit about your experience and how these numbers could be explained I appreciate that inside. I would also like to move on to the next section in the report where we talk about best practices, maintain your best practices and there are several sections that the report has best Practice around contributor experience, community governance and management, documentation, funding and other forms of support, diversity and preventing maintainer burnout And Anita I would like to start with you what have you found interesting in this section what have you taken away? Speaker 5 So personally I think I found the documentation section interesting and that's because it kind of says that most open source projects struggle with retaining documentation contributors However like just a few lines after that it also says that engineers are the ones that develop the documentation and when it comes to documentation there's always this problem of having An extensive documentation but dealers are not able to actually utilize this document to the purpose that it's supposed to oftentimes it's because it either has so much technical Juggles that contributors or users want benefit from these documents do not have an understanding of or it is just not developed in a way that makes it easy for people to just grasp content That they go over and it makes no wonder whether the problem of not having like was documentation that can be utilized by everyone and still serve the purpose might be because there's This lack of collaboration between contributors who are actually experts in writing and experience in this area and engineers so the documentation comes together you know having Engineers writes documentation definitely results in some of these minor challenges in documentation I think that's what I picked from that I don't know what any other present things Speaker 2 Of this so documentation is something that I hear about a lot keeps coming up it is something that projects are notoriously always struggling with and it's something that very few people Really want to work on from what I hear there are some that really enjoy it but then they need to get a lot of input and something that goes into documentation also is how it's created and Presented that is super important this comes out of research where people have different learning styles and they comprehend it differently and if we have someone who jumps in and To a documentation okay step one what do I need to open a shell okay I'll do that what step two oh I'll do that and try if I fail I go back to a previous step but other learning styles they are More inclined to read the whole process before they do something and they already want to know what's going to happen next so that they know they're on the right path and you can write The documentation in those different ways and and that thereby include or exclude certain learning types and to certain contributors so I was missing that kind of discussion here From the best practices acknowledging that the way the documentation is written is important the best practices I see in here was more about okay how do we make sure that there is documentation Speaker 4 Written so that's what I saw in here on that note they are do you think that there was an opportunity to connect more of the best practices like if we are being more aware of the type of learning Styles for documentation maybe this would address diversity and inclusion efforts you know like it seemed like there was an opportunity to understand how best practices in one of Speaker 2 These categories could potentially influence factors in another of the categories yes I do think so there are several items where it crosses between the different categories here In the section that were not explicitly called out another example that I saw was establishing a code of conduct which is listed on the community governance and management but it's Also super important for creating a welcoming community and for diversity maybe you could talk more about the diversity angle while we kind of touched on that topic yeah sure the diversity Section there are three recommendations in here one is to make diversity and inclusion of first order order ([Time 0:21:40](https://share.snipd.com/snip/2645e03e-a8f9-4605-8747-2e32b55cd262)) - **Diversity Efforts Remain Work In Progress** - Most projects lack well-structured diversity and inclusion efforts despite recognizing their importance. - Small projects face particular challenges in focusing on diversity due to resource constraints. Transcript: Speaker 2 One of the questions that I have since we are here in chaos and talking about measuring project health and looking at projects assessing them is there any connection that you see between What's in the report and what we're doing in chaos. I mean chaos project is called out explicitly on page 29 as a resource which I'm super excited about. Speaker 4 I mean at the very end of our 29 page paper but I guess it'd be only reading my conclusion we get to it. Speaker 3 This is a podcast from the chaos project I think we should think about metrics and how it applies in this report so they don't like Gary said they mentioned the chaos project but they didn't Really talk about a lot of metrics but I think as those was the work within the chaos project we can look at some of the things that they talked about in this report and think about how we Can measure those and how we might be able to think about measuring them with the eye towards improving them for our own projects. So things like one of the best practices was to respond personally to first-time contributors and you can measure this as time-first response, filtered by new contributors to see How long it takes them to get a response and see how your project's doing and that area is it doing is it you know you're responding quickly could you respond more quickly. Another one of the best practices was setting up automated greeting bots and an onboarding ([Time 0:30:41](https://share.snipd.com/snip/5c915c9c-65ba-45f4-aa89-14f2142bc8be)) - **Project Health Framework Insight** - Understanding open source projects by support, complexity, criticality, and lifecycle helps plan health strategies. - This approach offers tangible steps for projects to sustain and grow effectively. Transcript: Speaker 3 Particular project. Speaker 4 I'm curious to know what people thought of the conclusion and the attributes table. I thought this was an interesting suggestion and conclusion of the paper and I kind of wish there had been more space for it but you know the general recommendation is to kind of understand Part of an open source project to understand where you are across these four vectors. Are you supported? Are you complex? Are you critical? And where are you in your life cycle stage? And then to kind of determine a strategy and roadmap for project health based on those attributes. I thought this was an interesting and tangible conclusion that people could walk away with and I'm curious what else people thought about this strategy. Speaker 2 Sure for me it's kind of self-evident to say as a project roast we need to think about better supporting entire complexity and so on. One reference that comes to mind is the archetypes the open source project archetypes by Mozilla and so I'll drop that link into the show notes because it gives a much more nuanced look At different types of projects and how they ([Time 0:32:40](https://share.snipd.com/snip/adb590c1-3497-4fd2-89fe-189ad63c3d07)) - **Contributor Lifecycle Needs Focus** - Contributor lifecycle, including eventual stepping away, deserves more attention alongside project lifecycle. - Succession and knowledge transfer are critical as maintainers near retirement or exit. Transcript: Speaker 1 This I've heard discussions like this and I'm glad you referenced another paper that we can look at because I know I've seen other models like this but I thought what was missing from It for me is that it ended at mature versus end of life or contraction and I think we only think about the life cycle of a project they don't all live forever and I think they kind of came out In the burnout section of the like how much time are you expecting to engage with us and how long. And that made me think about we think a lot about the life cycle of a project but we don't think a lot about the life cycle of the contributor past the sort of on-ramp phase in the sense that There is sort of the discussion of how we onboard new contributors and I know I'm done has spent a lot of time advocating for practices that welcome new and mature and mentor new contributors Into more established positions and the need to do that as the population of my t-narrs wants to step away but there isn't really this idea of when we all step away or when we think about Handing the reins to the next generation and so something that I think was missing from this was yes we didn't talk about ages specifically in the demographics but how long have these Individuals been doing this work and do they expect to do it in perpetuity or is this something that they want to step away from and I think about that a lot in some of these older complex Projects because we talk about it in terms of there's this bucket of maintainers that has an incredible wealth of knowledge and say a project like the like the Kubernetes organization And when they step out they're going to take all that knowledge with them unless they can hand it down in a way that's systematic and sustainable but what's not being said is because they're All about to hit retirement age we're aging out of some of these these larger long-standing projects and it just seems interesting to me that we haven't really been talking about and I think part of it is because there is a sensitive issue on privacy and age and we never want to ask someone okay are you fibers at a retirement and are you going to keep working on this project When you retire but at a certain point if we want to think more systematically about sustaining our communities and health of our communities and we need to start thinking about the Engagement lifecycle of the individual and how long they want to participate and being more open about their own goals and participation and how long they would like to stay engaged So I'd love to see more conversation of that and how something like thinking about that cycle can be folded into these sorts of conversations around projects sustainability and project Lifecycle. All ([Time 0:34:00](https://share.snipd.com/snip/7ca8ad04-839a-4564-a156-4cae64ebf356))