CHAOSSCON Europe 2024 Keynote  ## Metadata - Author: [[CHAOSS]] - Full Title: CHAOSSCON Europe 2024 Keynote - Category: #articles - Document Note: --- tags: - #OpenSourceResearch - #Ethics - #EcosystemHealth Author: Julia Furlan (AWS, with collaborators from Google & University of Complex Systems) Source: CHAOSSCON Europe 2024 Keynote Reference: [YouTube Recording](https://www.youtube.com/watch?v=c6m9k0PxAX4) date created: 2025-08-22 23:59 --- # Title: Best Practices and Ethical Challenges in Open Source Ecosystem Research ## Table of Contents: - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📖 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🔍 DARVO Analysis](#-darvo-analysis) - [🔍 SEO Evaluation](#-seo-evaluation) - [🎯 Conclusion](#-conclusion) --- ## 📝 Summary: Julia Furlan’s keynote at **CHAOSSCON Europe 2024** highlights the **ethical, methodological, and human dimensions of researching open source ecosystems**. She critiques the tendency to treat open source solely as datasets while ignoring the human contributors and community contexts that shape the ecosystem. The talk calls for: - Respecting **consent** and **contextual integrity** in research. - Recognizing open source as **multiple ecosystems**, not a monolith. - Avoiding **over-reliance on commits and repositories** while neglecting social contributions. - Understanding the risks of unethical experiments (e.g., inserting flaws into Linux kernel). - Collaborating between researchers and practitioners for **ethical, accurate, and useful outputs**. --- ## 📖 Main Content: The keynote situates open source research within a **socio-technical framework**, emphasizing the **interplay between data, software, and humans**. Key contributions: - Open source must be studied as **ecosystems (plural)**—diverse, dynamic, and constantly evolving. - Public data ≠ ethical data. Researchers must respect **community consent**. - Misusing data (commit frequency, mental health inference, etc.) erodes **trust and authenticity**. - Experiments in open source (e.g., 2021 Linux kernel incident) are **human experiments** with downstream risks to infrastructure. - Ethical best practice = treating contributors as **human subjects**, not just data points. - Balance **openness, ethics, and privacy** to avoid harm. - Inequalities persist: time, money, and socio-economic status shape who contributes and who is absent from datasets. - Collaboration between researchers and practitioners strengthens ecosystem resiliency. --- ### 🗒️ Notes - [00:01](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=1) Julia introduces best practices in open source research. - [03:27](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=207) “Ecosystems” (plural) framing—open source is not monolithic. - [06:53](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=413) Open source = complex socio-technical ecosystem, not just repos. - [10:08](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=608) Community consent is vital in using public data. - [13:33](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=813) Collaboration + context = valid research outcomes. - [16:50](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=1010) Violating contextual integrity breaks trust (e.g., scraping emails). - [19:51](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=1191) Research bias: software data > human/community data. - [22:45](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=1365) Life changes drive contributor exit more than peer support loss. - [25:31](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=1531) 2021 Linux kernel ethics incident: researchers wasted community time. - [28:32](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=1712) Call for collaborative ground rules between practitioners & researchers. - [31:20](https://www.youtube.com/watch?v=c6m9k0PxAX4&t=1880) Shared norms improve outcomes; empathy for both researchers & practitioners. --- ## 📖 References: - Miller et al. (2019) study on contributors leaving FLOSS. - Helen Nissenbaum’s **Contextual Integrity** framework. - 2021 Linux kernel research ethics controversy. - GH Archive / GH Torrent dataset practices. --- ## 🗣️ Paraphrase: 1. **Original**: “Just because data is public doesn’t mean it’s ethical.” **Paraphrase**: Public availability of data should never be confused with ethical permission to use it. 2. **Original**: “Experiments in open source are human experiments.” **Paraphrase**: Any intervention in open source systems directly affects people and communities, not just code. --- ## 📰 Newsletter: 📢 **Open Source Research Isn’t Just About Code** At CHAOSSCON Europe 2024, Julia Furlan dropped a reality check: researchers must stop treating open source like a free buffet of commits and repos. Her keynote laid down best practices for **ethical open source research**: - Respect **community consent** (public ≠ ethical). - Balance **openness with privacy and integrity**. - Recognize that experiments in open source are **human experiments** with real-world consequences. Takeaway: If you’re researching open source, treat contributors as **human beings, not just datasets**. 👉 Watch the keynote: [CHAOSSCON Europe 2024](https://www.youtube.com/watch?v=c6m9k0PxAX4) --- ## ✅ Action Items: - Position GoDaddy OSPO as a leader in **ethical open source research standards**. - Incorporate **contextual integrity** into contributor data policies. - Advocate for **community consent frameworks** in WordPress & The WPCC. - Develop **educational workshops** on ethical research practices for WordPress contributors. - Use keynote insights to support **sustainable contributor engagement** strategies. --- ## 🧪 Hypotheses: - Ethical research frameworks increase **community trust and collaboration**. - Public audits on ethical use of contributor data boost **ecosystem resiliency**. ### 📊 Methodology: - Conduct surveys of contributors about comfort with data use. - Comparative study: projects with community consent vs. those without. ### ❓ Questions: - How can OSPOs formalize **community consent protocols**? - What unseen inequalities exist due to **time and money barriers**? ### 🔮 Future Research: - Case studies on **contributors who never start** contributing. - Longitudinal studies on **life events & FLOSS participation decline**. ### 💡 Implications: - Ethical lapses risk community attrition. - Stronger frameworks could reshape **policy and funding** for open source globally. --- ## 🧠 Learning Styles Assessment: - Visual: ❌ Minimal visuals—infographics on ecosystems would help. - Aural: ✅ Keynote talk (aural medium). - Verbal: ✅ Storytelling about case studies. - Physical: ❌ Could add hands-on exercises for researchers. - Logical: ✅ Clear arguments on ethics, causality. - Social: ✅ Collaborative research emphasis. - Solitary: ✅ Encourages reflection on researcher role. - Naturalistic: ❌ No environmental analogies; add sustainability framing. --- ## 📊 Accessibility Assessment: - Transcript provided but needs **clearer segmentation**. - No diagrams or alt-text—visual aids would improve accessibility. - Slides should ensure **contrast compliance** for readability. --- ## 🌍 DEIB Assessment: - Strength: Emphasizes inclusion of contributors overlooked in datasets. - Weakness: Limited global cultural perspectives. - Improve: Add case studies from **non-Western ecosystems**. - Tone: Empathetic, inclusive, ethical. --- ## ✅ Logical Validity: - **Strong reasoning**: Ethics → trust → better research outcomes. - **Causal reasoning**: Experiments in OSS = human impacts. - **Fallacies avoided**: Careful, nuanced arguments. - Tone: Thoughtful, reflective, candid. - Summary: The speaker is discussing best practices for researching open source ecosystems. They emphasize the importance of understanding the social elements and treating the human subjects involved with care and respect. They also highlight the need for consent and collaboration with community members who have expertise in their respective ecosystems. The speaker cautions against using public data in unethical ways and stresses the importance of considering context, privacy, and equity in research. - URL: https://www.youtube.com/watch?v=c6m9k0PxAX4 we have a recording the recording is start all right welcome back everyone to our keynote I'm going to turn it over to [Applause] Julia thanks for sticking around everyone um I am really excited to be here to talk about a paper that we had published last year um about what can go wrong in research as the researchers and practitioners dig into open source ecosystems so in contrast to the panel earlier today I will be here with no hope maybe maybe it's a glimmer of hope but we're going to talk about some best practices that we've identified when doing research in open source EOS systems so I'm Julia fi I work at AWS open source I am their Ai and ml strategist but that is not relevant for today so this this work was um completed before I started at a AWS and my collaborators are um at Google and at the University of complex systems CS but researches even though um I I am a PhD Dropout I'm sorry um I'll leave now that's just good judgment the leaving now or the dropping out the dropping out they're dropping out yeah they're dropping out to be clear um I've been I've kept involved in research since since I left um School however many years ago that was and I'm particularly interested in themes around motivations within open source and how understanding the motivations both from personal corporate uh individual perspective has the potential to increase ecosystem resiliency I also Co founded open open source stories a few of you have already been featured on it we tell the stories of um people in open source that don't typically get the recognition from thect that they they should so Amanda ceri um is at Google's open source programs office and she looks at open source with an intersectional lens and Dr Jun is an assistant professor she actually got her PhD between when this paper was published and and now and she's a multi-disciplinary complex system science scientist and the running theme for all of us is we're really interested in looking at open source as systems and that's a very deliberate pluralization of the word systems because it's not one system it's many what really inspired us to to delve into some best practices for researching open source was I don't want to say like outrage but a little bit of outrage on my part um I kept reading papers using open source as the data set um or as an area of of study and disagreeing with entire premises or figuring out places that they needed to ask more questions they needed to understand the open source ecosystems better so we came together and tackled these issues one by one to to figure out okay what are some of the best practices that we can recommend when people are doing research open source how can we break this feedback loop between research and practice because the research that's being published using open- Source ads data often times gets put into practice without understanding of context seeing not so how can we work together to benefit both researchers and practitioners so when we think about open source um I heard a lot of talk about code today a lot of talk about commits a lot of talk about um repositories and open source is that yes but it's so much more it's this complex multilevel ecosystem of human contributors who happen to come together to build something and they contribute in many different ways they may contribute code they may contribute developer experience expertise accessibility testing Community um management Etc but it's also this living breathing organism that changes over time and we really need to make sure the we're accounting for that change when we're looking into the ecosystems themselves or the ecosystems as proxies for other problems it is a very attractive research space because of how data Rich it is who here has used GitHub or GH Tor as a data set really it right it was renamed GH archive archive yes um so even though we have all of this data we have to make sure that we're taking into consideration that it is not complete in any way shape or form and it's exciting for researchers because there's so much data but that data often times comes without heavy so it can give you a limited snapshot of an ecosystem at a point in time but it doesn't give you full understanding so when we do refer to open source we're referring not just to the collection of repositories but also the community involved their interactions incentives behavioral norms culture and there are many open source ecosystems and one of the big mistakes that we tend to make both as practitioners and researchers is referring to open source as one ecosystem so let's talk about some best practices and how things can and have gone wrong in the past so we heard a lot about data wonder why it's like this this entire day is focused on data or something um the panel was really insightful and I'm going to touch on some of those things in this section as well but at the heart of all open source ecosystems are the humans involved we hear this in various SPS over the course of many many years that open source people open source would not be here without people the production of Open Source software reflects social activity and labor of a group of individuals working towards hopefully a shared goal if they don't if they're not working to Shared goal I think that's where for game comes in so but because we're talking about humans all the data associated with open source it's human it's involving human subjects and we need to treat that with the care and respect that it deserves scraping does not exempt you from dealing with human subjects because they're still implicated in the data collected so one guiding principle is just things can doesn't mean you should um I believe Jenn gold BL said something to this way back when like in the 90s or something um and we hear a lot that open source is public all the data is is out there and yes th those repositories are public it's part of what makes it both such a fantastic place to work as well as an attractive place for research problems but just because public doesn't mean it's ethical and I'm delighted to have heard so much about consent during the day here um Community consent is critical when working with public data and there's a real possibility probability to cause harm to the humans in Open Source by using data in ways that they do intend some of these papers may not have crossed um everyone's radar but we've seen people we've seen resarch resers use uh commit data for instance to assess productivity and what productivity should look like or could look like what is the optimal productivity for uh a software engineer and in um in some places we've seen that implement to the detriment of people who don't necessarily follow the normal pattern of commits who have different behavioral patterns we've also seen people analyze commit messages to get us to get a glimpse into mental health and if you analyze my commint messages might give you a good view of my mental health but not what I want you to have um and some analysis like that can actually be seen as a betrayal of trust it can make open source and the people within it less authentic if I get a bot that says hey Julia stop beating on yourself this was your first commit it was just a typo I'm not going to be really enthusiastic about committing with abandon right so we want to make sure that when we're doing research and when we're cooperating with researchers that we are still keeping an environment where people feel comfortable being themselves and continuing what has made open source such a powerful force in the world so when scoping research questions keep in mind it's not just code we do over index on code and their related artifacts because it's easy but we're not afraid to hard problems this is Linux we know this just got to throw another one in there um but making sure that we understand the socio element of a soci technical system helps us figure out where we are missing information where we are missing opportunities to collaborate with each other and it helps us remember that the people involved in the systems people behind the data deserve to be treated with care and respect that was my favorite catch all research best practices so we are in this weird spot where we've got a massive wealth of data we've got cheap computing power we've got cheap storage it's it it's like you're a kid in a candy store right you you can do so much with what we have available to us but understanding the context of the systems is important to really understanding the system as a whole and the best way to do this just really ask and I understand that time is valuable but in order to have valid outputs to your research consent is key and community members are often subject matter experts in their sub ecosystem they have knowledge and lived experience from their days months years decades working within that ecosystem so by collaborating together in the spirit of Open Source we can better help each other understand the parameters of the system involved so I I love to mention earlier I have a note of non incompatible purpose did I get that correct okay non incompatible purpose um so that is that was a phrase that was new to me today I love it um and I have a similar one which was put forth by Dr n bom the idea of contextual Integrity it's the idea that protectional privacy is tied to the Norms of specific contexts in which the information was gathered Dr Nom wrote a central tenant of contextual Integrity is that there are no Arenas of life not governed by Norms of information flow no information or spheres of life for which everything goes and so in 2016 talking about my confusion with GH GI arive um a GitHub user posted a comment on the repository for GH torrent saying can you please take my email address out of this data set because it used to have every buddies get commit email addresses in it this is very useful for a number of reasons um but they pointed out that there's a big difference between people browsing a profile and deciding to Ping them presumably about their work and getting spammed essentially by researchers because they've got this one data set that they're going to use and they're certain that they're the only person that has thought of doing this and eventually email addresses were redacted from that data it took a lot of doing um it was it was not a a light effort um on the maintainers part but there was enough justification that this was important so bringing it back to contextual Integrity this use of their email address violated contextual Integrity in which that email address was given similarly my self-deprecating commit messages I didn't volunteer those for Behavioral analysis I volunteered those for my own personal like hey maybe turn on spell check next time use so we can't use well the data is already public as this convenient phrase to circumvent ethical ises and that's part of why openness is not necessarily a good in of itself as much as I would like it to be we have to keep in mind that the ways in which information is disclosed can be very easily reverse engineering to identify a particular person it can violate trust it can make practitioners less inclined to collaborate with researchers in the future and it can increase the Schism between research and practice and I think that would be a shame so we do need to strive for a balance between openness ethics and privacy and we need to do so in consultation with the community being studied and we need to keep in mind that even within a community it's not a monolith which feeds nicely into respect and Equity when we research open source ecosystems for researching people but when we look at the data that we do have research about the software is far or data about the software is far more readily available than data about the people for a number of reasons but given that we need to take care not to exacerbate or reinforce inequalities in the existing system by failing to question what is absent from the data we have we have tons of commit messages I'm just going to keep saying commit messages apparently it's a fun phrase um but we don't have a lot of data about activities that aren't captured by GI or your Source control system which are when you're choose to use and when we over index on the data that we have we are inherently saying a value judgment saying this is more important than the data that we don't have in 2017 um Lawson published a blog post saying I've already told my partner that if and when we decide to start having kids I'll probably quit open source for good it just can't see how I'll be able to make the time for doing both there's been plenty of research done into who contributes to open source and why and what benefits they see from it and these benefits are not insignificant who here has a job in part because of Open Source okay so that's direct employment you can get direct compensation you can be sponsored and you can have you can build a business on open source um so the benefits that we do see people receiving from their work in open source can be quite quite large and for the people who don't receive compensation for their work they're dedicating their free time so they're actually paying to work source that payment is currency the currency is time but they're paying to do so in 2019 there was a great paper um by Miller at all and on the link at the end the study was um why do contributors give flossing kind of just love from a p perspective but the the thrust of it was that for all contributors occupational reasons such as major life changes were the most cited for leaving open's Force significantly more than lacking peer support or losing interest that are much more commonly discussed in the literature so other best practices that we have is assume that the economic incentives and the availability of people who keep the lights on are not evenly distributed one of the papers that I am dying to see please somebody again just throw this on my free dissertation idea pile I want to read the paper of why people never start flossing it's not dentists I promise but when you're not getting paid to work in open source you are choosing to spend your free time on it and if you don't have that free time that can impact your presence in the data that feeds into recruiters looking for people qualified for jobs which feeds into your longterm employment potential there is so there's so many Downstream effects and so I want to see that paper I want to see why people never start contributing to open source but that's the data yet so I think by now we know that open source is part of pretty much everything as used everything from space exploration to social networks to the very scary device that I had implanted in my head once um did not go through IRB approval or any sort of like checking but every single person on the planet is affected by open source software whether they know it or not the water treatment plan but they get the water from uses open source software the system that evaluates whether you're eligible for benefits uses a resource software and this became really apparent when in 2011 or 2021 actually there was a paper that had been retracted you may have caught this in the news it made quite a few rounds where researchers submitted known flaws to the lenux colonel they had no intention of allowing him to be merged Upstream but it it made some like headlines here and there uh and Greg Scott said these researchers crossed the line they shouldn't cross nobody asked them to do this and a lot of people wasted a whole lot of time and time is currency now to their credit This research was designed to have no side effects but they over indexed on the software and not on the socio technical system if they had considered that they were essentially running experiments on people not on software their approach might have changed they might have said maybe we should send this to an IRB for review or maybe we should not do this um but we had an entire University banned from contributing to the Linux kernel I don't know if that b is ever going to expire hopefully at some point maybe so if you are doing experiments in open source regardless if the root is technical or not you're running human experiments and it might impact the world's ecosystems and infrastructure in unknown and immeasurable ways it's hard to know the scope when you're talking about open source you don't know who is using a library if they aren't also open source even small changes to one part can be what breaks the mission critical system and there's just no way to evaluate that ahead of time so dooming room did i a my mission are you acutely aware of how fragile this whole socio technical system really is well have I got us for you I think there's a ton of potential there's a ton of potential for us to work together in both scoping research questions as well as doing Outreach and providing information to researchers and so when we do actively work with the ecosystems being studied we're going to have better outcomes and better not meaning more positive but better meaning more accurate and reflective of reality because we can't leave the witness here but it also means that if we have better research we can put research findings better into practice I read a great paper um last year about dependabot and how to reduce cognitive load so that people found to depend about alerts useful instead of noise and I think that's a fantastic place where research and practice can come together to collaborate we need to make sure that we're not assuming things about identity consent or a prior knowledge and to also not try to infer them inference about people not a good idea we've got plenty of counter examples there and we need to make sure that when we publishing research that we're considering the effects on the broader dig system so beyond the repository for factors that might influence your methodologies and findings it's the wild west out here in open source land and open source data land we're still figuring all this out and science is also an ecosystem that's always in production so we have to have a bit of empathy for both sides of the coin but the best way to move forward is to work together and it's really awesome because I see so many researchers who are newly interested in open source and not just in a hey this is a cool test bed way but in say oh I see the potential here and I want to contribute and we see practitioners doing the same just in the opposite direction so I think that I think that if we can all agree on some like nice ground rules and understand that nobody has all the answers we're all making it up as we go along we've got a lot of places we can make a lot of impact to make people's lives better so as promised references if you want to read the whole paper that's up there it's got more formal language um it's build for scientists and I just want to say thank you for having me and I hope this wasn't too much of a down to end no we need the warning labels that's true I should make sure to add those to the presentation may cause intense [Music] suffering does that have a place in open source I don't know I mean we could just like ha on my messages some more yeah yeah [Applause] do you want to take questions we have a little bit of time if if people would like, CHAOSSCON Europe 2024 Keynote, CHAOSS, 01hnjvgfdjqyvx72gs1tynh5ad.