Episode 77: Open Source Metrics at Microsoft  ## Metadata - Author: [[CHAOSScast]] - Full Title: Episode 77: Open Source Metrics at Microsoft - Category: #podcasts - URL: https://share.snipd.com/episode/ebcda5bd-6422-4100-891b-2e02de82bfac ## Document Notes ## Highlights - **Microsoft's Multi-Tier Metrics Approach** - Microsoft uses a multi-tiered approach to open source metrics focusing on maintainers, component intelligence, and engineering standards on GitHub. - Metrics are tailored to specific groups and projects, emphasizing motivation and iterative improvement. Transcript: Speaker 2 First, say that it's multi-pronged or multi-tier. There's no single approach to metrics at Microsoft. From the OSCO perspective, of course, Microsoft being much broader in metrics, having so many implications to different teams and organizations. But in the OSCO, there's a few different ways that we think about it and some of the problems that we solve there. A lot of things that I worked on are more about trying to understand what our maintainers are trying to understand what brings value to the products that they're building, to the communities That they're trying to grow, and looking for themes across multiple projects. So rather than any one thing or trying to be prescriptive about a metric, although that might be different, it was a security perspective. But around community health, really trying to understand what is most helpful and then trying to bring COBIL to co-design the solutions to that. So that's working with maintainers, I guess, is one way to put that. There's also a ton of work that actually Justin can speak more about, specifically to component intelligence, like helping people understand their upstream, making decisions about The upstream, and providing the metrics, many aligned with chaos metrics models. So we'll go too deep into that because I know Justin probably will. So that'd be a second one. And then a third one that I can think of, and I'm sure that I'm missing some, would be around some of the engineering, standards work that we do on GitHub. And that's really James's area of expertise, like trying to create an understanding, and motivation, and compelling metrics that move people to action around the things that we need Them to do on GitHub. So things like making sure their posts secure, secret scanning. Very standard to get too much into those that James can expand there. So those are the three that come to mind. And of course, within each one of those kind of three buckets, there's like, how do we motivate people? How do we iterate on these metrics? How do we identify a set of people or repositories or projects that most need these metrics? And thinking about cohorts that the group in pieces, something also that Justin can speak to. So it's not all things for all people or all things for all projects. But cross-all and piece three, I would say that that's important, more to that both Justin and James speak to. ([Time 0:02:09](https://share.snipd.com/snip/4afeb683-3998-4dd3-94a3-4aaa944e1683)) - **OpenSSF Scorecard Security Experiment** - Microsoft experimented with OpenSSF scorecard to motivate repository security improvements. - Although cultural integration of security practices is challenging, some teams align their work with these metrics for management visibility. Transcript: Speaker 3 Can you talk more about that and some of your observations about using chaos metrics for that? Speaker 2 So I can speak to the OpenSSF scorecard, which was the very, very first experiment that we ran around bringing. I think it's our first sort of implemented experiment, although we've done a lot of informal ones. And I believe the OpenSSF scorecard is part of a viability metrics model, but it's one that we've talked about a lot in chaos meetings around security, so it obviously turns up a lot. And our experiment was to see if we could make yourself, we have sort of like a dashboard, we call our OpenSSF portal. There's various informations from retainers about their repository. And so we posted the score for every single repository in our Microsoft organization and started to socialize that to see if we could get people to move the needle. So this is squarely around the motivation piece. We could get people to, first of all, go look at it, to do something, to improve it, and three continually to make that part of their culture. The third one, I'll say, haven't really cracked that nut, how you understand culture is its own metric. But I did try motivations around, you know, if you move this and report it, then I'll do acts like some sort of extrinsic award. We did see some uptake there. We did see some people not necessarily bite at the incentive, but more describe how they were already doing this, which was really already using the scorecard. So they felt like it was nice that they could fit into or surface that to their managers. Like here's something like the company cares about or that things as important and so they could align their work with it. Others, I think there's a good of action around it, you know, if I'm new ways to implement it for themselves. ([Time 0:05:53](https://share.snipd.com/snip/ea26632b-a275-4363-80b6-20bdf4d1363c)) - **Combining Health and Security Metrics** - Combining CHAOSS community health metrics with secure supply chain data helps assess component sustainability. - This integration enables developers to make informed dependency decisions considering security and sustainability risks. Transcript: Speaker 1 Justin. So we see a lot of opportunities here to combine metrics models from chaos and parts of secure software supply training efforts that overlap. So when you take chaos community health, Patrick's and look at them from the other end, from the standpoint of consumers of a package as opposed to like the maintainers of that package, That tells you something about the potential sustainability of the component. And so we've been using data source from APIs that have been built for secure supply chain reasons. So like depth.dev ecosystem.ms and APIs from the open source security foundation. And so the thing about those APIs, that's really really great is that they are populated for like every major package manager and all the get over repositories that build those components. And so we can then go to experience for developers who are evaluating whether or not they want to consume a particular dependency and give them warnings. And if there's a potential sustainability problem with that component. So we're combining chaos metric models about the sustainability of a component with all this like wrong metadata and components and repositories that spend it collected and put in Place by the security community. ([Time 0:07:41](https://share.snipd.com/snip/b0767774-eabd-49cb-afb2-475ab872dcbc)) - **Early Dependency Risk Warnings** - Provide developers starting new projects with warnings about dependency compliance, security, and sustainability risks before they write code. - Deliver this information on a single page to enhance ease and speed of informed decision-making. Transcript: Speaker 3 Me follow up on that just a little bit in doing that. What were some of the scale challenges that you experienced and what did you learn about overcoming some of those challenges and using metrics at scale? Because once you start analyzing all of these dependencies, you end up with an awful lot, right? Scale becomes a problem and I'm curious how you addressed that. Speaker 1 Yeah, right now, the experience that we've built so far is focused on developers who are starting a project. And so they have probably not written any code yet, or they've written only a very small monocode. And so can you give them project developers don't think about whether or not to like use every single dependency and a transitive dependency, right? They kind of think about maybe one or two at the most. And so for right now, we're just focusing on a experience that can help them make good data informed choices about those like one or two dependencies where they're not sure that they Should use like a B or C or like whether they should like stick to the thing that they've used in the past or try out the new thing that someone I told them about. So we're trying to give them information that's across a couple of different categories of like risk. So there is compliance rules that people have to follow in terms of like vulnerable dependencies and licenses. And so we're trying to provide some like warnings regarding those things to people before they actually start like are writing any code because our existing tooling typically only Tells people these things during build events. And by that time, save all of the like it in some code, which they probably don't want to like throw away. So we're trying to tell them those things before they have spent any time ready in a code. And then also we're trying to give people some warnings about like sustainability risks. So says they give them a repository or a given component of a active community behind you or do they tend to respond to things that are very slow or as the project just not seen any activity For a very long time. So like these are like all things that people could like figure out themselves if they went to the like source repository and like ride through things. But that takes time and people don't have time. And so we're trying to give people all those things on a single page really really fast. And then the last time to aspect is a T developer experience. So that's something that tends to only be a risk if you call a particular dependency on a lot. So it's like only on my wrist sometimes as opposed to most of the time. ([Time 0:09:20](https://share.snipd.com/snip/351d3c4f-2116-4ee1-9603-2c75d16119dc)) - **Internal Open Source Community Value** - Open source is deeply integrated in Microsoft’s software development across industries. - Building internal communities around open source reuse reduces duplicated efforts and accelerates product delivery. Transcript: Speaker 3 All know that open source impact is not always external, right? Lots of us have worked within companies. Can you talk more about the business impact of open source within Microsoft and in particular, maybe some of the metrics discussions that you have around that business impact? Speaker 1 Of course, open source is a core component of how Microsoft approaches software development at this point is prolific across almost all industries. We not only want to understand where it's being used, but also how it's being used, because we also want to build an internal community around the open source that we use. If somebody solved a problem, we don't need other people reinventing that wheel. And so one of the things that I know Justin has really been great at is surfacing that kind of information to users and trying to build internal community around that in hopes to reduce The overall amount of time it takes for our developers to get new features out new products out and the like. But it's also a huge concern for us when it comes to security and the supply chain. We're making big efforts towards trying to get S-Bombs to be more prolific across the industry and generating S-Bombs for all of our packages so that we can have proper provenance of Where software is coming from. We want to make sure that it's coming from trusted sources, it's being built on platforms that we don't have any concerns about malicious actors getting access and directly impacting Our end users. It's a key thing that I think about every day and I know it's the key thing that our security people think about every day. And so we do a lot of contacting our internal users about how many repose they own if there's an active community on it, ensuring that they understand that there's specific things that Might need to be changed or approved or taking an opportunity to even call the mountains. Say like, hey, you're doing a really good job. Thank you for the work that you're implementing. And we do that via we call it maintainer mailer that we try and send quarterly. And so it's just a personalized mailer that allows us to go through and do all of that all on a single location. I guess we like to kind of consolidate things into a single location because one of the big problems we've had is getting interaction with open source. ([Time 0:13:32](https://share.snipd.com/snip/a10bf100-ce50-4349-812d-199a7dfaa880)) - **Maintainer Mailers Boost Engagement** - Use personalized, consolidated communication like quarterly maintainer mailers to engage open source project maintainers. - This approach increases interaction, tracks project health, and aligns with security and compliance standards. Transcript: Speaker 1 And so we do a lot of contacting our internal users about how many repose they own if there's an active community on it, ensuring that they understand that there's specific things that Might need to be changed or approved or taking an opportunity to even call the mountains. Say like, hey, you're doing a really good job. Thank you for the work that you're implementing. And we do that via we call it maintainer mailer that we try and send quarterly. And so it's just a personalized mailer that allows us to go through and do all of that all on a single location. I guess we like to kind of consolidate things into a single location because one of the big problems we've had is getting interaction with open source. When people think open source, they don't particularly think that there's any rules. They just think, oh, free software. And so when we're at the scale that Microsoft's at, we need to put some guardrails on to make sure that our developers are safe and our users are safe. If we consolidate that down to the dashboard for looking up dependencies or single email that they received quarterly, we're putting all varigs in a basket, but we're definitely getting More interaction than we were and having a better ability to actually track that. ([Time 0:14:57](https://share.snipd.com/snip/aff2e10b-6add-4b1e-906d-0e716d1a2cd0)) - **Visualizing Contribution Impact** - Showing contributors how many internal Microsoft products benefit from their external contributions increases engagement. - This impact visibility helps in rewards discussions and resource pitches for open source teams. Transcript: Speaker 2 Easier. I would add on to the internal values or external participation. I think that's something that we run into it, like in helping maintainers is how do they describe that value in their reviews? Like how do they show that open source is having impact on the business? And when you're out contributing to open source, although like this is something we encourage every single person to do, I sometimes come to those reward times, like, how do we help People talk about that? And one of the line items in this maintainer email that James was describing was that we have a line about the number of internal Microsoft repositories or products that benefited from An external contribution. So I think we're continuing to build that out. People definitely perked up when they saw that from a personal perspective that they could see, oh, like this one or two contributions that I made had an impact on 700 products like that. You know, that's a tremendous type of thing to just be able to say just from a private perspective. I mean, an open source of you can talk about how you've made a difference. Most of us can write off that as a great thing, but if they can also then take it to a rewards discussion or help a team pitch for resources and that kind of thing, I think that we're a little Bit onto something there. ([Time 0:16:12](https://share.snipd.com/snip/21cac598-1621-47fc-9427-19c042a203f7)) - **Future Focus: Funding Metrics** - Microsoft explores leveraging metrics to optimize open source funding for better return on investment. - Targeting high-impact components for financial support could improve sustainability and developer decision-making. Transcript: Speaker 3 So you guys are doing lots of really great things. Speaker 1 So where is Microsoft going next with metrics and what are some of the things you'd like to measure in the future? One area that we've been talking about recently is in funding and trying to leverage some of the existing metrics work and data work that we've done to make funding both be a little easier Of a proposition and also get perhaps a better return on investment of that funding as well. So like Emma's done a lot of like funding of awesome work at Microsoft. And so like some areas that we're trying to understand if we can leverage them, I mean, could load the upstream contributions on data set. And if there is a way to direct some of those upstream contributions to components that have very high impact at Microsoft, and then also for the work that we've done to help developers Make good dependency consumption decisions, you can take some of those metrics from around sustainability and potentially to identify components where a financial attribution Would have a higher return on investment. So there's lots of unanswered questions there. But those are some of the areas that we've talked about potentially working on soon. ([Time 0:17:22](https://share.snipd.com/snip/c9550138-43d3-42d0-a8dd-c1a92a33a40f)) - **James Sierry's Creative Reconnection** - James Sierry rekindled his passion by returning to art and music after a 15-year break. - This creative outlet helped him reconnect with memories and refreshed his perspective beyond software. Transcript: Speaker 1 My value add is I have been recently reconnecting with creating art and music. I took about a 15 year break from doing both of those to make my way through the rest of high school and all the university and then get my career established. And now I'm kind of getting back to a point where I'm feeling back in that headspace. And so it's been really fun to reconnect with that. And it's crazy how much sense memory people have in connect music and art. And so I've been getting to re experience a lot of memories that I hadn't thought about for a long time and kind of start seeing the world a little bit differently again and start the act Of creation all over again. That's not just software. Software is great. And there's always a creative process in that. But it's nice to be able to hold something in your hands or hear a thing that you've put your blood sweat and tears into. So it's been something that I've been very fortunate to be able to do as a part of the community. ([Time 0:21:31](https://share.snipd.com/snip/05c3f7c2-76e9-4f09-a2ec-4670083a8995))