Episode 109: SBOMs and Project Health With Brittany Istenes  ## Metadata - Author: [[CHAOSScast]] - Full Title: Episode 109: SBOMs and Project Health With Brittany Istenes - Category: #podcasts - URL: https://share.snipd.com/episode/0d086fc1-f9f8-44ea-9071-190e77b5408d ## Document Notes ## Highlights - **Rise of Malicious Open Source Packages** - Malicious open source packages increased by 700% in two years, exposing software supply chains to attacks. - Only about 10% of 7 million yearly packages are actively maintained, making careful selection critical. Transcript: Georg Link Is very timely. And that is the Software Bill of Materials and Project Health. To give you a little background, we have seen in open source over the last couple of years, an increase in malicious packages being published. The Sonatype research showed that there's about 700% more malicious packages discovered in the last year than just two years before, which is insane. So apparently there are bad people who have discovered that the current state of open source is very fertile ground to attack our software supply chains. If we look at it, there is more than 7 million packages being released every year, and only about 10% of those are actively used or maintained. So there's a lot of noise, a lot of uncertainty when it comes to choosing the packages that we are including in our software supply chain. And there is a lot of work that we need to do to make sure we have good, healthy open source projects that we depend on. Because if we don't, software supply chain attack, in the case of Sullivan's, we know, costs 11% of annual revenue for the companies that had to spend their time, money, and effort to Mitigate it. So it can be very costly if we don't take care of our software supply chain. And Gartner has estimated that by this year, 45% of companies will have faced a software supply chain attack, which is a flip of a coin. So are we prepared to gamble with the health of our software by not doing our due diligence? Well, somehow, this will no longer be a choice going forward, because there is legislation coming out, or is already in place, and now is being worked on how to deal with it, like the Cyber Resilience Act from Europe that says you have to do your due diligence on your software supply chain, including the open source that you use. And the Software Bill of Materials that we are talking today and Project Health are two very important considerations in how we do this. And I will open it up, maybe start catching up our listeners to what is a software bill of material. Brittany Istenes You know, software bills and materials, basically, as we all know, it's just a vast library that basically shows into your applications, every single open source package, library, Everything that's being ran, right? It's also showing your project health and project scores, depending on which tool that you're using or depending on what solution that your company has decided to leverage. I think that one of the things before I pass it on over, we really want to understand what's in our software supply chain for massive remediation efforts and things like that. But we've been talking about SBOMs significantly, a lot more, particularly since end of 2021. But how are our developers leveraging this tool? Are they at all? Do they know that it's just running on the back end and they're just kind of not even looking at it or engaging in it? It's just another thing for them to use and leverage. I think that's what I'm seeing particularly quite a bit within our fintech spaces. And I think that that really needs to change. But in order for SBOMS to be impactful, how the company embraces it, how the automations are set up for your remediations, I think that making things easier on our devs enhances our innovation. And I think your automatic upgrades and package support and all of that should be happening in the back end without any intervention. Right. And I also do think from an SBOM perspective, there are some missing pieces. So, yeah, I'd love to hear what Kelly has to say about that, too. Yeah. Cali Dolfi Whenever I think of an introduction or the background to SBOMs, I think that the ([Time 0:01:59](https://share.snipd.com/snip/a7a62732-1a06-498e-8c8f-84ab2f93452a)) - **SBOM Standardization Challenges** - Lack of consensus on SBOM fields and formats hampers clear standardization. - Including upstream repository links in SBOMs enables better analysis of project health. Transcript: Cali Dolfi Is very important. And what do people consider to be an SBOM as in the fields, the format? This is something that is heavily, heavily debated and has been for years. And there has not really been a major conclusion agreement of here's the format here are the formats there's some form of consensus that's coming but not really nailed down you can get People on some pretty long rants if you bring this up and the agreement of what are the required fields for an s-bomb and what are the optional fields for an SBOM coming from the angle that We are of being like, OK, how can we use SBOMs to understand our upstreams better and use it to analyze community health? Something that I really want to advocate for and I've talked about with this group is making sure that the link to the upstream repository is across the board a part of SBOMs. That allows for a lot of this analysis to be able to happen. But when you talk about what the SBOM is, software, build, materials, the definition is one phase of it. The comms and more of the discussion around it and more or less the stalling that has occurred in the public space of what this means, what are the definitions. A lot of companies are doing this very differently. This is discussed in the open, but I think there's a lot of companies that are making choices and doing things more closed. Because it is a publication of the internals of your product. And people get really squirrely and scared when you start talking about that once you get more and more on the preparatory side of technology. But we all kind of know this that every software piece of software there is is going to have some upstream ([Time 0:06:19](https://share.snipd.com/snip/996c897a-9894-4930-8d29-e22a32f25b7b)) - **Historical Purpose of SBOMs** - SBOMs originated for license compliance to track obligations and copyrights. - They are akin to food labels, showing software ingredients and revealing potential "allergens" to companies. Transcript: Georg Link A little bit and provide a little historical context for our listeners. The software bill of materials, when they were initially created, was around license compliance. That was the first use case where, because open source is a licensing scheme, and we are using copyright in a way to protect the freedoms of users and developers and everyone who is using Open source, there are certain freedoms, but also obligations that come with different licenses. And because there is a proliferation of many different open source licenses, companies that have a really long software stack needed to identify what are the licenses that we actually Have? What did we pull in, in terms of open source libraries? And do we have to share the source code because it's a copy left license? Do we not? Do we have to give attribution? Are there any patent clauses? So the lawyers needed a list from the developers saying, what did you actually use? And what are our obligations? And Software Bill of Materials from that initial need. And we think of it in a similar way that you have on food, the package labels that say, if you buy this and eat this, you're actually consuming all of these ingredients. And the good thing is for people like me who have food sensitivity, I need to know if there's something I'm allergic to in the food. And similarly, companies needed to know, is there something that they didn't want to consume in a software package? And then Kali, you'd mentioned also that there is no standardized format and everyone does it differently. I want to add in the reference in the show notes a link to work that CISA has done. CISA is the United States Cybersecurity ([Time 0:09:13](https://share.snipd.com/snip/cf8f3abc-102f-48cb-87bd-8af83d680689)) - **SBOMs and Cybersecurity Evolvement** - Since 2021, SBOMs expanded from license tracking to security tools due to cybersecurity concerns. - The executive order strengthened cybersecurity, making SBOMs pivotal for software supply chain protection. Transcript: Georg Link There was an important event that happened back in 2021. Brittany Istenes I felt like we didn't need to reference it for the PTSD of it all. Georg Link That's okay. Then I will not name the event, but there was something that happened early in the year in May, 2021. President Joe Biden issued Executive Order 1408, the Executive Order on Strengthening and Promoting Innovation in the Nation's Cybersecurity, which is a pivotal moment in the open Source story, because now the software bill of materials that had been developed as a means to understand license compliance were now looked at to answer the questions around security, Cybersecurity, and preventing big, bad events from happening. Okay, we decided not to talk about what happened in November 2021, that big vulnerability that ruined everyone's Christmas. So I just want to provide that background because it kind of helps understand why, and Kali, you've mentioned this, there are certain informations that today we wish was in an S-BOM That just traditionally wasn't required. Where is the original source for this so that we can look at the health of open source projects? And so this is where we are today, where we ([Time 0:12:00](https://share.snipd.com/snip/03030df0-a80d-4194-984e-3fb30ff24202)) - **Three Pillars of Open Source Strategy** - Good open source strategy relies on three pillars: license compliance, security, and project health. - Combining SBOMs with analytics moves companies from reactive to proactive risk management. Transcript: Georg Link Appreciate it. I think there are three things when it comes to a good open source strategy that SBOMS helped with. Licenses was the first one. The security really got a lot of attention ever since 2021. And that's what, yeah, Brittany, you were saying you were thrust into. And the third aspect now, as we're looking for long term relationships with open source projects, the health of open source projects, and this is where the chaos community comes in, The health of open source projects is the third leg of any good open source strategy. Because if you're a follower of the podcast, you've been listening to us for the last several years about how the health of open source projects is important, how we can use metrics to Understand this. And combining the power of S-BOMs with the power of the analytics that we can do can catapult companies from being reactive to what is happening in open source ([Time 0:15:50](https://share.snipd.com/snip/41ba6d52-78b3-4786-98f5-569ed27ed565)) - **Prioritize Proactive Risk Analysis** - Standardize and embed risk analysis into the software development lifecycle early. - Proactively incorporate compliance, risk mitigation, and project health to avoid reactive fixes later. Transcript: Georg Link Examples for this. Would you like to share some with the listeners? Brittany Istenes Yes, because we have been so reactive. And thank you so much for letting me jump in on this one. I think it's very important to not play, let's predict the future, right? I think it's very important to actually put in some standards and practices in place into your entire software design lifecycle. We need to take a step back before we immediately thrust into the actual development and understand risk, understand what you're going to build, and understand its impact from the Very beginning of the architecture standpoint. Right? If you're going to build something net new, you need to look at all of the components, particularly around going back, licenses, health of the projects that you're going to be consuming. And I definitely, Callie's got some wonderful insights and so do you, Georg, on project health. I can't wait for y'all to share that. And actually building the project the right way the first time. Well, there's going to have bumps in the road, obviously, but getting the right stakeholders involved and fully understanding the impact. And I think that that's a really proactive way that a lot of different enterprises can build in from the beginning. Right now we're playing catch up and I don't think we need to do that moving forward. So standardizing the process, fully understanding all of your compliance, ([Time 0:17:01](https://share.snipd.com/snip/94e93932-4e52-40f7-842d-728a555bc16f)) - **Indicators of Project Health** - Measuring project health includes maintainer activity, patch frequency, and overall stability. - Abandoned or unmaintained projects pose significant security risks that companies must address. Transcript: Cali Dolfi Your legal compliance side of things. I think that the SBOMs allow for a path of how do you figure out the information that you need to be compliant to things like the CRA. I think a big part of that, you have a lot of companies that are running into, okay, what are all of my dependencies? Actually, what are my involvement? Am I considered responsible for any of these? And SBOMs can kind of be used as a stepping stool to start to gather that information that if we're being honest, we probably all should have had in the first place. But also trying to see, okay, we don't need to be living our lives from the side of a fire drill. We can strategically use the information that we get from SBOMs to understand what our upstreams are and then looking at it from a project health perspective. We can say project health, but a lot of these things I think are pretty basic of, okay, I'm dependent on this project. Is there a maintainer that's been active in the last six months? Is there two that have been active in the last six months? Is this a project that has had any type of patches, let's just say a year? How many maintainers have been involved in the last year? How many patches have been going on the last year? Some really basic signs of, is this project being maintained? And if not, what am I as a company going to do about it now before the infamous, I'll say the boogeyman, the next XD happens? I know a lot of people came to me and be like, oh, how would you have found this bad actor? And my response has always been, I'm not looking for the bad actor. It's the nature of the project allowed it to be vulnerable to the bad actor of having it to be a single over works maintainer sets a project up to be taken advantage of. If ([Time 0:18:50](https://share.snipd.com/snip/c160284d-b181-4908-b02c-164083314af6)) - **Engineer Education on Dependencies** - Educate engineers about the origins and health of dependencies they import. - Implement checks and balances to avoid unknowingly using risky or unhealthy open source packages. Transcript: Brittany Istenes Yes, we can look into the community. We can look at the project. We can determine all of these different facets, the patches, the level of maintainers. Open SSF scorecard is another great way to actually see project health, which is absolutely fantastic. So shout out to those maintainers there. Maintainers running an open source project for the security health. But even taking it at a level inside of your enterprise, educating your engineers, right? These packages were not just brought in all willy nilly, pardon my silly Midwest language there, but these packages, these dependencies, these libraries were brought into the organization To be used in these particular spaces. And then all of a sudden, the engineer that brought it in didn't have that system of checks and balances, which we all have to put in now, depending on our kind of where we're proxy leveling All these dependencies, all of that. Do they know what a healthy project is? Do they see it's a fork, a fork, a fork? Do they just say like, oh, this is Steve Jones, Node.js patch, right? Do they know that? I don't know Steve Jones is I hope that doesn't exist. If somebody's Googling on GitHub right now, sorry, Steve Jones. But I'm just saying, you know, not educating our engineers, not having a system of checks and balances in as well is definitely a huge risk. And I think education is very important when ([Time 0:21:30](https://share.snipd.com/snip/5db227ee-682a-4879-ad18-0ae2f80f014d)) - **Using SBOMs with CHAOSS Analytics** - Biturgia used SBOMs combined with CHAOSS metrics to assess software supply chain risk proactively. - Analytics revealed unmaintained components, enabling better-informed decisions about dependencies. Transcript: Georg Link Critical component. And I'll share experience that we have at Biturgia. We worked with a customer and I can talk about this because we talked about it publicly. I'll put a link in the show notes where we gave a talk on this. The idea was we, one, need to understand what all we are actually including in our bill. So building that software bill of material was step one. And then step two was doing an assessment on how risky do we judge this to be. And we used metrics standardized by the Chaos Project to assess things. Callie already called several of them out. How many developers does this project depend on for the majority of the work? Like the contributor absence factor, also known as the bus factor, or we call it the pony factor, many different names. We also look at the people who have the knowledge. Are they staying? Or is there other people leaving? Because there's a lot of information about architecture, why decisions were made, etc. You want that longevity and people to stay involved and continue to work in an open source project. What's the growth of the open source project or decline in terms of number of contributors? Are people leaving? Are there signs that the technology is in decline? Of course, there are nuances to this. Some projects are just done and we need to look at other things. Is there a maintainer that is themselves active and available to jump in if something were to happen? Projects that are under active development, those things are important to look at. Also, what's the backlog of issues and pull requests? Is the project staying on top of the work that is coming in and closing things, or do they get swamped? We just had a project with another foundation where they were cleaning up over 7,000 old bug reports. And through the analytics that we provided with them, it really helped narrow the field and make intelligent decisions on what to do. So it's not just by age, all things just get lost, but actually the importance of things and taking the right actions. And we talked about this on ChaosGast with the free BSD project. Check out episode 103 for the details. A link is in the show notes. So there are many different metrics that we can apply. And so the work that we've been doing, we are just taking to the next level right now is how do we get this, not just at an analytic standpoint later as a controller, looking at everything That we have, but how do we move that information closer to the developer, the shift left and integrate it in the continuous integration, continuous deliverer, restage so that through An API call to the metrics platform, we can already give feedback to the developer upfront. Hey, this project maybe is not as active as you thought it was. Maybe you should look at something different. We just had something for the GrimoireLab project. I don't think we've talked about it yet, but there's a component in there that is moved to maintenance three years ago. We didn't even know about it. And if we had these metrics already in place, we would have realized that, hey, this project no longer is being actively maintained. So that's kind of what we're working on and some interesting things happening. And I know, Carly, we talked at the Linux Foundation Member Summit, and you're also working on some interesting things. Do you want to share some highlights? Yeah, ([Time 0:23:53](https://share.snipd.com/snip/d4ed19d5-2a4e-417c-bb05-263e4fcb52d0)) - **Managing Abandoned Open Source Projects** - Red Hat focuses on detecting abandoned projects and managing their end-of-life responsibly. - Decisions include whether to maintain, fork, archive, or transition away depending on project health. Transcript: Cali Dolfi Looking, I think you have the S-bombs, but even more generally, it's like, how do we look at our open source communities and the health of them from a security And that is something that I'm thinking and talking about a lot. And a lot of times that that linchpin is, okay, from the S-bombs, we can get the information about the upstreams that are in our products, if you're from a company perspective. And so right now I'm trying to work with security professionals all over Red Hat of trying to determine, okay, from your perspective and trying to get pool of them, what is the factors That go into what makes a project more secure or more vulnerable? What are those factors that you want to look at and how would you weight them? And I said this a little bit earlier, but the biggest, most prevailing thing that people want to look at is, is this project being maintained at all? You have the abandoned category, which I think is like one of the most vulnerable states that you can be in of having a project dependency, something that was kind of dropped off the face Of the earth. There is no more active maintainers. Is no more active development happening. And there was no intentional archiving of the project. Because while a project that has not had any updates or anything in a couple of years, if it was closed out gracefully and you have that this is an archive, these are the dependencies that It works on, and that's all documented. You can stick with that project to an extent. Those are two different situations, because if it stays within the realm that the project has left off of, say, hey, these versions, these things are compatible. If the project that you're running or your product that you're having is staying within those boundaries, great. If it's completely just abandoned, how do we figure that out? I'd go from you're looking at when was the last pull request, when was the last issue, when was there the last person that was active? A lot of the times, I think we're actually looking at this from, okay, what are those completely abandoned projects that we aren't even aware of that we're still dependent on and making Those initial decisions and haven't even gotten to that second step of, okay, now how do we analyze the health of this more holistically and looking at it more from how do we make a better Decision going forward? I think the initial ones on that side is, okay, how many maintainers are there? How consistent are they with their project? Is there a consistent churn in the project? Something moving super fast or super, like in those are arbitrary terms, something has a really fast development speed and something has a slower development speed. I wouldn't say that it's automatically one is better than the other for your use case, but that's good information to know. Can it keep up with the issues? Can it keep up with its pull request? Those type of things are you want to start to look at. But the thing that we're really focused on in Red Hat right now is how do we figure out the projects that are unmaintained that we are depending on? How do we properly end of life them if they were something that we were in some way involved in? Or how do we go about getting because it's a whole other thing is that if it's like been abandoned there's nobody there what do you do about it because you would want to go and end of life It directly on that repository make contributing upstream but you might be forced to make a fork and close out, archive that fork and kind of wrap up the loose ends. But those are the type of things, I think, sticking with that initial phase of what are the projects that have been abandoned or archived and what as a company that has this within our Products, ([Time 0:27:45](https://share.snipd.com/snip/4ad51775-30ba-4943-a084-8e3048445438))