The World of Open Source Metadata  ## Metadata - Author: [[The Changelog: Software Development, Open Source]] - Full Title: The World of Open Source Metadata - Category: #podcasts - URL: https://share.snipd.com/episode/329ad937-b7cb-47ad-932b-93e114602f20 ## Document Notes ## Highlights - **How 24PullRequests Sparked A Data Mission** - Andrew started with 24pullrequests to get people contributing to open source and found many projects were abandoned or hard to contribute to. - That experience pushed him to build metrics and tooling to surface healthy, active projects at scale. Transcript: Andrew Nesbitt Yeah, okay. So I have been swirling around the world of open source metadata for, must be coming up to 10 years now, starting with 24 pull requests. That's right, 24 pull requests. That didn't kind of start from metadata, but the idea of that project was to encourage people to contribute to open source as part of kind of the run-up to Christmas. And after kind of like first getting that off the ground, we quickly ran into like, oh, how do we suggest, like where should people go and contribute to? And a lot of people would try and send a pull request to a project that had no activity and like the maintainer was gone. Or just like were struggling to be able to even like work out how to send a pull request to some projects because they were really not very friendly or easy to contribute to. And that kind of led me down this path of like okay well what's a how do you define what a good project is and then like can we scale it up rather than manually having to have people kind of Like submit their things and keep those things up to date every year because that project would just kind of come and go every December and shut down afterwards so the maintenance there Couldn't be entirely human because there was thousands of people contributing to that project and like sending pull requests. And it was a lot of data to ([Time 0:04:36](https://share.snipd.com/snip/be322909-888b-4b6a-97df-c69a81f46a02)) - **Dependencies Reveal Real Usage** - Mining dependency graphs gives a far stronger signal of real usage than stars or raw downloads. - Removing a dependency reduces counts, revealing breakage or abandonment more reliably than stars or downloads. Transcript: Andrew Nesbitt But download stats, again, available for most package managers, but not all, is often kind of wildly all over the place for certain projects, especially if they're used a lot in CI, That you'll just see like really inflated download stats. For dev dependencies, the things that people, especially maintainers, are installing on their laptops to be able to work on those projects. But they're not necessarily a runtime dependency of all the applications. There are definitely gems that Ruby and Rails devs use locally, but aren't shipped with the Rails app. So you would never see those numbers. And the insight that I kind of accidentally tripped over was, if we go mining the dependency information out of open source repositories, at a large scale, you actually start to get A really good picture of how people really open, like use open source, how they don't use open source. Like if a project breaks, you probably don't go and unstar that project. Let's be honest. Like not many people are unstarring things. They don't remember. And also you don't like undownload a thing. ([Time 0:07:32](https://share.snipd.com/snip/bb8de1cc-e15e-4c16-aa2c-5ffd1cc49a1a)) - **Libraries.io: Growth, Sale, And A Do-Over** - Libraries.io grew big, expensive, and hard to sustain; GitHub's dependency graph and Dependabot shifted the landscape. - Andrew sold to Tidelift, open-sourced the code and data, then later rebuilt the project as ecosystems for research infrastructure. Transcript: Andrew Nesbitt And we, me and Ben, who are working on it, were like, well, what are we going to do? How can we turn this into a, you know, a sustainable project that can fund itself? And we, at the time, GitHub had just implemented its own dependency graph as well, along with purchasing Dependabot. And that basically, they started giving that away for free. That pulled the rug out of any plans we had to monetize libraries.io directly, as well as a project I was building called dependency CI, uh, which never really got off the ground, but Was back in the day was like, Oh, this is really cool. Cause it could literally like block your pull request to say, you're trying to add a dependency here that is not good because it doesn't have a license or it's got security issues or other, Um, things. And so we ended up selling to Tidelift, uh, to try and find some way of recouping the costs of building out that project. But just before we did, we also made all of the code open source and all of the data open source. ([Time 0:09:59](https://share.snipd.com/snip/3f56a51f-74e6-4f5c-8486-891f33e20562)) - **Give Maintainers Downstream Visibility** - Build tooling to show maintainers their top dependents, pinned versions, and transitive users so they can proactively coordinate. - Use that visibility to prevent breakages, reach out to key downstream projects, and prioritize fixes. Transcript: Adam Stacoviak Those are all still users right i mean if i'm that person uh releasing 100 times i'm still pulling the packages right every time they commit yeah it's like yeah boom new version because Andrew Nesbitt I can you know and also to be able to go like if we can flip that graph upside down and show you here are the key, like people downstream dependencies of your library, then rather than you Find out that you broke them because they come into your issue tracker after you just published that release and say like you broke stuff. Building a CI that is like an inverse that goes, okay, well, you committed something. Let me go and test this against your downstream, like your most popular downstream users to make sure that you didn't break those things. And there's some difficult bits there in making sure, you know, those downstream CIs are reliable. They're not just going to be like, oh, actually tests pass all the time regardless or they fail all the time so you can't trust like if you actually broke anything or not but to be able to Do that would give maintainers insights that would be like they can actually be proactive about some of these things and maybe even be able to coordinate and go like oh i'm able to reach Out to these projects and say like i'm gonna break this thing or i'm going to change this thing to make it better. ([Time 1:17:19](https://share.snipd.com/snip/4e5daad3-5d1c-4913-a65d-60db20138fea)) - **Project-Level Telemetry Respects Maintainers** - Ecosystems prefers project-level telemetry over invasive individual tracking to respect volunteer maintainers. - Aggregate signals (projects, dependents) offer actionable insights without spying on people. Transcript: Andrew Nesbitt Yeah, so I can imagine you would see like, okay, well, for this particular package, and maybe I've got lots of packages, i just drill down to one of them then i can see here are like my top Dependents and top being there's lots of different ways you can define what a top would be but we can just use the ecosystems like usage metrics is one thing here are like the key projects That are using your stuff and then which versions they're currently pinned on as well so they might be just like oh i always pick up the latest version i'm i've got depend about doing the Updates but maybe there's someone who's really heavily using your stuff but they're actually pinned to an old like an old major version and that's like an insight into, okay, well, Why were they stuck? Like, maybe I can go and help them upgrade, or I can learn that actually, I made a, the most horrific breaking change ever. And they really, really don't want to upgrade because of, you know, it completely causes them too many headaches to do that. And maybe I can consider that in, like how I then continue to maintain that project going forwards. You could also then use that interface to say, okay, well, can you show me everyone that's on this specific version? Like, or which is a 50% of my users like stuck on an old version? Or are they stuck on an insecure version as well to be able to go like well we had this CVE three months ago and most people especially thinking about this from a like individual packages That depend on me to be able to see the knock-on effect of like all the users of those packages are like my transitive users there's a lot of data there but being able to highlight like where Those key points are of leverage that are like these things could be improved here that um would be one way of that kind of being manifest the other way you could do it is rather than it be A ui is more like a notification system of being like oh did you like you got the proactive kind of things of like your dependents have updated to your latest version uh or that like your Dependents are having problems with they tried to upgrade to this thing and like here's the context of this depender bot pull request and the discussion that they had and they haven't Yet merged it to be able to like show you that oh wow okay that's interesting like it's having a problem for them that we didn't even imagine because uh we're not using the same database As they are for our testing purposes something like that uh and there's maybe there's an ai element there once you get to like very large amounts of users that you're like actually this Is too many people to kind of like too many downstream users to reach out to maybe i can empower uh copilot or claude through some kind of prompt that is like i've described the changes In my changelog in a way that helps them upgrade from one version to the next. But there's a lot of people that are very reluctant to take on some of those things because they can be wildly unreliable sometimes when you try to do things the same way over and over and Over. Adam Stacoviak It's kind of like telemetry via exhaust too. You're not like literally tracking your users. You're tracking them through natural usage pattern of the ecosystem of open source. So you're not like asking them to opt into too much telemetry either. ([Time 1:21:31](https://share.snipd.com/snip/f4cc05bc-827c-4af1-8059-8d1d4664fb29)) - **Community Signals Are Hard To Scale** - Non-code contribution signals (chat, Discord, Slack) are valuable but often private and messy to index at scale. - Large-scale community measurement quickly becomes costly and ethically fraught. Transcript: Andrew Nesbitt Oh, so yeah, the funding one is a big one. The other thing that I'd really like is kind of more data around the non-code contributions, but that's really hard to get, right? Yeah. Your Discords and your Slacks are not open enough to be able to really index without, you know, you need an API key or you need a ghost user sat in a discord. Right. Now you're getting creepy. You're getting real creepy. It is way too much. There are tools. There's ecosystems again tracking us. Adam Stacoviak Get out of here. There are ecosystems. Andrew Nesbitt Start joining all the community Zoom calls with an ai trend like um chat log kind of thing but now the there are tools like that um bitergia has one that can uh you can configure it to track Your own community and like you can feed in mailing lists and you can feed in your slack uh or your discord or similar but you're kind of doing that at a per community or even just a per repository Level. Trying to do that at a mass scale is, you know, is stepping into worlds that I'm not really comfortable in terms of the amount of tracking of stuff. Also, it's just really, really messy. Like, open source metadata messy but it's not it is like tangibly okay yeah i can see how i can connect the dots here whereas once you get into like unstructured text of discussions of Things is uh you're quickly into like right well we're just going to try and have LLMs process everything here. And it's a horrible mess and it's incredibly expensive. We use no LLM stuff in ecosystems because we just don't have any budget for that kind of stuff. The amount of processing to analyze 12 million packages. ([Time 1:27:15](https://share.snipd.com/snip/beccaebe-8fdb-43f7-a556-1c2483cab36d)) - **Help Build A Practical OSS Taxonomy** - Contribute to Andrew's OSS taxonomy to improve discovery across dimensions like role, technology, and domain. - Use standardized facets instead of free-text keywords to make project discovery more precise. Transcript: Andrew Nesbitt The other thing I just started noodling on is an open source taxonomy. So to try and define like a taxonomy that describes the different facets of what makes an open source project. You know, what does it do? Who is it for? What technologies does it use? There's about six different facets and about 130 different terms that I put together as like a V1 kind of thing of going like, if you were to put these packages into a box or six boxes, like Which ones would it do? Rather than just going like, here's some free text keywords. Like here's a load of the kind of chunks of things, including like the role of the user as well, rather than just thinking about like, oh, it's a front end react app. It's like, but it's for a, is it for an end user or is it for a sysadmin or for a developer like to be able to and then what domain is it in as well uh it's like really early but i'm hoping it is Another way that can produce some alignment in this open source discovery world because you know i worked at github for a while on open source discovery and didn't wasn't really able To make a good dent in it there but i think there's still a lot of low-hanging fruit in terms of like just helping people find the right kind of tools to use because not many other people Have really it's also there's just not a lot of money in that space it's a lost leader right for most companies is like searching for open source is not going to turn you into like oh yeah You can't even run a lot of ads against that kind of stuff because open source developers are like the number one user of ad block so those ads will disappear pretty quickly uh but the i'm Hoping that that taxonomy will be like here's a nice blueprint of ways that you can define your project and put it into ways that then allow you to kind of go like, okay, well, I've got five Dimensions here, but I want to rotate around one of them. I want a web framework for researchers, but I want to rotate about the technology. Like, what are my options there or i'm definitely in this technology space uh and looking at this kind of like position in the stack but what options do i have here for different like users Uh and to be able to kind of like twist the picture a little bit but in a fairly defined space rather than in, you know, just arbitrary free text. Because again, you run, you just end up in this like soup of words, which is like, yeah, we kind of just get very fluffy. And often projects just don't have very well-defined ways of finding things. You know, like they don't add a description to their GitHub repo or any keywords or topics. So you just kind of like never find it unless it's in a generic search engine, which is then really hard in terms of like, oh, well, what are my options in this space? And I made this as just like, surely someone has made one of these already. And I found a taxonomy of software in the research space, but I did not find a taxonomy of open source software. So I was like, okay, I can make a stab at one of these. I like, I've never made a taxonomy before, but I put it together as like, this should be interesting. And it's been useful so far. And it started some interesting conversations but i really need some people with more experience in you know actually defining taxonomies than i have to to give more input and also Expand it and like cover the problems because i'm pretty sure there's going to be loads of problems in it because i basically just like put it together in a couple of days as like okay i Think this this should work but you know yeah mostly where does that where does that live that is on the ecosystems github is also a really uh quick um web page i made of taxonomy.ecosystems Um it's not it's literally just a few days ago. ([Time 1:31:59](https://share.snipd.com/snip/12d0df2f-271d-4ab3-8d1d-e090cac08d38)) - **From Solo Builder To Open Collaboration** - Andrew has been heads-down rebuilding ecosystems while later seeking collaboration and contributors to shape its future. - He welcomes feature requests, partners for security work, and people to join working groups on package metadata. Transcript: Andrew Nesbitt Well, I have had my head down for, like, basically since leaving Tidelift and then COVID happening, I basically just, like, got my head down and just started, like, plugging away. I also started doing track days in a Subaru BRZ, which is an excellent way get away from the computer if you if you've got a interesting cars track days is brilliant fun but ecosystems Has kind of like been building up and building up and it's now reached the point where I'm like I need more people helping kind of like not just contributing to the code but like helping It work out where it should go next because i can definitely come up with lots of things i would like to see happen but there are i need more input from more people on like how would you like To have a impact on the open source world like through data um So that's input in like feature requests or thinking about that from a slightly higher level of like collaborations, ways That ecosystems can support different efforts, be it like security, searching for projects that are like, oh, there are ways we can improve this part of an ecosystem. The like collaboration is really what i would like to see more of and i am starting to do more um podcasts and like various kinds of i started a working group with the chaos metrics people Around package manager metadata as trying to share the kind of learnings that I've done in developing ecosystems and being able to kind of like map metadata across different ecosystems Into standardized ways. ([Time 1:39:39](https://share.snipd.com/snip/6ed01c4c-6d8a-47e0-abfe-3f3da6ecf78c))