Visualizing Open Source Data in React With Brian Douglas  ## Metadata - Author: [[PodRocket]] - Full Title: Visualizing Open Source Data in React With Brian Douglas - Category: #podcasts - URL: https://share.snipd.com/episode/4add0230-ed93-4356-9808-b3a0582e6ebc ## Document Notes ## Highlights - Navigating Challenges and Visualizing Open Source Data Open source data presents unique challenges, particularly due to the lack of standardized metrics for evaluating project health and contributions. As the industry has evolved, projects like React have emerged as central frameworks while others like VEAT enable various meta frameworks. Visualizing active contributions through tools such as scatter plots can help assess the vibrancy of a project. For instance, a concentration of recent pull requests indicates active engagement, while stagnation suggests bottlenecks, often linked to maintainers' review processes. Despite the wealth of available metrics from initiatives like the Chaos Foundation, confusion arises when users must navigate over 200 metrics without guidance on which are relevant for specific projects. Addressing this requires targeted visualization that highlights key indicators for different project stages, allowing for clearer insights into growth and governance within the open-source community. Transcript: Speaker 2 Yeah, no, that's a good example. My question was going to be like, what are the unique challenges with open source data in particular? I think you've given us a few there. Yeah. Like it is an interesting space. There's people thinking about this, but this kind of like tools for people that are working in open source as these like meta level tools is maybe not a thing we're thinking about a lot. Yeah. Is there anything in particular that kind of drew you to this problem or was just organically happened as you were like, this is a pain. Speaker 1 So I mentioned in passing 10 years of react to like that's a huge milestone. Like I came from a space where there were way too many front end libraries and we've centralized as industry around react today. Yeah. Obviously, Angular and view. It's like weird flex. I'm based out here in San Francisco. I was hanging out with Emma the last night from view fame. And he was talking about like the VEAT libraries. So VEAT's like the build tool that all these meta frameworks are built on top of. So remix and redwood and pretty much everything except next JS is built on top of VE. Right. So when you start looking at that like that impact, like how do you visualize if VEAT was removed from the situation or if there's another thing that will replace VEAT, like how do you See that visually in contributions? So we had this at the chart I called contribution distribution. So in the talk that I gave it all beginning again at Figma's conference, the first chart I built was contribution distribution, which is basically a 30 day scatter plot. I always joke because I got to see him statistics in college. So I'm not a statistician at all and like charts and math and stuff like that. Yeah. I can do it, but I'm more interested now because I have a pain point I'm trying to solve. So like how do you visualize if there's active movement in a project? So we have a scatter plot to the right is the most recent PRs in a project. And then to the left is like oldest PRs. So when you start seeing like way more PRs on the right, you see it's a healthy, active, contributing society project. If you see a bunch of distribution, like projects on the left that are just sitting there or maybe getting stale, then you can make a decision like, okay, there's a bottleneck here. Like whether it's like the maintainers, they have a long review cycle or maybe it says everything's this long review cycle or this thing like shipping crazy. There's a couple of projects like next JS where they'll have a bunch of PRs. So like ship daily. So you might have like 20 PRs like getting merged within the week and you'll see that distribution like our next JS shirt. But then you see something that moves a lot slower. I could go library. They get one or two. Maybe if you're lucky one PR a week, but there's a lot of stability in this like cloud infrastructure. We don't need to constantly swash a bug with rusters, type safety and type checking. So you get some validation that everything's safe. They don't have to always constantly try to reiterate and fix some like a hot fix. So like the answer your question that you were going to ask before, like the challenge is really there's no standards when it comes to like looking at data. And I think we've community called the chaos foundation, which is a community health and open source. They have a bunch of metrics. And I think what's good about that is there's a bunch of metrics. But the bad part is no one knows what metric to look at. So like if you have 200 metrics to look at and no one's a statistician, everyone gets seasoned in their college courses, somebody needs to say, okay, this is actually important for this Project and this is important for that project. And that's what we're trying to solve with is visualizing data in Durb react, but in open source is to say, hey, growth stage projects, early stuff that's come out in the last 60 days or Last year, like these are property metrics that you pay attention to, like new contributors. But when you're like a Kubernetes, which is getting like 200 new contributors above, okay, there's a lot of stability in there. There's like really strong governance. Yeah. ([Time 0:07:50](https://share.snipd.com/snip/911f26ac-32b5-4768-a277-3ce4d2b2b7ca)) - Exploring Ethical Representation and Categorization of Open Source Data The categorization and representation of open source projects raise ethical concerns due to the inherent subjectivity in showcasing project metrics. Different benchmarks may not accurately reflect a project's stability or success, as highly active repositories may mask underlying stability concerns. The complexity of categorizing open source projects arises from their relatively short history and the challenge of defining maturity stages akin to traditional businesses. Current metrics, like stars, biases perception during early project life and fail to represent long-term adoption. More valuable indicators include self-selection rates and external contributions, emphasizing the need for transparent, nuanced metrics to accurately reflect project viability and community engagement. This approach would shift focus from superficial metrics to substantive indicators of open source project health and community involvement. Transcript: Speaker 2 And that is represented in the data for fear of getting into stats land a little bit. And you know, the ethical concerns, there is a lot of decision power in that like, how do we represent this? And what do we showcase? What numbers do we show? Because as you said, it might not necessarily always be a bad thing for every repo if there's like a ton of front heavy activity and then like one PR a year. It's like some low level rust library that is in super heavy use, but it's super stable. No one's really even touching it. Yeah. Like how do you balance that? Because again, there's a bit of a responsibility there. We don't want to misrepresent any projects that are probably okay, but they're just not performing well by a lot of the benchmarks that we would typically look at. Speaker 1 Yeah. This is something that we do want to solve for. We don't want to do without the community support on it, but there's like a categorization of projects. Right. Like when you put like a project into a bucket of okay, this is a stable or feature complete. Like I think sometimes as developers, we feel like we can't say this is done. Right. So what am I engineers? He maintains a project called Cobra, which is a how do you build C.O.I.s and go? And it's such a feature complete stable project that it doesn't take much maintenance to maintain it anymore. Yeah. Because it's being levered by like almost every C.O.I. That's powered through go is built on top of the Cobra. Sure. So that's making changes. And for the most part, if you want something new, you go build something new, or you build an extension or a plugin. So like being able to categorize until like, Hey, this is now when the trajectory of startups, you have like early stage or pre seed funded startups and then you have growth stage startups Are IPO'd enterprise companies. And I think that's a very similar trajectory. And when you look at open source, an open source is still new. Like it's hard to put in a category on all these projects because we're like really less than 40 years of open source, like 30 of you really kind of like shake the tree a bit. Right. But in our lifetimes, like most projects will only be as old as we are. And that's wild to say out loud, but like what we need is not a foundation, but like a centralized like place to say, okay, this is what we're looking at. We have that at GitHub. So GitHub does give us like issue counts and PR's. Yeah. They don't make any statements about whether 1000 PR's are good or 1000 issues are good. But we do have stars as a metric. And I did a talk in London, very similar talk about unconventional metrics and open source. So visualizing these things. And everyone centralizes around the star and stars only really matter in the first like couple years of the project. You're not looking at adoption and traction through stars. You're looking at adoption, traction to downloads and contributions. But when the project first starts, you can't look at downloads and contributions because it's too early to make a judgment. So a lot of projects will get 10,000 stars in a month, but get zero stars in the next week. And I say that a lot because I think it should be okay that cool. Who cares about stars at this point? Like projects live, let's just see if people actually use it. So things that we've been looking at are things like rate of self selection, which is take the core team, take the organization who contributes outside that circle. So it's like something we're working on next. Like we already actually have in our API, but we now have a number associated with a percentage of who contributes outside the organization. So in Q4 of 2023, 0% outside contributions to TypeScript. So no one outside of Microsoft made a contribution to TypeScript between September and December of three, which is not a bad thing. It's an interesting statement, especially depending on like anybody does JavaScript, TypeScript is like leading the way of a future development. So if you're dependent on TypeScript, there's not a lot of influence you have in the TypeScript ecosystem. There's no TypeScript foundation. There's just Microsoft employees. And then frankly, X Microsoft employees are the ones that make outside contributions. Yeah. So at that point, do we question that or do we just continue to chug along? And like those are types of things that we can now show that story within this data. ([Time 0:11:24](https://share.snipd.com/snip/6cfc2b40-fba4-4a49-a93b-674e38c61f68)) - Exploring the Dynamics of Open Source Development Open source development has evolved into a complex landscape where individual contributors and large entities intersect, leading to nuanced relationships and interactions. The traditional dichotomy between the corporate world and open source has blurred, prompting developers to rely on major players for tools while also fostering small-scale projects. To navigate this landscape, new entrants are encouraged to engage with open source principles through structured courses that focus not just on technical skills, but also on networking and community interaction. The emphasis is on understanding how to contribute meaningfully, rather than merely fixing low-impact issues like typos. Platforms are emerging that aid in discovering active projects, thus enabling developers to find opportunities based on their interests and skills. Additionally, initiatives like 'Star Search' aim to spotlight up-and-coming developers who traditionally remain unnoticed, ensuring that contributions from less-recognized maintainers gain visibility. There is a clear movement towards building a more inclusive recognition system within the open source community to highlight contributors who engage in essential, often uncelebrated, work. Transcript: Speaker 2 Yeah, it is super interesting. Like you said, open source is fairly new 40 ideas or something like that. It does feel like we're hitting more dimensions to this now than there were really historic like Microsoft's big role and like the Linux community has been there for a long time, I think, But it always felt a little bit more disparate from the traditional corporate world. And there was like the open source world, not, but there's all this weird strata in between now. It's like all these funky interactions and relationships. We still have a lot of just individual contributors who have their few little handful of projects. This is the thing I maintain, but yeah, a lot more of just like the discussion around open source, I think is on these like big things that are shepherded by one entity. And this has been a pattern that's been emerging for a while. It's just so many of us as developers are relying on these tools that are like one kind of firm is really the. Guide here and yeah, it's interesting to think about that relationship and how these different players are like navigating the space and making decisions about what code to pull in Or work on day to day. For someone coming in new, what would you recommend? Like how do they even make sense of any of this? Cause it's like it is, it is such an interesting landscape at this point. Speaker 1 Yeah, the warm new and open source, like a lot of things gets attributed to like brand new college students bootcamp grass. If you're in that space, like we have an intro to open source course called intro.open sauce pizza. We put everybody there so that way you know how to interact and now how to look at PRs and issues and like really avoid like. Cause like there's a huge movement around like this fixing typos in open source. And I think there's a space for that, but I think we've over indexed on that. And we prefer to have folks like level up and know how to operate. Like I was just at an event last night is like the networking is the thing that's challenging to people. Everyone asked a question like, Oh, so what do you do? Yeah. What does that mean? Yeah. I write that question. What do you have for lunch? Like any cool restaurants local? Yeah. That's a better opener. So like we try to teach people like how to ask about their lunch as opposed to like, Hey, job, please or give me issue. But there are a lot of folks who have been engineering for years, folks who are stakeholders that depend on open source and don't even know how to interact as well. How do you even know where to start? And so our opportunity there is like we have this explore page where you could just. If you know JavaScript or if you're interested in JavaScript, we've got a bunch of JavaScript projects. You can see basically we have a list of last committed repos. So we have a list that we cash over in the last hour and we'll just show you what got a last push to and then like most active. So projects that are like up and coming in the go ecosystem that actually take contributions and still have merges that happen. Like we show you those projects. And it's really just like this kind of what you're appetite to know. There's open source that's happening. But the real goal is like if you depend on open source, if you participate in the ecosystem by like consuming open source, like the best thing you could do is just know who's building and What we're really trying to build right now is we have these profiles. We have this thing that we're going to be shipping in two weeks called Star Search. And the idea there is in the 90s, I used to go to my grandma's house and we'd watch a show called Star Search and like Britney Spears, Beyonce, Justin Timberlake, Ryan Gosling. They're all on Star Search as kids. And the product that we're building is like, what if you could find developers who are up and coming before they're famous, like before they're standing on stage and doing keynotes? And this was like, come down to really a concern that we had when we shipped get up sponsors, which was if we shipped get up sponsors, the folks who get sponsored or the folks who are the Most famous. So like, how do we combat against up and coming folks who don't get the attention they need? It hasn't been a problem that's been solved yet. And there's not a lot of attention being driven to that. So there are maintainers who are doing thankless work behind the scenes who have never been seen. My goal is like new people find new people and become friends. The hope is that we can now showcase folks who are really doing the thankless work and can now be showcased. We sponsored a tool called OSS.Love, which it's really the stinky cards to open source maintainers. And the hope is that we can start integrating some of that stuff into our platform as we build this discovery engine for open source. ([Time 0:23:09](https://share.snipd.com/snip/7fd19218-9537-430c-b13f-88d148efe858))