How Is AlmaLinux OS Is Community-Driven? With Benny Vasquez  ## Metadata - Author: [[Hanselminutes with Scott Hanselman]] - Full Title: How Is AlmaLinux OS Is Community-Driven? With Benny Vasquez - Category: #podcasts - URL: https://share.snipd.com/episode/59cd8ff6-7d5c-4ebd-9fe6-52481da23f31 ## Document Notes ## Highlights - **Benny's Path to AlmaLinux** - Benny Vasquez got involved with AlmaLinux because he was a CentOS user. - He was asked to help translate technical details for the community and agreed enthusiastically. Transcript: Benny Vasquez Interestingly enough So I got invested and involved in Amalynix because I was a user of CentOS. And when Red Hat announced that they weren't going to be producing the downstream version anymore, we as a community needed a new version to replace it. And Alma Linux got spun up the people that were involved for people that I already knew in the industry. And they came to me and said, hey, we need somebody who's good at translating tech to non-tech and just generally good at community in general. So, like, do you want to come help? And I said, yes, absolutely. ([Time 0:00:53](https://share.snipd.com/snip/290ad46a-cbcc-48ce-9864-257cf1ad1dcb)) - **CentOS's Evolution and AlmaLinux's Need** - CentOS was a community-built, free downstream version of Red Hat for nearly 20 years. - Red Hat's shift to CentOS Stream left a gap that AlmaLinux stepped in to fill, ensuring enterprise-grade Linux availability. Transcript: Scott Hanselman That what CentOS did when they changed their focus. Benny Vasquez Yeah. So let's go back even a little bit further, because for anybody that doesn't know, CentOS used to be a version of Red Hat that was just built by the community for free because Red Hat is Open source. So anybody that wanted to use Red Hat but didn't want to pay for it, they could use this thing called CentOS, but they were using it with their own support. They were using it with community support, I guess is the way to put it. It was people supporting each other. There were almost 20 years of this operating system that was enterprise grade but free and didn't come with support. And if you understood Red Hat, then you could use CentOS, and it was fine. It was used everywhere, in every single industry that you could possibly consider, it has been used. I think when they announced the end of the traditional downstream CentOS port, there were something like 40 million installs, active, getting updates that they knew about. So a lot. And when that organization was set up, there wasn't really a central ownership structure. There wasn't really any kind of centralized governance of it at all. And over 20 years, you know, people get tired and it was kind of hard to bring people in and it was kind of hard to really know how to hand off the project. And they ended up working on a deal with Red Hat where Red Hat hired all the engineers that worked on CentOS and took like ownership and responsibility for the project. That meant that, you know, now Red Hat kind of gets to drive the project. And a couple of years into them having brought everybody on, they introduced this new thing called CentOS Stream. CentOS Stream sits in the build tree upstream from Red Hat. So it's just ahead of Red Hat. It's slightly more updated, slightly more definitely updated more frequently, that kind of stuff. And then about a year later, they announced that they would no longer be producing the downstream version, which is, like I said, the version that everybody had been using forever. And as users, we needed it. So we started building it. ([Time 0:01:40](https://share.snipd.com/snip/e550def3-af6e-4438-b113-d404b9f480d9)) - **Red Hat’s Business Perspective on CentOS Stream** - Red Hat created CentOS Stream to get earlier feedback for enterprise releases. - Red Hat viewed downstream CentOS as a competitor harming their business model. Transcript: Scott Hanselman Like, they just like, don't want to pay for this thing that we're not getting any money for? Benny Vasquez Well, I don't think they ever actually provided a reason that wasn't, you know, this is better for our business or whatever. But if you look at it from their perspective, the goal with CentOS was to get, for from a business perspective, is to get their users invested in testing and in providing feedback. But if CentOS is downstream from Red Hat, they're getting feedback too late. That's why they created the thing called CentOS Stream, so they get feedback just before Red Hat. They get it just before they're building the actual enterprise version, right? And that means that it was a lot easier for them to get their customers invested and a lot easier to get good quality feedback. And from that perspective, all they saw was that they were building a competitor to their product, right? ([Time 0:03:58](https://share.snipd.com/snip/6149c93d-0929-4b3a-ad4f-3c78be96d5a7)) - **AlmaLinux's Build Process Adaptation** - AlmaLinux builds from the same source Red Hat uses but no longer from the restricted git location. - They maintain release compatibility without strict one-to-one rebuild of CentOS Stream. Transcript: Scott Hanselman Is it downstream? We're downstream. Benny Vasquez Yep. Yeah, we take Red Hats, well, not anymore. We kind of take Red Hat sources. We match Red Hat for what they are building and releasing and are Red Hat compatible. Scott Hanselman Okay. And that compatibility is the most important thing, right? Like, that's the thing, because it's all about a promise of one-to binary compatibility with Red Hat. Benny Vasquez Well, yeah. So we would not call ourselves one-to Oh, really? Because, yeah, because in 2023, Red Hat announced that they would no longer be shipping the source code to git.sentos like they always had. They'd always published the source code for Red Hat there. They changed where it was to a place behind a user login. And that user login, when you create an account there, you agree that any code that you access there, you will not use to build an operating system. And that meant that, you know, it effectively completely broke our pipeline. And in order to fix that, we started building from the same source as they do, from SoundToS Stream. But we still are matching them for releases, release numbers, release versions, all of that kind of stuff. We're not rebuilding SoundToS Stream because that wouldn't make any sense. We're still building Red Hat just from the same place that they do. ([Time 0:05:23](https://share.snipd.com/snip/3238d10d-ad5f-47ee-ba84-65e788077a42)) - **AlmaLinux's Flexibility and Responsiveness** - AlmaLinux can add missing features and provide extended hardware support that Red Hat dropped. - It delivers bug fixes faster than traditional vendors, improving enterprise usability. Transcript: Benny Vasquez Because one of the things that we can do now that we're really excited about since the Red Hat source code shift, we actually get to build and ship things that are outside of what Red Hat Does. So we've been able to add on top of it, add features or add back in hardware support, which is one of the big things that shifted. For example, in 9, there were a bunch of like raid driver or raid controllers and that kind of stuff that got dropped in support from Red Hat for reasons that we couldn't really find. So we added them back in because we could, because it's easy and it doesn't, since we don't have to match Red Hat exactly anymore, we have that kind of freedom. Everything that still works on Red Hat, everything that works on Red Hat will still work on Alma, but it's that kind of stuff. So people that need additional hardware support, people that need, one of the other big benefits we get is we can ship bug fixes faster. So if there's a bug that's been fixed upstream, but that fix hasn't been pulled into Red Hat for some reason, we can pull it in. And it's that kind of stuff that really starts to differentiate us. ([Time 0:08:06](https://share.snipd.com/snip/a186f115-4749-4e79-991d-e36bd68a2b2a)) - **Red Hat Supports AlmaLinux Differentiation** - Red Hat encourages AlmaLinux to innovate rather than just rebuild Red Hat. - AlmaLinux maintains friendly relations with Red Hat as both projects coexist. Transcript: Benny Vasquez Like one of the biggest things was what they said was you can't just rebuild red hat and not improve it like you can if you want to do stuff make it something different and that was obviously Something that sent to us never did right they it was just red hat and because it was you know a collection of people a handful people, there was no motivation to add features or add anything Else. But one of the things that we've been able to do is kind of improve on it. And so they say, yeah, no, that's fine. Do that. Serve the community that you need to serve that we're not serving. We're okay with that. We're actually, we're buddies with all the people at Red Hat. Like, it would be silly to not be because if they don't exist, we don't exist. ([Time 0:09:18](https://share.snipd.com/snip/c8432dd6-1540-4b72-86f3-a17d11cd658f)) - **Choosing Between Red Hat and AlmaLinux** - Choosing Red Hat is mainly for commercial support offered directly by Red Hat. - AlmaLinux users can get commercial support from certified partners independently. Transcript: Benny Vasquez That's the thing that ends up happening. There are definitely commercial support providers that you can sign up with that will give you support for Alma Linux. Scott Hanselman But like Alma Linux has Cloud Linux and certified partners. And like there's folks that you could buy support for. Why would I want to get Red Hat when I could get Alma and then buy support from an Alma Linux managed service? Benny Vasquez I don't know. I think it's, I said, I think it boils down to, if you want to have the commercial relationship with Red Hat, because then you have somebody to call. ([Time 0:10:37](https://share.snipd.com/snip/abc831da-a521-4cf8-bd1b-cb8d74c3ac38)) - **Open Source Governance Challenges Today** - AlmaLinux embraces meritocratic, community-driven governance as a return to open source ideals. - Operating openly is challenging today due to new business and social complexities. Transcript: Scott Hanselman Well, which is great because that brings me to the larger question, which is this is a truly community-driven distribution, right? Contributions from anyone, you have a goal of a meritocracy when it comes to decision-making. Like, it sounds weird because it is the ideal, but it also feels like a throwback to, like, the way that it used to be in the 90s. Like, has it been challenging in the 2020s to be running a project as if it was the benevolent open source of the late 90s? Benny Vasquez Oh, of course. Especially with the way that the shifting of both business feelings around open source and public perception of open source has shifted. There's two things, right? There's especially the business side. You've got people who say, you know if you say you're open source and you the only familiarity that you have with open source is like single contributor open source projects like i can't Think of any of the big ones right now but like companies that own an open source project and they control it and they decide what actually goes into it without accepting any external Work or very little external work, then you're not going to really understand what an open source operating system would look like. And so that is definitely a challenge in and of itself, is just proving and pointing out to people how it works. And the other side is the, what was the other part of the question you asked me? Scott Hanselman No worries. It's trying to understand how you've created a foundation and you're running it in an idealized way. Like it's 2025, social media exists, politics exists, technology is political, but you're running it in a very chill way. ([Time 0:11:28](https://share.snipd.com/snip/023b8720-b0e7-49e7-8b7c-9a5b85cccca8)) - **Necessity of Neutral Governance** - Governance must remain neutral to avoid dominance by vested business interests. - This neutrality ensures AlmaLinux serves the broader community, not just one agenda. Transcript: Benny Vasquez Other thing that we have to pull into, or the other thing we have to fight with, is because people understand the value of open source now, it is absolutely critical to maintain that neutrality In our governance. Because if you, as a business, understand the benefit of open source and try to come into a project that you have a vested interest in its continuation and really want to have the ability To push the project in the way that benefits you the most, that's not going to serve the community for AlmaLinux very well, right? So we have to make sure that whatever governance we set up is neutral and has the ability and the requirement to consider more than just a few opinions. ([Time 0:13:14](https://share.snipd.com/snip/d127001f-ddd1-4363-8d2c-ce7aa8dff143)) - **Project Integrity Over Funding Influence** - Funders paying full-time staff must let the project direct the work, not the funders. - AlmaLinux prioritizes project integrity over influence by money. Transcript: Benny Vasquez Funny because that exact thing has been coming up for us recently. Okay, cool. So it's been, it's been a discussion because we have to decide where we want to land. Like the foundation right now, our annual budget is $,000. We don't have... Yeah, this is not millions. No, it's not millions. Scott Hanselman Yeah, I've looked up the filings. Benny Vasquez Yeah, we do not have much money, right? And the benefit that we get right now from our sponsors, those range from obviously some cash but also we get some just like credits on like azure or aws or anything like that or we get hardware Or hosting we get bandwidth we get services that kind of stuff and a couple of our bigger sponsors actually employ people to work on the project which is one of the things that i learned About when i was working at microsoft with you i was i figured out that we could you know convince people to hire people full-time to work solely on elma linux and it's been really really Great and those are the sorts of situations where people start to feel like they have the ability to say yes or no to things right well i pay employees, so I get to decide what they work on Or what they don't work on. And we have to say, no, actually, the project decides what they work on, because that's what you're doing. You're paying them to work for the project. You're not paying them to work for you. And when that distinction has to be made, I think it gets really interesting. And you start to see, let's say, integrity of the project, right? You get to start to see if the project really is willing to stand up and say, no, we're serving our community, whether or not you give us money. And we like your money because it helps keep us going. And we like that you pay developers because those do the work. ([Time 0:14:31](https://share.snipd.com/snip/43100dec-4207-435a-a208-144ff5b19625)) - **Current Funding Limits and Growth Plans** - AlmaLinux foundation currently pays no employees due to limited funds. - Significant funding would allow hiring to strengthen neutrality and capacity. Transcript: Scott Hanselman Zero people. Okay. And that's on purpose? Yeah. Benny Vasquez Okay. Scott Hanselman Yeah. Benny Vasquez I mean, at this point it is because we don't have the money to pay them. Right. Scott Hanselman But if like 2 million bucks showed up tomorrow, would that mean you'd hire people or would you support your infrastructure? Where would you do with an infusion like that? Benny Vasquez We have a list of like, we wish we could do this. If two million dollars showed up and we thought it was going to stick around, we would definitely hire somebody. We'd hire maybe a couple of somebodies, depending on where we wanted to be able to put things. Because then we'd start to like pull even further into that like absolute guaranteed neutrality right ([Time 0:16:45](https://share.snipd.com/snip/ff649e5a-d47c-4d9c-857f-5b70507c7667)) - **Sustainability as Success Metric** - Success means 10 years of stable OS support and a foundation that outlasts current contributors. - AlmaLinux’s governance structure is designed for longevity beyond the current team. Transcript: Scott Hanselman So then what does, what does success look like? Is it just ongoing continued support and making sure that people can count on things? Cause it seems to me like y'all have a long perspective. Like you want people to have reliable and stable operating systems for a decade, not six months of support. Benny Vasquez Yeah, exactly. I want the, the big thing for me is both from the operating perspective, the operating systems perspective, which is, you know, 10 years of support and making sure that things continue To be stable and long-term the solution that people need, but also for the organization. So I don't want to set up an organization that relies on me. Like I am, you know, a relatively stable person now that I'm over 40 and have all of the things that I need in my life. But I also know that I'm probably not going to want to do this for 20 years. So making it independent in a way that is going to outlast me, is going to outlast the people that are working on it now, is going to continue to have the long-term support from the community That it needs in order to survive for 20 years or more. That for me is the success. Setting it up so that it outlasts everybody that's here right now. ([Time 0:21:45](https://share.snipd.com/snip/6834adcc-4382-4820-8a71-5ec50bbec058)) - **Balanced Governance Model** - AlmaLinux balances elected board governance with a self-managed engineering steering committee. - This hybrid model enables both community representation and technical direction. Transcript: Scott Hanselman Yeah. Benny Vasquez It was really important for us that it's both completely independent and cannot be bought or sold. So setting up a nonprofit was the top of the priority list because in the U S nonprofits can't be bought or sold. So it was super important to us that that distinction was made. And then all of the governance, the board is all community elected. It's all elected by members of the AlmaLinux Foundation. And the balance to the elected board is the self-managed and self-appointed Olesco. It's our AlmaLinux engineering steering committee. And to me, that balance of both elected and self-managed means that we're getting the best of both worlds and we can kind of steer the project in the way that it needs to be steered. ([Time 0:23:44](https://share.snipd.com/snip/43480c85-1a0e-4eeb-900f-82e6ed9c7420)) - **AlmaLinux in VFX and Desktop** - AlmaLinux adoption extends beyond enterprise servers to desktops and VFX render farms. - At SIGGRAPH, Vasquez learned about widespread use in VFX industry for rendering and desktops. Transcript: Benny Vasquez It honestly doesn't because the thing I keep calling back to is it's, you can use it anywhere you want anywhere that you would use Alma or, red hat i mean and we actually even have an atomic Sig that is focused on a desktop version of almonix like desktop images because they find it to be so stable and so good and so useful in their environments that's what they want to do our Whole uh i went to an event for the first time ever last last year called siggraph siggraph is a vfx focused event and i had no idea the adoption that we had gotten in the vfx industry until I went there and there's people from cine site there's people from ilm there's people like all over the place that are like yeah we just switched our entire render farm to you and all of Our desktops use you i just like, this is insanity to me. Scott Hanselman Because they value stability. Yeah, stability. It has to just be stable and work. Benny Vasquez Exactly, exactly. And we're customizable and like all of the things that they need. It's just, it's been awesome to see that, you know, we are being adopted both for crazy, you know, the render farms and the high performance computing and all that kind of stuff. But also desktops. Because you said the year of the Linux desktop, and that's been one of the long running jokes in the Linux world, right? It's been happening for 40 years or 30 years. Exactly, exactly. But we actually hit 5% like a month ago. ([Time 0:24:44](https://share.snipd.com/snip/9bcea5af-f557-40f9-820d-7cc373ff0e7d)) - **Linux Revives Old Hardware** - Linux enables old hardware reuse, helping users extend life of older laptops. - AlmaLinux supports this trend actively including hardware dropped by Red Hat's recent versions. Transcript: Benny Vasquez Because one of the big things is there's a bunch of really valid hardware that can continue to be used, but isn't going to be supported in windows 11 but it's you know very easy to keep keep Scott Hanselman Alive and continue to to be very useful if you switch to a linux distro yeah the uh i think that some of my best laptops have been like 20 year old like thinkpads and if you find a thinkpad It's like how could i bring this back and whether it be like lubuntu or puppy or like linux like or alma just these little linuxes that are just fast as heck bring bring a 20 year old laptop Back to life like you take you take one of those you throw an ssd in it double the ram with whatever sticks you have around and you're out there with even eight gigs zipping around exactly And non-technical parent can use that very easily ([Time 0:26:48](https://share.snipd.com/snip/ac25e29f-8ef3-4da5-8183-68f20a93dec8)) - **Restoring Support for Older Chipsets** - AlmaLinux restored support for older V2 chipsets dropped by Red Hat 10. - This decision helped prevent costly hardware replacements for many users. Transcript: Benny Vasquez They're only V2. Scott Hanselman Oh, yeah. Yeah. Benny Vasquez That means that there are tons of pieces of hardware that people said, well, I thought this was going to last me for a while. And they're getting really stuck. So they came to us and said, hey, will you release a version of Amalyn X that supports V2? And we, you know, we went through the engineering steering committee and said what do we think how much work is this can we actually make this happen and we did because it was very valuable And we we had a bunch of people especially at science labs that said we're gonna you know we're gonna have a huge amount of overhead in hardware if we have to replace all of this, just because Red Hat has decided we'll probably just leave the Red Hat universe and we'll go someplace else. Yeah. And so we didn't want to see that. Like we like the Enterprise Linux ecosystem. So we helped with that. And then the next thing we had to do was build the extra packages for Enterprise Linux repo. There's tons of packages that they, it's a Fedora project, and there's tons of packages that they maintain. And because they follow Red Hat, they weren't going to be supported for V2 either. So we worked with the Apple folks and said, hey, can we, do you mind if we build these so that they'll work with our V2 version? And they said, you know, they had some feedback and said, let's do it in this way. Let's make sure it works this way. Let's make sure the bugs get reported in the right place, like that kind of stuff. So now we're building all of Apple for this V2 version. And we've gotten way more adoption than I honestly expected. There was a lot of people that, you know, went to install an operating system and couldn't and said, well, what does this error mean? And then figured it out and came to Alma and said, OK, this works now. ([Time 0:28:19](https://share.snipd.com/snip/c57f425d-aa4f-4499-8a1d-cf599c3b4c78))