# Episode 103: GrimoireLab at FreeBSD  ## Episode metadata - Episode title: Episode 103: GrimoireLab at FreeBSD - Show: CHAOSScast - Owner / Host: CHAOSS Project - Guests: [Miguel Ángel Fernández](https://share.snipd.com/person/e01b337f-642e-4a67-af54-09176770779d), [Moin Rahman](https://share.snipd.com/person/650692a9-46e0-4330-a374-b990df958250), [Ed Maste](https://share.snipd.com/person/dfabf0cb-23c9-46ac-8ff5-d20858f83e11) - Episode publish date: 2025-02-06 - Episode AI description: Miguel Ángel Fernández, a data scientist and GrimoireLab contributor, joins Ed Maste, a long-time FreeBSD committer, and Moin Rahman, a seasoned contributor focused on documentation. They dive into how FreeBSD adopted GrimoireLab to manage its extensive bug backlog and optimize project insights. Discussions reveal the integration challenges and how metrics enhancements have improved bug management and community engagement. The guests share personal growth experiences within the open-source community, showcasing the blend of technology and passion. - Duration: 36:01 - Episode URL: [Open in Snipd](https://share.snipd.com/episode/85599b7b-dd2c-48ef-8aa0-2025efc8400a) - Show URL: [Open in Snipd](https://share.snipd.com/show/53d35173-57a3-4157-8040-c3e76d6df197) - Export date: 2026-02-17T23:29:48 ## Snips ### [FreeBSD's Huge Bug Backlog](https://share.snipd.com/snip/424cb2f2-59e2-4445-805f-f7e0917b4c62) 🎧 07:38 - 09:27 (01:48) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/b246194e-4adb-4318-8c3d-062fe4404e15" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - FreeBSD had over 7,000 open bugs dating back decades, making backlog management overwhelming. - They needed a structured way to assess bug relevancy and take actionable steps for progress. #### 💬 Quote > so when we started this process in the base system... I think we had over 7,000 open bugs. And these date some of them back decades in some cases. > — Ed Maste Ed Maste on FreeBSD's bug backlog challenge #### 📚 Transcript **Alice Sowerby:** Certainly. **Ed Maste:** I think one of the challenges that we started that prompted us to look for tooling to help understand our bug backlog is just that we have in FreeBSD quite a large number of open bugs and we were struggling with understanding information about the massive quantity of issues that we had open. So in FreeBSD, our bug tracking is done in Bugzilla. And we have basically six high level categories of bugs. And we've got three main repositories in FreeBSD. So we've got the base system, the documentation tree, and the ports and packages collection. And then there's other categories for community issues or services that the project runs, things like that. But the three main repositories are really where the majority of the bugs and the understanding that we wanted to gain exist. And when we started this process in the base system, which is the kernel in the userland and the toolchain, that sort of integrated whole that we release that I mentioned earlier, I think we had over 7,000 open bugs. And these date some of them back decades in some cases. And really, it's just sort of an overwhelming quantity of data to try and understand and try to find ways that we can say in that list of massive number of bugs, clearly there are some that are overcome by events that are against releases and reporting bugs in functionality that doesn't even exist in PreVSD anymore, things like that. We want to be able to go through and understand what the situation is. We don't want to just close bugs that are older than 30 days or six months or whatever it is. We really want to be able to understand what the reality of the situation is and be able to do something actionable and significant and make good progress in actually fixing the issues that we have. --- ### [Start Metrics With Clear Questions](https://share.snipd.com/snip/0dd4ebc0-2cce-46d4-a2fc-10daa9f29a23) 🎧 11:19 - 15:06 (03:47) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/f5962fa2-d36b-4eaf-a9ff-8d8b5952343c" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Start with clear questions before diving into project metrics to avoid getting lost in data. - Organize information by product and key bug data fields to tailor dashboards effectively. #### 💬 Quote > compared to other open source projects and foundations, you were really advanced in the process of, okay, let's raise the questions we want to answer. > — Miguel Ángel Fernández Miguel Ángel Fernández on creating useful metrics and dashboards #### 📚 Transcript **Alice Sowerby:** And so we ended up with five, I think it is, different dashboards. And I wonder if you might be able to say a little bit about how you took the questions and turned them into dashboards and what kinds of things dashboards had on them. **Miguel Ángel Fernández:** Okay. So first of all, I have to say it was super easy to work with you because compared to other open source projects and foundations, you really advanced in the process of, okay, let's raise the questions we want to answer. This may be simple, but it's not because many people start having a look at the metrics without raising up the questions. And I think this is the first step. So the first step is not having a look at the metrics because you will end up lost in a lake data, you know, without knowing what to look at. And in your case, it was super easy because you already had very clear set of questions that you would like to answer, right? So the process was pretty straightforward in that sense. So the approach was, okay, so Baxilla is a data source that is not that common, at least in other foundations or in other open source projects. Maybe they are using other platforms such as GitHub or GitLab to fill their issues and so on. So first it was, Vaxilla is a data source supported by the Grimoire platform, but it's true that we needed to have a look at the structure of the data. And the challenging part was that indeed there was a huge history of data, I think it's 30 years old. So it was awesome to see this very big history in front of our eyes, right? Of bugs and so on. So the approach we took is, okay, taking into account the product information and the classification that Ed was mentioning before. It was, okay, how we could organize the information in a way that is useful. So we tried to research on which fields could help us organize this information better, and then select a set of metrics that help us understand more about the problem. backlog, it means that we need to see the status of things, how many bugs are open, how many bugs are closed, which is something that is not easy sometimes because the definition of close varies from one product to another maybe, but also other bugs were solved in a way that maybe they are not solved in that way anymore. Or someone posts a comment in an older bug because the bug gets referenced and so on. So there are some challenges to take into account. The first thing is to have a look at the historical data, have a look at the structure and see which fields can help us understand what's going on. The product was the base information to fill that. And also we try to understand if the keywords, the labels of the bugs were useful. In this case, we discovered that many of the bugs were not labeled, so we needed to look for different approaches. So the dashboards, we went from a general view to a more specific view. So first we have a dashboard offering an overview of the bugs submitted for all the products in a selected period. From there, we go to another dashboard focused on efficiency in closing bugs and it's showing if the community is able to digest the bugs in a specific period. And then we have two more dashboards with more specific viewing to individual bugs. One to looking at the bugs needing more attention and focusing on those with higher values on the time to first attention and time since the last comment. --- ### [Backlog Management Index Insight](https://share.snipd.com/snip/14bdf571-81fb-48b4-acb9-21eefdd46168) 🎧 15:32 - 17:37 (02:05) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/f751096f-bebb-4daa-a657-c4c1ad348f48" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - The Backlog Management Index (BMI) shows whether bug closing rate surpasses incoming bugs. - This metric gives a clearer picture of backlog progress than total open bug counts alone. #### 💬 Quote > when the line's above 100%, we are staying ahead of our incoming bug rate. > — Ed Maste Ed Maste explaining the usefulness of the BMI metric #### 📚 Transcript **Alice Sowerby:** And I know that you've been finding that very helpful, Ed, so maybe you could say some more about that. Yeah, **Ed Maste:** certainly. I mean, I think when we started this process, I was aware of Grimoor and Baturgia, having met some folks at a conference in the past, but I didn't really know what sort of capabilities existed. And so it was an iterative process for us to come up with the different dashboards that we've presented. And I find the bug backlog dashboard and the graphs that are produced there, I think they are very useful in answering a question, is the project that we've undertaken within the foundation to try to reduce our backlog and deal with sort of accumulated technical debt? Is that being successful or not? And especially being able to communicate that outwards. We've had a high level graph that is just the total number of open bugs over time that Bugzilla produces. And it's really hard to draw any conclusion from that because it's over all time and it's not at all detailed enough to really understand what it's showing. And I think the bug backlog is easy to understand. It's just showing whether the count of open bugs is going up or down. And so people can easily look at that and see, you know, that the line is trending down and that's good. Open bugs going down is, it looks good. But I think the BMI graph, it takes maybe a moment to explain what it's actually graphing. And it's just showing the ratio of closed issues to new issues over time. Although it takes a moment to kind of explain, it presents data in a much more easy to grasp way. Once you understand what it's showing, it's very easy to see when the line's above 100%, we are staying ahead of our incoming bug rate. And so I'm looking at the now and I see it's actually has quite a steep rise towards the right of the graph, which says that we are increasing our velocity of getting ahead of the outstanding bug count. right now, the moving averages, it shows it at 200%. So basically, we are closing two bugs for every bug that's submitted over the last little while. --- ### [Bug Busting With GrimoireLab](https://share.snipd.com/snip/9ec874dc-ad23-4a2e-ad2c-a482eec8f463) 🎧 18:17 - 19:12 (00:54) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/7e7f9358-9237-4388-a6ce-2f0d038303ff" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Bug busting sessions used GrimoireLab dashboards to quickly find old bugs irrelevant to the current project version. - This helped close several outdated bugs efficiently during community events. #### 💬 Quote > Mark was actually able to quickly find out like which bugs were lurking around for a long time... We closed a couple of bugs. > — Moin Rahman Moin Rahman on using GrimoireLab for bug busting #### 📚 Transcript **Alice Sowerby:** And if so, maybe you'd like to speak about that a bit. So **Moin Rahman:** it was actually interesting bug busting sessions. So what happened was we couldn't actually go through all the oldest stories, but through the dashboard, Mark was actually able to quickly find out like which bugs were lurking around for a long time. And if those were actually even effective for today's date, because as the evolutes from the earlier days to the latest version, there may be some bugs which actually do not make any sense in today's date. So should we actually keep those open and lurking around in the bugzilla? So this was really actually helpful in sense. And we closed a couple of bugs. We were just there open, but we didn't make any sense in today's standing of the FreeBSD project itself or with the latest version of the FreeBSD project. That --- ### [Deploy GrimoireLab Natively on FreeBSD](https://share.snipd.com/snip/fd33c33c-e4e5-49bc-b84b-96ca9eb3701f) 🎧 20:10 - 23:10 (02:59) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/c5801420-cb2a-4b7f-898e-b706eb846434" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Run GrimoireLab natively on FreeBSD for security, transparency, and stability. - Avoid relying on Linux compatibility layers or Docker to better integrate with FreeBSD's ecosystem. #### 💬 Quote > running Grimoire Lab natively on FreeBSD actually means that I have full control... without relying on a black box container image. > — Moin Rahman Moin Rahman on deploying GrimoireLab on FreeBSD #### 📚 Transcript **Alice Sowerby:** So how was it for you doing that work? Since **Moin Rahman:** we are running Grimaldab for FreeBSD, the first priority for us was to actually ensure that it runs natively on FreeBSD without requiring any external Linux installations, compatibility layers, or some sort of Linux-based infrastructure. FreeBSD has its own ecosystem and its own package management system and its own security model. So forcing a Linux base that would have actually defeated the main purpose. The biggest challenge was adopting the dependencies and ensuring compatibility, Grimoire Lab is actually not one piece of software. It actually involves multiple components, and each one actually integrates with itself in a different manner. So you have to have a good understanding of which part actually does what. And most of the official Grimaule Lab documentation actually assumes Docker, but Docker isn't natively supported on FreeBSD. While it can be run inside of Linux compatibility layer that introduces actually unnecessary complexity and security risks for us. I actually installed and configured everything natively using FreeBSD, PKG, and port system. One of the key reasons I actually avoid Docker is security. Many people follow setup guides without understanding what's actually running behind the scenes. Sometimes they actually pull on verified images or sometimes expose services without authentication or run containers in privileged mode. So running Dreamer Lab natively on PPSC actually means that I have full control over the configurations, dependencies, the access control, without actually relying on a black box container image. And we also don't use any traditional configuration management tools like Ansible or Puppet inside our cluster. Instead, we actually have built our own Fosix shell compliant configuration system tailored only for FreeBSD. We leverage multiple tools like Jails, Cthrc, and ZFS, which keeps things very much lightweight, predictable, and secure. For everyone, actually, who is looking to running Grimoire apps, I would advise that you run it in with native compatibility by avoiding relying on the Linux-based workaround. Try to use the previous strengths like ZFS for snapshots and jails for isolation of the services. We're running three jails, and one jails is running Elastic. In the other two, we have a development instance for the dashboards. Also, don't please blindly follow installation guides. Understand what each command does and what is happening behind the scene. As a result, you will have actually a much more secure, transparent, and stable deployment that actually fully integrates into FreeBSD without external Linux dependency. But FreeBSD is actually a fantastic platform for running production services. You just need to step away from the Linux mindset and embrace what it offers, I think. --- ### [New Bug Classification Metric](https://share.snipd.com/snip/7a801916-d0a6-4e58-a71c-fd64a64b941b) 🎧 24:23 - 27:54 (03:31) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/af39299e-d94d-4024-92db-d2aaa34ebb42" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Classifying bugs as unattended or abandoned helps pinpoint bugs needing attention and guides cleanup. - This classification enhanced GrimoireLab metrics and can apply to other issue tracking systems. #### 💬 Quote > we decided to classify the bugs according to this time to first attention... we created some new categories: unattended tickets and abandoned tickets. > — Miguel Ángel Fernández Miguel Ángel Fernández on new bug classification in GrimoireLab #### 📚 Transcript **Alice Sowerby:** Could you say a little bit more about that, please? **Miguel Ángel Fernández:** Yeah, sure. First of all, let me say that as a GrimoireLab contributor, it's an awesome milestone and an honor to have GreenWard Lab running on PrePSD. This is awesome. So I'm really excited to see that documentation coming in the future. I think it's a huge milestone for the project. So kudos to you for the great work. About the metrics, yes, you're right. I think as you were mentioning before, and I was also mentioning this set of questions we discussed, also Ed mentioned the BMI metric and how this metric helps to understand if the community is able to digest the issues and the bugs that they are being created and closing and solving, right? So we needed to answer the question of, okay, how many bugs are there that are not properly attended? Or maybe there are bugs that are being abandoned. This was one of the main questions that raised up during our first conversation. And I think this question really changed our perspective on some metrics that we apply almost weekly in other communities and in other projects or efficiency metrics, like the first time to first attention. The first time to first attention in a ticket, in this case for a bug, is the time between someone creating a ticket and another person posting a comment. And this comment needs to come from a person different from the author. And then we have the time to first attention. So this is a common metric we use to understand if the tickets are properly attended or not. But in this case, the question was slightly different. So we decided to classify the bugs according to this time to first attention. And we created some new categories. One is unattended tickets and another one is abandoned tickets. Unattended tickets means that someone opened a ticket and no one posted a comment rather than the author who opened the bag, right? And abandoned tickets are those who were left open and they don't have any since the last three months or six months. So we tag them as abandoned. Then we have the tickets and the bags classified in four categories. Those that are open and from those that are open, we have those that are attended and also recently commented, but also we have unattended tickets. So no comments from any user different from the author and also abandoned tickets. So last comment was submitted more than three months ago. And this classification really changed the game when visualizing this information, right? And I think it helps to quickly see where the bottlenecks are, or maybe applying this housekeeping rules you were mentioning before. Okay, maybe these bugs are not making any sense anymore. And the good thing is this got also to GrimorLab. We missed this feature in the Baccilla backend in this case. So we applied that in grammar lab. This one of the features. And also the interesting part is that this metric definition we created together, it can be applied to other sources too. And I think it's really useful to apply this concept in other sources and in other items. I think it's a good way to visualize what's going on and which items we need to focus on. Thanks, --- ### [Bimodal Bug Response Insight](https://share.snipd.com/snip/78bf16b1-f829-4c7f-90ed-94474abadb51) 🎧 28:06 - 30:22 (02:16) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/ca6ced61-4295-46b6-8cf5-368e1df52d07" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Bug response times show a bimodal pattern: some bugs get quick attention while others never get addressed. - Early identification allows better communication with reporters and management of unrealistic bug expectations. #### 💬 Quote > We have the quantities for bugs that receive a response in less than a day equal to the ones that are unattended, that don't get a response at all. > — Ed Maste Ed Maste on insights from bug response time data #### 📚 Transcript **Ed Maste:** I think this data that we're getting from these two categories, I would say in general, still working on holistically applying what we're learning from the metrics that we're gathering to changing the way that we approach things. But what's really interesting is just when I look at the graph of open bugs by the time to first detention, I see that it's sort of a bimodal distribution. We have the quantities for bugs that receive a response in less than a day is equal to the ones that are unattended, that don't get a response at all. And I mean, less than a week, less than a month are relatively low. So what it's telling me is we are very good at responding quickly if there is some action we can take. We have a number of bugs that will come in and it's not immediately obvious what we should do with them. And when we get into that category, it's clear that very early on we can tell which path this bug is gonna take. It either gets a response and things happen, or we can tell that within a week or so, if it hasn't had a response, it's almost certainly never gonna get looked at. And so that sort of insight is something that we can use to factor in how we deal with it and what information we can either ask the bug submitter to try and make progress. Or if we know that the bug is not going to get a response up front, we know from the data that this bug is, if it hasn't been dealt with in a week or, you know, whatever the threshold is we know it's not going to get a response let's figure out what action we can take up front it's much better to tell the user either we're not going to be able to address this or you know we can't reproduce it whatever it is that bug needs to make progress it's much better if we can provide that feedback and that insight early on rather than having the bug sit there for a year or two years. then someone much later on comes along and says, is this still reproducible on a currently supported version? And there's no response. And then it's closed six months later with feedback timeout. That's not a great experience for the person who submitted the bug. It's not a great experience for the developer who triages it a year rather later. It's not a great experience for anyone. And so being able to figure out this is a significant issue that we have and how can we address it in a holistic way, I think, is the kind insight that we're very happy to get from this sort of data. --- Created with [Snipd](https://www.snipd.com) | Highlight & Take Notes from Podcasts