Augur8-Knot WG March 17, 2026 - CourtneyR.dev's 2nd Brain

Augur/8-Knot WG March 17, 2026 ![rw-book-cover](https://img.youtube.com/vi/hkejHxiCb5s/maxresdefault.jpg) 🔗 **Source:** [Augur/8-Knot WG March 17, 2026](https://www.youtube.com/watch?v=hkejHxiCb5s) > [!note]+ Document Notes > tags: #augur #8knot #database #materialized-views #performance #hourly-processing #contributor-data #schema-normalization > author: Augur/8-Knot WG > source: Zoom Meeting > reference: https://www.youtube.com/watch?v=hkejHxiCb5s > date created: 2025-01-02 10:30 > > # 🏷️ Title: Augur/8-Knot WG March 17, 2026 - Memory Fixes and Database Schema Discussion >  ## 🧰 Action Inbox - [H1] Fix hourly contributor processing memory leak — production memory issues #performance - [H2] Implement materialized views in Alembic migration system — 8knot integration needs #database - [H3] Develop automated data cleanup for collection bugs — maintainer feedback #data-quality ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Augur GitHub Repository](https://github.com/chaoss/augur) - [8knot Project](https://github.com/augurlabs/8knot) ## 📝 Summary - Augur production instance experiencing memory leaks from hourly contributor processing task - Team discussing integration challenges between 8knot's materialized views and Augur's migration system - Database schema normalization being considered to improve performance and reduce learning curve - Contributors working on automated fixes for collection bugs and better telemetry - Clone activity showing unusual spikes potentially from AI scrapers ## 📖 Main Content The meeting focused on critical performance issues affecting Augur's production instance and long-term architectural improvements. The primary concern was an hourly task processing contributor data that was causing memory exhaustion by attempting to process 50,000+ contributors every hour without completion, leading to multiple copies accumulating in memory. A proposed fix limits processing to 500 contributors per hour, which would handle large instances over approximately one week. The team also discussed challenges integrating 8knot's materialized views back into Augur's main database migration system. These views were previously disabled due to performance issues but are being re-enabled with improvements. The current migration system using Alembic doesn't handle materialized views well, requiring research into proper implementation approaches. Longer-term discussions centered on database schema normalization, particularly around file handling across different contexts (pull requests, commits, repo analysis). The current approach stores duplicate file path information across multiple tables, with the repo_labor table being particularly large. Proposed solutions involve creating centralized file tables with foreign key relationships, though this introduces complexity around historical file tracking during refactoring events. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/hkejHxiCb5s?t=0) — [H1] Hourly Processing Memory Issue - Key idea: Production instance running out of memory from contributor processing task - Evidence: "It's basically pulling in like 50,000 contributors every hour, not getting through all of them" #### ⏱️ [02:45](https://youtu.be/hkejHxiCb5s?t=165) — [H2] Memory Leak Root Cause - Key idea: Multiple copies of large datasets accumulating hourly without cleanup - Evidence: "Memory usage would step down every hour basically consistently" #### ⏱️ [04:30](https://youtu.be/hkejHxiCb5s?t=270) — [H2] Proposed 500 Limit Fix - Key idea: Limiting to 500 contributors per hour prevents memory overload - Evidence: "500 an hour is 12,000 a day. So like even for like 50,000 plus contributors" #### ⏱️ [08:15](https://youtu.be/hkejHxiCb5s?t=495) — [H1] Automatic Fix Philosophy - Key idea: If Augur breaks data automatically, it should fix it automatically - Evidence: "If Augur broke it automatically, Augger should fix it automatically" #### ⏱️ [12:00](https://youtu.be/hkejHxiCb5s?t=720) — [H3] Bandit Security Scanning - Key idea: Adding security scanning to catch sensitive information in logs - Evidence: "Remove the logging of like sensitive information in the log like censoring API keys" #### ⏱️ [18:30](https://youtu.be/hkejHxiCb5s?t=1110) — [H2] Database Connection Labeling - Key idea: Adding application identifiers to database connections for debugging - Evidence: "Everything had no name except for the connections that DBaver itself was making" #### ⏱️ [22:00](https://youtu.be/hkejHxiCb5s?t=1320) — [H2] GitHub Performance Issues - Key idea: GitHub experiencing widespread performance problems affecting builds - Evidence: "GitHub's like status page has been all the rainbow colors" #### ⏱️ [25:45](https://youtu.be/hkejHxiCb5s?t=1545) — [H3] Unusual Clone Activity - Key idea: Repository seeing hundreds of daily clones, possibly from AI scrapers - Evidence: "We've just been getting like hundreds of clones a day" #### ⏱️ [32:15](https://youtu.be/hkejHxiCb5s?t=1935) — [H2] 8knot Heat Map Reintegration - Key idea: Re-enabling file touch heat maps to identify unmaintained code areas - Evidence: "Files have been touched at what times sort of in a grid view" #### ⏱️ [35:00](https://youtu.be/hkejHxiCb5s?t=2100) — [H1] Materialized Views Migration Challenge - Key idea: Current Alembic setup cannot handle materialized view migrations - Evidence: "Our current Olympic like database migration config is able to do uh materialized views" #### ⏱️ [42:30](https://youtu.be/hkejHxiCb5s?t=2550) — [H1] Repo Labor Table Size Problem - Key idea: Massive repo_labor table causing performance issues and disk usage - Evidence: "That table is a huge percentage of the disk utilization" #### ⏱️ [48:15](https://youtu.be/hkejHxiCb5s?t=2895) — [H2] File Normalization Challenges - Key idea: Files exist differently across pull requests, commits, and analysis contexts - Evidence: "Files are added and removed routinely. So there isn't actually in augur anything that is the canonical here are the files" #### ⏱️ [55:00](https://youtu.be/hkejHxiCb5s?t=3300) — [H3] Materialized Views as API Layer - Key idea: Using materialized views to provide stable API while refactoring underlying schema - Evidence: "Use the materialized views as sort of like an outer castle wall" #### ⏱️ [58:30](https://youtu.be/hkejHxiCb5s?t=3510) — [H3] Long-term Schema Normalization Goals - Key idea: Moving toward normalized database structure for better maintainability - Evidence: "Think about okay at least make sure that all the small changes we're making are at least taking steps either towards or not away from like these larger goals" ## 📚 References - [Alembic Documentation](https://alembic.sqlalchemy.org/) - [PostgreSQL Materialized Views](https://www.postgresql.org/docs/current/sql-creatematerializedview.html) - [SQLAlchemy ORM](https://www.sqlalchemy.org/) ## 🗣️ Paraphrase - Augur's production system is struggling with memory management due to an hourly task that processes too many contributor records simultaneously, requiring a rate-limiting fix to prevent system crashes - The integration between 8knot's analytical features and Augur's core database presents migration challenges that need systematic solutions for materialized view management - Long-term database architecture improvements focus on normalizing file storage patterns while maintaining historical data integrity across different analysis contexts ## 📰 Newsletter The Augur development team tackled critical performance issues affecting production deployments this week. A memory leak in contributor data processing has been identified and addressed with rate-limiting improvements, ensuring stable operation for large-scale instances. Looking ahead, the team is working on better integration between Augur and 8knot analytics, including improved database migration tools and schema normalization. These improvements will make Augur more maintainable and reduce the learning curve for new contributors and data scientists. For developers running Augur instances, keep an eye out for the upcoming release with memory management fixes and consider the new materialized view features being developed for enhanced analytics capabilities. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer experience: Document materialized view setup process for 8knot integration - Education: Create guidelines for database schema changes and migration best practices #### OSPO - Software Inventory: Monitor clone activity patterns to distinguish legitimate usage from scraping - Security Management: Implement bandit security scanning for sensitive data detection #### Community - Maintainers: Establish automated data cleanup processes for collection errors - Contributors: Research Alembic materialized view support and implementation patterns ## 🧪 Hypotheses - Rate-limiting contributor processing to 500 records per hour will prevent memory exhaustion while maintaining data completeness over time - Materializing database views can provide stable APIs while allowing underlying schema refactoring without breaking downstream applications - Normalizing file storage across different analysis contexts will reduce database size and improve query performance - AI scrapers are responsible for unusual repository clone patterns rather than legitimate user growth ### 📊 Methodology - Memory usage monitoring during contributor processing with rate limits - Performance benchmarking of materialized view refresh operations - Database size analysis before and after schema normalization - Clone pattern analysis correlation with merge activity ### ❓ Questions - How can Alembic be extended to properly handle materialized view migrations in production environments? - What is the optimal balance between data normalization and query performance for large-scale repositories? - How can automated data cleanup be implemented without risking data integrity? ### 🔮 Future Research - Investigation of semantic layer tools for data engineering improvements - Analysis of file history tracking during major refactoring events - Development of telemetry systems for better instance monitoring ### 💡 Implications - Practice: Production instances need careful memory management and rate limiting for background tasks - Policy: Automated fixes should be prioritized over manual intervention for data consistency issues - Security: Regular scanning for sensitive information in logs and code is essential - Education: Database migration patterns need documentation for complex features like materialized views ### 🧠 Learning Styles Assessment Reading level estimate: 14th Grade (technical discussion with domain-specific terminology) #### 👀 Visual learners - Present: Database schema discussions, table relationships mentioned - Improve: Add entity-relationship diagrams for proposed schema changes #### 🔊 Aural learners - Present: Full meeting recording available with detailed technical discussion - Improve: Provide chapter markers for different topic segments #### 🗣️ Verbal learners - Present: Technical terms explained in context, analogies used for complex concepts - Improve: Add glossary for database and performance terminology #### 🧩 Kinesthetic learners - Present: Code examples and practical implementation discussed - Improve: Provide hands-on migration examples and testing procedures #### 🧠 Logical learners - Present: Clear cause-and-effect relationships for memory issues and solutions - Improve: Add decision trees for schema design choices #### 🧑‍🤝‍🧑 Social learners - Present: Collaborative discussion format with multiple perspectives - Improve: Add structured feedback mechanisms for proposed changes #### 🧘‍♀️ Solitary learners - Present: Detailed technical discussion allows for individual analysis - Improve: Provide self-assessment questions for understanding #### 🌿 Naturalistic learners - Present: System health and sustainability considerations discussed - Improve: Connect performance optimization to resource conservation ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Meeting transcript provides full context - Improve: Add structured headings and topic markers #### Keyboard navigation - Present: Standard video controls available - Improve: Add keyboard shortcuts for topic navigation #### Contrast and readability - Present: Text transcript meets contrast requirements - Improve: Provide high-contrast video player options #### Text resizing and reflow - Present: Transcript text scales appropriately - Improve: Ensure technical diagrams remain readable when enlarged #### Media and captions - Present: Full transcript available - Improve: Add synchronized captions to video #### Forms and interactions - Present: Basic video controls accessible - Improve: Add accessible feedback forms for meeting content ### 🌍 DEIB Assessment #### Cultural breadth - Present: Technical discussion accessible across cultural contexts - Improve: Consider time zone impacts for global participation #### Inclusive language - Present: Technical terminology used appropriately - Improve: Define acronyms and technical terms consistently #### Neurodiversity support - Present: Structured agenda and clear topic transitions - Improve: Provide advance materials and clear time expectations #### Representation - Present: Multiple team members contributing to discussion - Improve: Ensure balanced speaking time and participation opportunities #### Psychological safety - Present: Respectful technical disagreement and collaborative problem-solving - Improve: Establish clear guidelines for technical critique and feedback ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Memory leak analysis follows clear cause-effect logic - Inductive strength: Clone pattern analysis based on observed data trends - Causal reasoning: Direct correlation established between hourly tasks and memory issues - Analogical strength: Database normalization compared to established patterns #### Fallacies - False dilemma: Assumption that materialized views must be either manual or fully automated, missing hybrid approaches - Appeal to authority: Relying on "John's comments" without fully exploring alternative perspectives #### Tone of voice - Overall tone: Collaborative and problem-solving focused with appropriate technical detail and mutual respect ### 🛡️ DARVO Analysis - Deny: No denial patterns observed in technical discussion - Attack: No attacks on individuals or groups - Reverse victim/offender: No victim/offender dynamics present - Emotional cues and rhetoric: Professional technical discussion without manipulation ## 🎯 Conclusion The meeting successfully identified and addressed critical performance issues affecting Augur's production deployment while laying groundwork for significant architectural improvements. The rate-limiting solution for contributor processing provides immediate relief, while the longer-term schema normalization and materialized view integration efforts will enhance system maintainability and performance. The collaborative approach to problem-solving demonstrates effective technical leadership and community coordination. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad Purpose: Stage CEE units that become Literature notes. Split level: H3 ### [H1] Memory management requires rate limiting for background tasks > [!quote] [Augur/8-Knot WG March 17, 2026](https://youtu.be/hkejHxiCb5s) by Augur/8-Knot WG via Zoom Meeting. Published 2026-03-17. Accessed 2025-01-02. > "It's basically pulling in like 50,000 contributors every hour, not getting through all of them" Demonstrates how unbounded background processing can exhaust system resources in production environments. [view highlight](https://youtu.be/hkejHxiCb5s?t=165#H1) #performance #memory-management ID:rw_001 ### [H2] Materialized views provide stable APIs during schema migration > [!quote] [Augur/8-Knot WG March 17, 2026](https://youtu.be/hkejHxiCb5s) > "Use the materialized views as sort of like an outer castle wall" Strategic approach to maintaining API compatibility while refactoring underlying database structure. [ --- ## Full Transcript [00:00] All [00:01] right, welcome to the Auger meeting for [00:04] March 17th, St. Patrick's Day, [00:07] 26th. [00:10] Um, so this is the one that you the [00:13] limit hourly processing for. [00:15] >> Yeah, that's so that's something I [00:16] noticed. So I I noticed our prod over [00:19] the weekend [00:21] slash my week off last week um wasn't [00:25] behaving great. I think the first one [00:27] first issue was the just the large fi [00:29] large large file downloads from [00:32] scorecard which is a different issue but [00:34] then it also had an issue where it was [00:36] just running out of memory and I was [00:38] like hang on I thought we fixed all the [00:39] memory problems and I looked into it and [00:41] it turns out that yeah it's basically I [00:43] noticed that the after I restarted it [00:46] the graph in graphana of the memory [00:48] usage would step down every hour [00:50] basically consistently and that was like [00:52] huh we just enabled an hourly task I [00:54] wonder if that's the task And so it I it [00:57] did seem like it was exactly the issue [00:59] we discussed in the maintainers call [01:01] last week, which is for large instances [01:03] with tens of thousands of contributors, [01:06] many of them may or may not have the two [01:10] date fields that this job is looking for [01:13] to decide like, hey, has this been [01:15] filled in with profile data? [01:17] >> So it was basically pulling in like [01:19] 50,000 contributors every hour, not [01:21] getting through all of them, and just [01:23] like it's going to continue. And then [01:25] the ne the next time the next hour a new [01:26] one starts. And so it was just making [01:28] copies and copies and copies of [01:31] >> those 50,000. I think I think this task [01:33] has two copies of the 50k of all the [01:35] records in memory. [01:36] >> Yeah. [01:37] >> And it was starting a new one every hour [01:40] and not finishing. So it was just like [01:41] eating it had eaten like huge amounts of [01:44] memory. Um this one's still being [01:46] tested. [01:47] >> So okay, [01:48] >> I'm still running it in our prod. Um, [01:50] it's I don't see why it would go wrong. [01:52] If you feel like just yolo merging it, [01:55] if you wait a little bit, I can confirm. [01:57] >> The only thing I see difference is, and [02:00] tell me if I'm getting this wrong, the [02:01] only real difference I see [02:04] is that you've made the syntax a little [02:07] bit neater and you've put a limit of 500 [02:10] items per query on it. But [02:13] >> yeah, so that that's basically 500 per [02:15] hour effectively. [02:16] >> Yeah. Which just seems like an [02:18] appropriate [02:19] Yeah, that I so as I noted in the in the [02:22] PR description, that might be a little [02:24] bit low because I I think I initial when [02:26] I was thinking of it, I was thinking of [02:28] the 5,000 [02:30] >> limit on REST API calls and I was like, [02:32] okay, 500's reasonable relative to that, [02:34] but I wasn't I was this was me like at 6 [02:37] p.m. last night, tired brain. Um, [02:40] >> wow. [02:41] >> I was thinking that big instances are [02:43] going to have multiple keys and so we a [02:45] thousand might be a little bit more [02:46] reasonable, but this will do the 500 an [02:48] hour is 12,000 a day. [02:50] >> So like even for like 50 50,000 plus [02:53] contributors that haven't had profile [02:55] data, this will like over a week this [02:57] should this should get to full [02:59] population. [03:00] >> Yeah, it's going to tick through [03:01] everything. And yeah, I mean I this is [03:05] another one where I think it's pretty [03:07] straightforward what you're doing here. [03:09] And if it didn't fail on execution, the [03:11] logic is [03:13] simple and straightforward. You really [03:16] haven't changed it other than [03:18] there's there's really no there's no [03:20] change in logic other than a a better [03:22] syntax [03:24] really. You just kind of laid it out [03:25] better and this limit. That's it, right? [03:27] >> Yep. [03:29] Am [03:32] I seeing that right? [03:33] >> Yeah, that's that's the two commits that [03:35] >> Yeah, I think the uh the third commit I [03:38] think there might have been a third [03:38] commit for something, but yeah, I think [03:39] it was all just basically the formatting [03:41] and the [03:44] Yeah. [03:47] Um, [03:54] and then I think one thing if we're [03:55] looking at things that we need for this [03:57] coming release, um, I think it's also [04:00] given that given John's comments in the [04:02] the maintainers call last week, um, like [04:05] since we had an error with how we were [04:09] doing various parts of facade collection [04:11] and contributor resolution that was [04:13] causing data to be bad, like because it [04:15] was kind of an automatic error in in [04:17] augur. It should kind of be an automatic [04:19] solution ideally. I don't know. I think [04:23] Cali Callie's also been kind of [04:24] brainstorming on things. It seems like [04:26] you maybe had some items later in the [04:27] agenda, but um I think that's going to [04:31] be something that's important to include [04:33] in [04:35] >> which which John said which piece that [04:37] John said. [04:39] just just like he just I think he was [04:42] just promoting the concept of [04:44] >> if Augur broke it automatically, Augger [04:46] should fix it automatically, [04:48] >> right? Like I don't like it it [04:50] >> as much as like [04:51] >> I agree with that philosophy. [04:52] >> Previous issues we've had like we've [04:54] just been like hey like for the [04:55] duplicate whatever duplicate repos issue [04:58] >> it's just like hey Cali go run this [05:00] giant script. Um, and it's like that [05:02] that I mean I guess if we make the [05:04] script nice and easy to use and put it [05:05] in the release notes, like maybe we can [05:07] rely on people to run it, but that's [05:09] just another point of failure of like, [05:10] oh crap, someone didn't read all the [05:13] release notes from they upgraded from an [05:14] old version and didn't see it [05:16] >> and now they're on a new instance that [05:18] has duplicate or like something's broken [05:20] because [05:21] >> that happen. Whereas if we if we have it [05:24] sort of somehow automatic [05:27] >> then we we can kind of be more sure that [05:29] everyone's in a a known consistent [05:30] state. [05:33] >> I think that makes sense. Um so on the [05:36] contributor fixing I've been testing a [05:39] set of scripts that would need to be [05:41] then rolled into augur. Um [05:45] not quite done testing them yet. [05:49] And so I think that based on what you're [05:52] saying, I think that's I think that's [05:53] what you're saying. [05:56] >> Yeah, I think I think we'd eventually [05:57] want to roll them into Auger. I'd want [05:58] to make sure that they're like obviously [06:00] robust and good good quality. Um [06:03] >> I think Callie's also been sharing some [06:04] notes with me like just we've got [06:06] obviously we've got our own internal [06:07] task tracking of what do we do about our [06:10] instance. So I think I think um once [06:12] Callie is kind of back from [06:15] whatever she's working on um she'll [06:17] probably like have some idea like there. [06:19] I think [06:21] we're kind of taking advantage of like [06:23] the the the mental space of being out on [06:26] break and then being able to sort of [06:28] ruminate on things in the background. Um [06:30] so hopefully there's maybe some [06:31] interesting ideas that come from that [06:32] that we can merge with your scripts and [06:34] and just kind of come to a solution [06:35] that's like [06:37] >> sorry reliable for everyone. [06:38] >> You're on break. What [06:40] >> So Cali and I were both on PTO last week [06:42] and I think Callie's taking a fair few [06:44] days. [06:45] >> Oh, okay. I didn't realize that. Yeah, [06:48] >> because you were on a call last You were [06:49] on a maintainer call last week. [06:50] >> I Yeah, I I sometimes I'll try and pop [06:52] in because the maintainers call the it's [06:54] just regularly scheduled call like [06:56] that's I usually try and pop into those [06:58] regardless. But [06:59] >> if you take PTO, you should take PTO as [07:02] this is a life lesson from a person who [07:04] hasn't always done that. [07:07] Um you do burn out if you don't take [07:10] your PTO. [07:12] >> Yeah. I know generally I was I was [07:13] generally doing it all but um [07:15] >> I'm not yeah I'm not my intent my [07:17] intention is only to provide [07:20] uh the less the purpose of my life may [07:22] be to serve as a warning to others [07:25] and I have I have uh done work on PTO in [07:28] fact I wrote a good deal of augur [07:31] um the augur schema in um bise in uh [07:36] 2018 [07:45] So is this band this bandit thing this [07:48] is probably in response to my [07:51] observation that bandit was just [07:53] littering the security tab. [07:55] >> Um that that and also I noticed that I [07:58] think there was a PR that got filed. I [08:01] forget which one. It's probably linked [08:02] in the underlying issue, but um there it [08:05] was also [08:07] uh do kind of spamming some of the PR [08:09] reviews with unit test things. [08:12] >> That's what I thought. I saw that too. [08:15] >> I sort of just grabbed I think I just [08:17] grabbed that line of PI project tommo [08:19] from the I think from the bandit docs. [08:21] So it should work. Um given that we also [08:23] have the R directory is also called [08:24] tests. But [08:28] um yeah, that would that's probably [08:29] going to help cut through it. I I did I [08:31] did sort of add Bandit as sort of a [08:33] redundant because I think I know GitHub [08:34] already does their own they've got their [08:36] own code scanning thing, [08:37] >> but I'm not I wasn't sure it was [08:39] necessarily catching because I know I [08:41] know there's probably a lot of just like [08:43] I've I've made PRs just in the last [08:47] maybe not last few months but like re [08:49] some relatively recently since Docker [08:51] started working in Augur to like remove [08:55] the logging of like sensitive [08:57] information in the log like censoring [08:59] API keys and stuff like just easy stuff [09:02] like that. So I just was thinking like [09:04] hey maybe this is another layer to to [09:05] catch more of that kind of thing just so [09:07] that people don't look at Augur and like [09:10] oh my god is this a security nightmare. [09:12] Um, [09:13] >> yeah. [09:14] >> I mean, I just [09:15] >> making something that easy so broken. [09:17] Like, what else is broken, right? [09:21] >> But I threw that in the milestone 9 like [09:23] that's um Oh, that this is a different [09:26] Yeah, this is we're [09:27] >> Yeah, I already [09:29] that was so straightforward. I [09:32] >> saw no reason not to. [09:35] >> Um, [09:37] >> this is effectively [09:40] sort of a logging change anyway. So this [09:42] isn't any [09:43] >> and version. When you say version, that [09:46] made me think version of augur. [09:49] >> Yeah, version of augur. I the PR has [09:52] actually changed from that now because I [09:53] was trying to import the version number [09:55] of augur from that uh metadata file, [09:59] >> but it was it was causing issues because [10:01] I think that's not technically part of [10:02] the augur like project tree like augur [10:06] module. So it couldn't import it. And so [10:08] I just I think I just defaulted to like [10:10] I think there's one place where we [10:11] already import the version number. So I [10:13] was able to reuse that. All the other [10:15] places I just have augur and then like a [10:17] little piece of text of like oh this is [10:19] coming from the CLI, this is coming from [10:21] the API, this is coming from whatever. [10:24] Um [10:26] I'm going to double check to see if I [10:27] actually [10:28] >> So what is what is setting the augur [10:29] version? Um, [10:32] what's it? I just I just [10:34] >> No, the the version is this. So, this [10:35] this is just setting a string that [10:36] basically identifies which application [10:38] the database connections are coming from [10:40] cuz I like I've been I was looking in [10:42] DBver just to see the list of [10:44] connections and like I saw some that [10:47] were obviously from the user account I [10:49] created for for Graphfana access. [10:52] >> Um, and then there was a couple, but [10:54] basically everything had no name except [10:56] for the connections that DBaver itself [10:57] was making. And I guess they they've set [10:59] it up to to set their like set like hey [11:02] this is dbaver whatever. Um so that you [11:06] know like this connection is like which [11:08] application like where this connection [11:09] is coming from basically. [11:11] >> Um so I was just like this is sort of [11:13] just a like nice to have quality of life [11:15] like [11:17] >> almost no one's probably going to even [11:18] notice this but it might help with [11:20] dugging of [11:21] >> connections. Yeah. like um because I [11:24] have my 8 knot server on a different IP [11:28] than Augur, I can now tell the [11:32] difference in the connections, but [11:33] before that I really couldn't. So, I [11:37] think this would be pretty helpful. [11:42] I don't know. probably wait for someone [11:44] else to look at it just because [11:46] >> yeah this is this is I think this was I [11:47] like I mentioned this um if you look at [11:49] the milestone of this PR it's I put it [11:52] in 0.94 like I was like okay not this [11:54] next release but the one after just [11:56] >> this.93 [11:57] >> this isn't that important it's just it [11:59] like I I think I'll usually maintain two [12:02] milestones open at any given time is the [12:04] the next release and the release after [12:06] so I can kind of like okay this is more [12:07] immediate priority than and this is like [12:10] we get there when we get there kind of [12:13] Yeah. [12:14] >> So, [12:16] um [12:22] I don't [12:22] >> And I'm guessing these builds are just a [12:24] timeout crap. [12:26] >> Um both of them failing is interesting. [12:30] I need to I would probably that might be [12:32] worth looking into. Uh, let me rerun [12:35] them before you do that. [12:37] Just because like depending on I' I've [12:41] seen the [12:43] I've seen it fail when [12:47] GitHub gets busy, right? [12:49] >> Oh, GitHub. Yeah. No, GitHub has I think [12:52] GitHub's like status page has been all [12:56] the rainbow colors in the last [12:58] >> Yeah. [12:58] >> I think they've been getting it pretty [13:00] pretty hard with the AI scrapers. Um, [13:02] >> yeah. I mean, I know [13:05] Oh, [13:05] >> during our maintainer call last week, my [13:08] GitHub was extremely slow. [13:10] >> Yeah, it was I think it was glitching [13:12] out pretty hard for me as well. Um I [13:14] also I bet I bet our that observation [13:18] you made on our insights tab about how [13:20] um we've just been getting like hundreds [13:22] of clones a day. [13:23] >> Oh, you think that's probably AI [13:25] scrapers? That would make sense. Well, I [13:27] don't but the thing the thing that is [13:28] quite is making me wonder more is like [13:30] if it wasn't AI scrapers and it's people [13:32] actually using Augur then [13:34] >> what do we hear from [13:35] >> people more people using Augur means [13:37] potentially exponentially more calls [13:40] made to GitHub's API which they might I [13:43] don't know if they'll be a like which is [13:45] probably going to look a little bit like [13:46] a scraper and so it's like I feel like [13:49] we're almost contributing to GitHub's [13:51] downtime problems [13:53] not in a like bad way just in a like, [13:56] hm, we we have an application that [13:58] hammers their API pretty hard. [14:02] >> Yeah. I just I mean, I still don't [14:04] understand where who are all these [14:06] people using Augur [14:09] >> or who are all these people cloning [14:10] Augur. I think it's bots. I love I love [14:12] the visual clones in the last 14 days, [14:15] too. It's like it looks like it just [14:16] goes through the roof. I love that. Hang [14:18] on, let me take a screenshot of that. [14:20] >> That was yesterday. Like that's just [14:21] because yesterday it was uh 470, [14:25] but the day before that was a small [14:27] number. [14:28] >> That's just hilarious. I I just love the [14:30] meme of that image. [14:35] >> Yeah. [14:37] So like, yeah, I don't know. Unique [14:39] visitors seems pretty normal. [14:43] Total views seems like these seem like [14:46] normal numbers. It's the clone numbers [14:48] that seem whack. It it's I bet you it's [14:51] AI scrapers. [14:53] >> Yeah. And I looked I looked and I've [14:56] looked around the GitHub API and there's [14:58] really no way to get any information [15:00] about who's cloning it. [15:02] >> You you you really can't like I don't [15:04] you wouldn't you wouldn't be able to [15:05] know. I don't think [15:08] >> the best GitHub could probably give you [15:10] is an IP address, but then they're not [15:11] going to be that's like technically PII [15:13] and they're not going to share that. Um, [15:16] >> yeah, I thought about putting in an [15:18] optional telemetry [15:20] um [15:22] snippet, but just because I'm curious, [15:27] >> it might be I wonder if it's worth like [15:29] because I wonder if we could do that [15:30] kind of diagnostic script idea of like, [15:33] hey, like do a little selfch check on [15:34] your augur instance and then that can be [15:37] kind of like we we can have it generate [15:38] a little like text file format that's [15:40] like, hey, here's a report like here's a [15:42] little debugging report that you can [15:43] submit if if you have an issue and then [15:46] just encourage people to contact us with [15:47] it or maybe I know places like Pi like [15:50] applications like Pi Hole will have an [15:51] option in their little debug like their [15:54] little like self test menu like hey [15:55] upload this report and make like it'll [15:57] upload it to like a paste bin site or [15:59] something or their own version of that [16:02] >> and then give you a link that you can [16:03] share in your issue or something. So [16:05] maybe maybe we can do that. I don't [16:07] know. [16:09] I want to be careful not to like turn [16:11] too many people away with with [16:12] telemetry, but [16:13] >> Yeah. No, exactly. It would have to be [16:15] optional. I actually like your idea [16:17] better because we're offering to help. [16:20] And honestly, if you're running augur, [16:22] you're going to want to check it and [16:23] you're probably going to have something [16:24] to submit if you've been running it for [16:26] any period of time. And if nobody sends [16:29] us anything and we've still got all [16:30] these cloners, then I know there's some [16:32] kind of odd bot activity happening. I [16:35] also wonder too if this clones graph is [16:38] it counting fresh clones or is it [16:40] counting pull like pulls of new commits [16:43] because if we've got like this almost [16:45] looks like it could just be like a bunch [16:49] of people [16:51] who just have auger cloned already and [16:53] are just periodically pulling for [16:55] updates like if and so like I because I [16:57] wonder so when were the dates that we [16:59] last [17:01] merged stuff to main because I know we [17:04] did just now [17:05] >> two months since since we um Oh, since [17:09] we merged Yeah, we actually been merging [17:11] stuff to Maine with pretty regrettity, [17:15] >> right? [17:16] >> Yeah. So, well, I'm curious if I'm [17:17] curious if some of those dates line up [17:18] with some of those big spikes on that [17:20] graph because the LA So, the last one I [17:22] see is March 12th. What was the What was [17:24] the graph again? [17:26] Uh, [17:29] we only get two weeks, but let me [17:32] >> Yeah, we can't I don't know if we can go [17:33] back too far, but I'm like [17:35] >> we can only go back two weeks unless [17:37] we're pulling the data regular. [17:38] >> Okay, March. So, March 12th. So, that [17:40] doesn't really line up with that spike, [17:41] but was it what was the [17:43] >> like Okay, the 12th to the 16th, that's [17:45] four days, and today's a Tuesday. [17:49] >> So, March 10th March 10th was a a day we [17:52] merged. There was some stuff on Maine. [17:54] And that does line up with that big [17:56] spike [17:58] uh with the that sort of middle spike. [18:00] >> Actually, we merged we merged a couple [18:02] things. We merged quite a few things on [18:04] March 10th. [18:06] >> Yeah, that that would be the maintainers [18:08] meeting. So, [18:09] >> and then March 4th was the one before [18:10] that which does line which lines up with [18:12] the previous spike. So, I I I do sus I [18:14] suspect it's I I wonder if that could [18:16] just be I mean I imagine some amount of [18:19] it is AI scrapers. [18:21] I some of it is probably also just [18:23] people like who have Auger cloned are [18:25] like updating their clones which I guess [18:27] is a good sign that people are keeping [18:28] their instances up to date. [18:30] >> Yeah. I mean anybody restarting Docker [18:32] is going to pull this. [18:35] >> Uh restarting yeah if you update your [18:38] Docker install it'll you'll do that but [18:39] I that's not necessarily an automatic [18:41] thing. [18:42] >> Okay. Um [18:44] anyway is so I also wanted to I know uh [18:47] there was kind of some efforts to [18:48] include more 8not in these meetings as [18:50] well. So I had I I think I know Kaio's [18:52] Kyle's out today like he mentioned on [18:54] Slack but [18:55] >> um I did have some kind of updates that [18:57] I think might be relevant to sort of [18:58] bring [19:00] >> you it's just in the agenda if you want [19:02] to if you wanted to pull that up. Um [19:04] >> but it's basically just um so we [19:07] recently we're working on currently is [19:10] reenabling the heat there are some heat [19:12] map diagrams to kind of like which files [19:15] have been touched at what times sort of [19:17] in a in a grid view to help help you [19:20] kind of see is a particular group of [19:23] files or a particular subset of the repo [19:24] like missing a maintainer and not been [19:26] touched in a long time. I.e. I imagine [19:28] you'd see something like jump start in [19:30] our repository to that because that [19:31] folder hasn't been touched in two years. [19:33] Um, but it's just kind of another health [19:35] metric. I think we disabled it because [19:36] the queries were taking a really long [19:38] time to run and we recently I think [19:40] since we disabled it. Um, we we there's [19:44] been other improvements to augur like [19:46] ignoring being able to ignore repos. And [19:49] so it's we're we're slowly bringing back [19:52] we're trying to maintain we're trying to [19:53] bring that back into 8 and re like [19:55] reenable the queries move them into [19:57] materialized views things like that. So [20:00] there, so the first point with this is [20:01] that there's now that we have some [20:03] additional materialized to use to bring [20:05] into auger upstream. [20:06] >> Um, [20:08] >> yeah, [20:09] >> I'm not sure. I don't know [20:10] >> especially if they're in especially if [20:12] eight's expecting them. [20:14] >> Yeah. So the problem with that is though [20:16] I don't think our current Olympic like [20:18] database migration config [20:21] is able to do [20:24] uh materialized views. I think a lot of [20:26] our materialized views have just existed [20:28] forever and that like they're in some of [20:30] the old SQL scripts that are like the [20:32] legacy migrations. [20:33] >> I think Kelly and I created most of the [20:35] materialized views like four years ago, [20:38] maybe five year five years ago. [20:40] >> Yeah. So do you think that was [20:41] potentially before we added Olympic? [20:47] it we were using. So it was it was kind [20:49] of the state that you saw when you [20:51] started working on augur where we had [20:52] that giant script that created [20:54] everything up to a point and then [20:56] everything after that was Olympic but I [20:59] guess no it was yeah so you know how we [21:02] >> it sounds like [21:04] >> yes I I think it was possibly before [21:06] Olympic. Yeah, because it sounds like it [21:09] sounds like a lot of like some of the [21:10] stuff that Cali was referencing in that [21:11] issue for bringing these data migration [21:14] these uh materialized views in was [21:17] looking at some of those older SQL files [21:18] and I'm like we shouldn't edit those [21:20] because a they're for older migrations [21:22] and only apply to people upgrading from [21:24] that old versions of augur and also like [21:27] that's not going to apply to people [21:29] upgrading their instance from this last [21:32] version to this version. So I think we [21:34] need to figure out a better way like a [21:36] figure out a way to get Olympic to see [21:38] materialized views and be able to define [21:41] those in [21:43] I don't know how we would do that like [21:45] define them in maybe a Python model in [21:48] our [21:50] um in SQL Alchemy or something just [21:52] because I like I think we definitely [21:53] need to we need to have those versioned [21:56] and controlled the same way we do our [21:58] regular schema. Um [22:00] >> yeah I'm just where are those scripts? [22:03] >> Uh they are in application DB. It's in [22:06] the it's alongside I think some of them [22:09] are alongside all the other if you go to [22:11] schema. No not sorry I lied. I said DB. [22:14] Uh [22:14] >> if you go to schema [22:16] >> I think YIC there's there's a like some [22:19] of the legacy migrations there's some [22:21] files in there. I think there's several [22:23] SQL files everywhere and [22:26] I don't remember uh [22:29] actually [22:30] >> materialized view update. So yeah, [22:31] explore materialized view update. This [22:33] is three years ago. [22:34] >> What's in that? Is that Oh, is no that's [22:37] an Olympic file. What's what's in that? [22:39] Um [22:42] number four. Can you open that one? [22:44] >> Yeah. [22:45] >> Explore the one you were just looking [22:46] at. [22:47] >> Yeah. [22:47] >> I love the um lexical graphic ordering. [22:51] >> Yeah. I mean [22:52] >> add material. So what's that function [22:53] do? Add materialized views 15. Oh, [22:56] that's just [22:58] >> that's just raw. Okay. So I guess we [22:59] could do it the raw SQL way. That's I [23:02] don't like that. [23:05] But [23:08] it would be nice if we could get Olympic [23:10] to like generate migrations to [23:14] an app. [23:14] >> I mean syntactically syntactically views [23:17] are queries, right? materialized views [23:20] are just physically persisted queries [23:23] because that's a lot faster. [23:26] Um, [23:27] >> yeah, [23:28] >> like logically for what a limbic does, I [23:32] would wonder if [23:34] it could support all the things that we [23:36] need in a materialized view. Like um one [23:40] of the things you may or may not know [23:41] this that you want to make sure to have [23:42] in any materialized view, you want to [23:44] have a primary key because without a [23:47] primary key, you can't refresh data [23:49] concurrently. And if you can't refresh [23:52] data concurrently, that locks that [23:54] materialized view from any connection or [23:57] use while it's being refreshed. [24:03] Yeah, I mean that that's less of a [24:04] problem for 8 because we we just [24:06] regenerate all the materialized views [24:07] when we clone the DB, but that's that's [24:10] a very not typical setup because we're [24:14] operating in such a scale that we can't [24:16] have augur collection and 8 not running [24:19] off the same DB or at least that was [24:22] when we were behind on collection. So I [24:24] think now it might be better now that [24:25] we're caught we're relatively caught up. [24:27] But [24:27] >> yeah, [24:28] >> I I'd like to make some more performance [24:30] fixes before we try and remerge those [24:31] two. Um, but speaking of performance [24:33] too, I think one of the other things [24:34] that this materialized view conversation [24:36] brought up was I I was I was kind of [24:39] poking around and I noticed [24:41] >> I I filed an issue a little while ago [24:43] about just the sheer size of the repo [24:45] labor table and I realized [24:49] >> that a lot of the queries for this a lot [24:51] of the materialized views for this heat [24:52] map join on that table. So I I suspect [24:56] that the the bulk of that table is [24:59] potentially what was causing this query [25:01] to be so slow in the first place that we [25:03] disabled it in 8 knot to begin with. [25:06] >> Yeah. Um [25:06] >> I that table is a huge percentage of the [25:10] disk utilization [25:12] >> because it's persisting the analysis [25:15] from all time. [25:17] um that's been extremely useful in uh [25:21] I'm we're working my research group is [25:24] working on understanding the evolution [25:26] of complexity of software uh with the [25:29] dawn of AI and having that history has [25:32] been of phenomenal use to us. Uh however [25:36] it is a giant table and the things that [25:39] are useful could in could almost [25:42] certainly be summarized down into a [25:44] smaller table. [25:46] So what I'm not even saying like remove [25:48] the like I think having that history is [25:50] useful. I'm just thinking that there's a [25:52] lot of duplication between what we're [25:54] storing in this table and what's [25:55] existing in other places. Like I think [25:58] we already have like PR files commit the [26:01] commits table is already effectively [26:02] commit files. We just need to rename [26:05] that and fix that. But I think it might [26:08] make sense to move towards Marie having [26:10] a table that's like files that is [26:12] referenced in other from all the other [26:14] places that we can have files and then [26:17] because then the repo labor is [26:18] essentially just sort of analysis of [26:20] those files, right? like it's looking at [26:21] how many lines of this code is comments [26:24] versus code versus I I forget exactly [26:26] what it looks at, [26:27] >> but it's got those like like the B the [26:29] core of that table is like five columns [26:31] that are just small integers. [26:33] >> So, [26:34] >> so I think if we can dduplicate storing [26:36] like the file paths and the file names [26:38] and things and put those in a different [26:40] table and have that be a foreign key [26:42] reference. So we did try that initially [26:47] and the reason that we we sort of [26:50] evolved away from it and I'm not saying [26:53] that this is an unsolvable problem. I'm [26:54] saying that we decided not to try to [26:56] solve it. [26:58] The when you're analyzing the pull [27:00] requests and the files that's the files [27:02] in that pull request at that point in [27:04] time. When you're analyzing repo labor, [27:08] those are the files in that repo at that [27:11] point in time. Files are added and [27:14] removed routinely. So [27:18] there isn't actually in augur anything [27:21] that is the canonical here are the files [27:24] in this repo except for the most recent [27:27] uh commit files. But but we don't [27:30] actually have a a like a file reference [27:34] that is independent from some analysis [27:38] of the file whether it's pull request [27:40] files uh commit files or the repo labor [27:44] files. [27:46] So there is no there is no list of files [27:49] historically and I think we would need [27:51] something like that to have a central [27:53] files table because those three entities [27:56] are going to be they'll have different [27:58] inventories of files at any moment in [28:01] time. There'll be one or two files that [28:04] are new in one of them and one or two [28:06] files that are missing in one of them. [28:08] Do you see what I'm saying? [28:09] >> Yeah. I think that would still be um you [28:13] could still have that even with sort of [28:15] centralizing the files table cuz like [28:17] the files table could essentially like [28:18] if any of like whether it comes from a [28:20] PR or a commit or whatever like there's [28:23] information that is the same across all [28:25] of those sources where it's like what [28:27] file is it like some of the some of the [28:30] basic there's things that are common to [28:31] the concept of a file I think [28:35] >> like how putting those in one table [28:36] would we might be able to do some [28:38] dduplication but that's a a issue, but I [28:42] think we'd still be able to look at it [28:43] from the different views because it'll [28:45] all be foreign key. Like we could do [28:47] okay pull requests like we could have a [28:51] table that links pull requests and the [28:52] files that are in that pull request and [28:54] you could do a join through that. [28:57] >> What about [28:59] So I'm just I'm just like thinking [29:00] through the problems that we had when we [29:02] tried to do this. [29:05] What about when [29:09] like I have a centralized list of files [29:12] when there's a significant refactoring? [29:15] Sometimes nearly every file moves [29:18] around. So now what do we do with the [29:21] history of the file? [29:23] Do we keep that in a table? Because if [29:26] we're using a centralized files [29:28] inventory, the pull request files and [29:32] the repo labor files path, they both [29:35] need the history. They need to see the [29:37] files at the moment in time that those [29:40] analyses ran. [29:42] Yeah, we don't need like I don't think [29:45] we need to necessarily be trying to like [29:48] like we could still store [29:51] like multiple records for like if if [29:53] you're looking at a particular like set [29:57] of rows that are like for for this repo [29:59] and like what like you you could like [30:01] for example if you were to query this [30:03] hypothetical future files table for a [30:06] given repo I think it would be [30:08] reasonable for you to see like okay [30:10] here's like a thousand entries for [30:13] basically the same file path all with [30:16] different collection dates like okay [30:17] this file was seen at all these [30:19] different times maybe half of them are [30:21] from a PR half of them are from a commit [30:24] like several of them are like you could [30:26] kind of [30:29] I I don't I mean and then you could get [30:31] into some of dduplication where like [30:32] okay maybe it's one file and then but [30:35] you could access it from from either a [30:37] foreign key from the like pull request s [30:41] or you could do a foreign key from the [30:42] repo labor like multiple things looking [30:45] at that one file but you'd still have [30:47] several entries for that file path as [30:50] it's like over time like every time it [30:52] changes bas or something like [30:54] >> okay [30:55] >> uh you it it sort of depends on how you [30:57] lay out the tables but I think we you'd [30:59] still have the history I think I think [31:00] it's entirely possible to still keep [31:02] like okay this file like this file got [31:05] added here [31:08] and and to be able to find ways of like [31:10] where it's where it was last seen and [31:12] stuff. Um, [31:14] >> okay. I mean, [31:15] >> it's definitely going to require a lot [31:16] of thinking through and planning, but I [31:17] I think that's definitely a [31:20] worthy direction to try and go in. Like, [31:22] so I think I think a database should [31:23] really be trying to map these concepts [31:26] of like, okay, here is a file, here is a [31:28] pull request, here is the relationship [31:30] between the two, like, okay, here's how [31:32] a file relates to a commit. um [31:37] and try and kind of move towards [31:38] normalizing things a little bit. [31:40] >> So I have so one question I will have I [31:43] have for Cali who's not here I [31:45] understand but is the foreign analytical [31:49] utility of that which I think what I'm [31:52] hearing is it would lower the learning [31:53] curve basically like a lot of these [31:55] changes would reduce the learning curve [31:57] for data scientists leveraging augur [32:00] right? [32:03] Yeah, that sounds that I think that's a [32:06] yeah, that would probably be a benefit. [32:07] Like that's I think that's just a [32:08] benefit to helping kind of normalize the [32:11] tables a little bit in like maybe don't [32:13] go too overboard, but like yeah, having [32:16] like having just a one place where you [32:18] can look at files. [32:20] Um, and then you can you can get [32:23] different views of that depending on [32:24] whether you're sort of joining into that [32:26] table from a poll request or a commit or [32:29] a repo labor or something. So then the [32:32] then another question becomes is [32:35] does the cost of doing all of this [32:39] coordination when we're collecting data [32:42] make sense or is this an example of a [32:46] place where perhaps we need some kind of [32:50] analytical version that we extract data [32:53] from augur into. [32:57] Um, I think that I think the costs [32:59] during collection are probably [33:00] relatively negligible. I know we could [33:02] probably rely on I mean we can like SQL [33:05] Alchemy does have pretty strong support [33:08] for like doing relationships and things [33:09] and and we could we could kind of [33:12] rely on it a little bit more to like [33:15] handle. Okay, we're going to just create [33:17] this object and populate this like files [33:22] files thing with whatever. And then it [33:24] just sort of figures out where that [33:25] table where that goes in the table. Um [33:30] I I think I was thinking of something [33:32] else too. Um [33:34] we another one of the approaches I was [33:36] also thinking of as far as like moving [33:37] towards sort of trying to normalize our [33:39] database is yeah taking advantage of [33:41] things like materialized views to sort [33:45] of since since 8n's already basically [33:48] relying on the API the database schema [33:50] anyway largely through materialized [33:52] views I think we can kind of take [33:54] advantage of that and use the [33:55] materialized views as sort of like an [33:56] outer castle wall of here is our sort of [34:01] suggested like we'd rather you query [34:03] these. Um, and then so then we if we [34:07] need to shuffle other tables around, we [34:10] can like if we we make a change to the [34:12] schema for like some of the like PR [34:15] files or something, we can make that [34:18] change both in the in the core schema [34:20] and in the materialized view such that [34:22] the output of the materialized view [34:24] doesn't change and so downstream [34:26] applications don't like really have to [34:28] worry about it. same with if we slightly [34:30] tweak how we're processing the data in [34:32] one of our REST endpoints, but the [34:34] output JSON is still the same. Like I [34:36] think we can kind of move towards that [34:39] with materialized views or or I think [34:41] even some of the um [34:42] >> I think Michael in in the data [34:44] engineering channel was trying to start [34:46] a discussion on some like semantic [34:47] layers and more sort of data tools for [34:51] what it sounds like is doing a very [34:52] similar thing. Um, so I think maybe we [34:55] could do that and we could just sort of [34:57] maybe define what our ideal perfect [34:59] schema is, make materialized views for [35:02] all those tables and then figure out how [35:04] to cram whatever jank tables we have now [35:07] into that and have that be sort of our [35:10] defined public API. And then that'll [35:12] build us some space so that we can [35:14] internally take whatever time we need to [35:16] migrate our core tables such that it's [35:19] maybe a better like the queries that [35:22] convert to those materialized views [35:23] aren't as horrible and crusty and like [35:27] terrible. [35:28] >> Yeah. [35:28] >> Um [35:31] >> that makes I mean okay that that seems [35:33] like a logical [35:34] >> Yeah. It's it's it's a large effort. I [35:36] don't think it's necessarily in the [35:37] cards for this year, but I think like [35:39] >> a lot of this stuff is like long like I [35:41] I think I've been definitely trying to [35:42] kind of look at issues of like okay [35:44] what's like are there these big [35:46] long-term goals that we can just sort of [35:48] think about like every time we do a data [35:50] migration can we just think about okay [35:52] at least make sure that all the small [35:54] changes we're making are at least taking [35:56] steps either towards or not away from [36:01] like these larger goals of normalizing [36:04] the table like I think some of the [36:05] smaller changes that I think we were [36:07] talking about in previous weeks as far [36:09] as potentially like using the um [36:14] using like the the contributor aliases [36:16] as sort of just the canonical source of [36:17] emails instead of having an email in the [36:20] contributors table and also a list of [36:21] aliases like just having that all like [36:24] here is a table that just lists the many [36:26] emails that one contributor can have [36:28] like just a simple many to one table [36:30] structure. Um, [36:34] I feel like I've been talking for a long [36:35] time. Just I know SH just joined. So, is [36:37] anyone else got other things to add to [36:39] the agenda? [36:40] >> Yeah. Hey, Schlock, we see you come in. [36:43] We've just been looking at PRs and [36:45] >> Hey guys. Yeah, I've I've been seeing I [36:48] get those GitHub emails. It's just that [36:51] I've been really busy with sort of [36:52] interviews and everything. Uh, I haven't [36:54] had a chance to do much, but I have my [36:58] final Red Hat interview in like 15 [37:00] minutes. So hoping that I'll be done [37:03] with everything by the end of this week [37:04] and then, you know, actually get back to [37:06] order because [37:07] >> I really don't have anything else to do [37:09] recent. Uh I've, you know, defended my [37:11] thesis. I'm I've submitted a couple of [37:14] papers. So sort of like a prelude now. [37:17] Just wrapping up all of the stuff and [37:19] then get back to order. [37:21] >> Yeah. [37:21] >> Well, that's awesome. Congrats on the [37:23] final round. [37:24] >> Yeah. Thank you. Yeah. Let's see. Uh but [37:27] yeah, other than that, I don't think I [37:28] really have much to add. I'm just trying [37:30] to, you know, listen more and see, you [37:34] know, what are the biggest pain points [37:36] and if there's something that I could [37:37] contribute back. [37:39] >> Yeah, if if you've got time. I think um [37:41] I don't really have much like I my focus [37:44] this week is mostly going to be on [37:45] working with Cali on figuring out how do [37:47] we clean up our data from some of the [37:49] the collection bugs of the last few [37:51] weeks. Um, if you really wanted [37:53] something to do, I think getting started [37:55] maybe on figuring out how how we can get [37:58] to to like how how we can sort of bring [38:00] in the materialized views into our [38:02] database sort of versioning process with [38:04] YIC I think would be helpful. I think I [38:06] dro I think I found a couple of links to [38:08] to sort of start the research process in [38:10] whatever relevant issue is in this in [38:13] these notes. Um, but I haven't really [38:16] looked into it more than just a cursory [38:18] click the first stock overflow link I [38:20] found. [38:20] >> Um, [38:22] So if that's something that you've got [38:24] you've got time for like towards the end [38:25] of this week like it's not super high [38:27] priority but it's like [38:28] >> eight nod is starting to kind of [38:30] introduce its own materialized views and [38:32] that's not a very good user experience [38:33] for someone who's just expecting docker [38:35] compose up on two stacks and having it [38:37] work. [38:38] >> Yep. Yep. Yep. Obviously. Yeah. All [38:40] right. Yeah, that makes sense. I'm not [38:42] sure if [38:44] supports materialize views. I think [38:46] there's like a plugin or like a third [38:48] party thing is what? Okay. [38:50] >> What branch would I have? What branch of [38:52] 8 are those materialized views required [38:55] for Adrian? [38:56] >> So right now that would be I'm worried [38:58] that would be a breaking change for my 8 [39:00] knot instance when I pull it the next [39:02] time. [39:02] >> So we we just released a new version [39:05] just to pull in mostly performance like [39:07] a particular sort of performance fix for [39:09] the search bar. the the reenabling the [39:11] it's reenabling the heat maps that's the [39:13] thing that's gonna sort of cause a lot [39:15] of this that has technically not been [39:17] merged yet I think it's it's pretty [39:19] close I'm probably going to be looking [39:20] at it today and or this week um [39:23] >> is it in dev [39:24] >> it's just a feature PR it's just a PR [39:27] feature branch [39:28] >> um [39:29] >> but yeah those feature branches do merge [39:30] into dev so dev is equivalent to our [39:32] main and then release and then main and [39:35] 8 knot is equivalent to our release the [39:37] naming is a little bit backwards but [39:38] it's conceptually the same process of [39:41] PRs go into a dev branch, go into a [39:44] release branch. [39:45] >> Yeah, I know. I think yours is a little [39:47] bit easier for people to understand that [39:48] they're working with a dev branch when [39:50] they hit the repo. Honestly, like if I [39:52] go back in time, I might rename our [39:53] branches following your pattern. In [39:56] fact, we can do that, [39:59] but I'm not suggesting we do that right [40:01] now. Just [40:02] >> Yeah, [40:03] >> it might make sense if the two projects [40:05] were consistent [40:07] as I think about that. [40:09] So yeah, I mean I think if if it gets [40:11] into dev if there would be a way that [40:14] you could at least I don't know [40:17] I think we should put something into [40:19] augur to indicate that even if we don't [40:21] have it automated it would be helpful to [40:23] know that you have to create that [40:25] materialized view or those materialized [40:27] views whatever they were and have a [40:28] script for it. Yeah, I know. I know. [40:31] I've definitely I I think for our most [40:32] recent sort of release of 8 knot I I did [40:35] sort of push for having like a tagged [40:37] version number and I think that was sort [40:39] of moving towards the idea of putting in [40:41] the eight not read me like okay here are [40:44] the versions of augur that these [40:45] versions of 8 knot was sort of designed [40:48] alongside [40:50] um [40:50] >> right [40:50] >> so that like if you're just trying to [40:52] run an old version you know okay I need [40:55] to be using one of these versions of [40:56] augur with this version of 8not it as I [40:59] don't really wanna [41:00] necessarily have to like be super strict [41:03] about that. Like I'd like the [41:04] compatibility to be pretty open, but I [41:07] think [41:08] >> like especially if we've got things like [41:10] this where like we're like there's new [41:13] materialized views and new new things [41:15] happening like that's why I think this [41:16] process of being able to bring the [41:18] materialized views in upstream because [41:21] basically what's happened is our process [41:23] for versioning DB has changed since we [41:26] last realistically looked at bringing [41:28] materialized views in. So we need to [41:30] figure out okay how does that new [41:32] Olympic process [41:34] >> handle materialized views like I'd like [41:37] to use Olympic if we at all can but like [41:39] >> utter worst case is like we just have a [41:41] folder of SQL files that is like the [41:43] current materialized views and every [41:46] time you upgrade augur we just wipe them [41:47] and regenerate them which is going to be [41:50] a royal pain for huge instances like [41:52] ours but for the majority of the smaller [41:55] instances it shouldn't be that impactful [41:58] I don't know [41:58] >> yeah Um, [41:59] >> I mean I think I mean I'm just I'm [42:01] thinking of myself right now. I I just [42:03] don't want my 8 knot instances to break [42:06] the next time I upgrade update 8 knot [42:10] >> and I think most people are like that. [42:12] >> Adrian Adrian quick question. Is there [42:14] an existing issue that's tracking this [42:16] materialized views uh issue? [42:19] >> Yeah. So I think I put it it should be [42:21] in [42:22] >> this one here I think. [42:23] >> Yeah. So there I think there's an issue [42:24] on the a side and there's an issue on [42:26] the eight knot side. [42:28] >> Um Yeah, [42:28] >> I think I made an eight not one that was [42:30] like, "Hey, we should probably upstream [42:32] these." And then I think Callie made [42:34] this issue that was like, [42:35] >> "Got it. [42:36] >> Here's the upstream side of this [42:37] conversation." [42:38] >> Yeah. [42:38] >> Um, [42:39] >> gotcha. Gotcha. Okay. [42:42] Okay. All right. Sounds good. I'll go [42:44] through read through that conversation. [42:45] Okay. I also see a Stack Overflow link [42:47] as well. I'll read through that as well. [42:49] Okay. Sounds good. [42:55] Yeah, I think other than that my my [42:56] focus in the sort of slightly more [42:59] distant future is like okay starting to [43:02] landing more like sort of more of those [43:04] kinds of PRs Sean that we were merging [43:06] last week as far as just like okay let's [43:09] start moving like moving some of the [43:11] functions we have to generate URLs into [43:14] our actual GitHub API class and sort of [43:17] starting to reorganize some code so that [43:19] things [43:20] >> we can sort of be like okay now Let's [43:22] take this GitHub class and swap it out [43:24] for a 4Jo1 or something and hopefully [43:27] things work. [43:29] >> Yeah, there's one other PR that Adrian [43:32] did that I just wanted another set of [43:33] eyes on before [43:35] >> we it's this one. I'm looking for my [43:38] chat window in Zoom as we run out of [43:40] time in this meeting. [43:41] >> This is I think this was actually this [43:42] wasn't in the notes, but yeah, this is [43:43] this is relatively recent. Uh it's a [43:45] it's just a small quality of life. It [43:48] just has augur basically assign its own [43:51] name to connections that it's making [43:54] into the database so that when you're in [43:56] the like um sessions view whatever it's [43:59] called in DB or your other tool of [44:00] choice like if you're looking at just [44:02] the list of all the sessions of that are [44:03] connected to the database you can see [44:05] which ones are from augur and which ones [44:07] might be unnamed from other places. Um [44:15] >> yeah, if you have a chance to review [44:16] that schlock, but we're out of time now, [44:18] so [44:18] >> y [44:19] >> we'll um let everybody go and um thank [44:23] you both for all of the work that you do [44:26] and making augur so much better than it [44:29] was six months ago. [44:32] >> Yeah, thank you guys for helping helping [44:34] review all the PRs. Like I know we're [44:36] still got a huge backlog, but I think [44:37] we're working through it. hopefully do [44:39] five or so small ones for for the big [44:42] for each release and hopefully get [44:44] through it eventually. [44:45] >> Yeah. [44:45] >> Yeah. And Yeah. Yeah. Yeah. We could go [44:48] We should get at some point we should [44:50] decide how to get rid of the ones that [44:51] are stale and that the people aren't [44:53] coming back to. [44:53] >> I've been I've been starting to close [44:55] some of those and just kind of like I' [44:57] I'm trying to I think I tend to leave [44:58] open the ones that like have maybe a [45:00] kernel of being potentially [45:02] like it might be worthy of someone like [45:05] hey there's there's a kernel of good [45:07] stuff here. Maybe maybe someone can take [45:08] it over and and clean it up. [45:10] >> Yeah, [45:11] >> but I I might need to start being a [45:13] little bit less friendly on that just [45:15] because there's so many of them and it's [45:17] just like I think I need to start [45:18] learning the power of saying no by [45:20] default. [45:21] >> Yeah. Yeah. All right. Thank you both. I [45:25] will see you see you online.  📄 **Full text:** ![[Full Document Contents/Articles/Augur8-Knot WG March 17, 2026.md|Augur/8-Knot WG March 17, 2026 by YouTube]] ## Highlights - >> saw no reason not to. >> Um, >> this is effectively sort of a logging change anyway. So this isn't any >> and version. When you say version, that made me think version of augur. >> Yeah, version of augur. I the PR has actually changed from that now because I was trying to import the version number of augur from that uh metadata file, >> but it was it was causing issues because ([View Highlight](https://read.readwise.io/read/01kpb540dkjw57y2rwcz3g72dp)) - Tags: 👻 ai highlighted - >> Actually, we merged we merged a couple things. We merged quite a few things on March 10th. >> Yeah, that that would be the maintainers meeting. So, >> and then March 4th was the one before that which does line which lines up with the previous spike. So, I I I do sus I suspect it's I I wonder if that could just be I mean I imagine some amount of it is AI scrapers. I some of it is probably also just people like who have Auger cloned are like updating their clones which I guess is a good sign that people are keeping their instances up to date. ([View Highlight](https://read.readwise.io/read/01kpb540fbkcpsm57zpcvrj7x5)) - Tags: 👻 ai highlighted - >> you it's just in the agenda if you want to if you wanted to pull that up. Um >> but it's basically just um so we recently we're working on currently is reenabling the heat there are some heat map diagrams to kind of like which files have been touched at what times sort of in a in a grid view to help help you kind of see is a particular group of files or a particular subset of the repo like missing a maintainer and not been touched in a long time. I.e. I imagine you'd see something like jump start in ([View Highlight](https://read.readwise.io/read/01kpb540h80pms03jehm78es8p)) - Tags: 👻 ai highlighted - our repository to that because that folder hasn't been touched in two years. Um, but it's just kind of another health metric. I think we disabled it because the queries were taking a really long time to run and we recently I think since we disabled it. Um, we we there's been other improvements to augur like ignoring being able to ignore repos. And so it's we're we're slowly bringing back we're trying to maintain we're trying to bring that back into 8 and re like reenable the queries move them into materialized views things like that. So there, so the first point with this is ([View Highlight](https://read.readwise.io/read/01kpb540mvxpn6wykfspvdr4bz)) - Tags: 👻 ai highlighted - that there's now that we have some additional materialized to use to bring into auger upstream. >> Um, >> yeah, >> I'm not sure. I don't know >> especially if they're in especially if eight's expecting them. >> Yeah. So the problem with that is though I don't think our current Olympic like database migration config is able to do uh materialized views. I think a lot of our materialized views have just existed forever and that like they're in some of ([View Highlight](https://read.readwise.io/read/01kpb540rycfzbvfz02zeg15z7)) - Tags: 👻 ai highlighted - Yeah, I mean that that's less of a problem for 8 because we we just regenerate all the materialized views when we clone the DB, but that's that's a very not typical setup because we're operating in such a scale that we can't have augur collection and 8 not running off the same DB or at least that was when we were behind on collection. So I think now it might be better now that we're caught we're relatively caught up. But >> yeah, >> I I'd like to make some more performance fixes before we try and remerge those ([View Highlight](https://read.readwise.io/read/01kpb540v0ss6edeg6zm6dbt7a)) - Tags: 👻 ai highlighted - now. Just >> Yeah, >> it might make sense if the two projects were consistent as I think about that. So yeah, I mean I think if if it gets into dev if there would be a way that you could at least I don't know I think we should put something into augur to indicate that even if we don't have it automated it would be helpful to know that you have to create that materialized view or those materialized views whatever they were and have a script for it. Yeah, I know. I know. ([View Highlight](https://read.readwise.io/read/01kpb54151xhfeyy66k668kv57)) - Tags: 👻 ai highlighted