Augur8 Knot WG Meeting, Jan 20, 2026 - CourtneyR.dev's 2nd Brain

Augur/8 Knot WG Meeting, Jan 20, 2026 ![rw-book-cover](https://img.youtube.com/vi/X_p_v8HahQk/maxresdefault.jpg) 🔗 **Source:** [Augur/8 Knot WG Meeting, Jan 20, 2026](https://www.youtube.com/watch?v=X_p_v8HahQk&pp=0gcJCdMKAYcqIYzv) > [!note]+ Document Notes > tags: #augur #chaos-engineering #celery #data-collection #debugging > author: Augur/8 Knot WG > source: YouTube > reference: https://www.youtube.com/watch?v=X_p_v8HahQk > date created: 2026-01-20 16:45 > > # 🏷️ Title: Augur/8 Knot WG Meeting - Debugging Secondary Collection Issues Before Chaos Con >  ## 🧰 Action Inbox - [H1] Deploy Andrew's workaround for celery chaining bug — test before maintainers call #celery #debugging - [H2] Document release process for redundancy — create issue after Chaos Con #documentation #process - [H3] Query other instances to check secondary collection status — community validation #data-collection ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Augur Repository](https://github.com/chaoss/augur) - [Celery Documentation](https://docs.celeryproject.org/) ## 📝 Summary - **What happened**: Augur team debugging critical secondary collection failure where pull request review comments task never schedules - **Why it matters**: Blocking all secondary data collection since December, threatening release readiness for Chaos Con - **Key stakeholders**: Augur maintainers (Adrian, Callie, Sean), community instances - **Risks and tradeoffs**: Workaround maintains data integrity but doesn't fix root celery chaining bug - **Bottom line**: Team identified celery bug in nested chains within groups, testing Andrew's workaround before maintainers call ## 📖 Main Content The Augur team is experiencing a critical bug where secondary data collection has been failing since December 2026. The issue stems from a celery chaining problem where the "collect pull request review comments" task never gets scheduled, even though all other secondary collection tasks complete successfully. This appears to be related to known celery bugs when nesting chains within groups of tasks. The team has identified that their Red Hat instance is the only one reporting this issue, making debugging more complex. Adrian tested an initial workaround that didn't maintain proper task ordering, prompting Andrew to suggest embedding the comments function call directly within the pull request reviews task to ensure proper sequencing. The urgency is heightened by the approaching Chaos Con deadline, where the team needs a stable release. Core and facade collection are working well, but secondary collection is completely blocked by this scheduling issue. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/X_p_v8HahQk?t=0) — [H1] Meeting opening and high velocity release prep - Key idea: Light agenda due to Chaos Con preparation timeline - Evidence: "We're kind of like in a high velocity release process right now in preparation for Chaos Con" #### ⏱️ [02:30](https://youtu.be/X_p_v8HahQk?t=150) — [H1] Secondary collection blocking issue identified - Key idea: Celery chaining bug preventing pull request review comments from scheduling - Evidence: "This is pretty tendering pretty much all progress on secondary collection" #### ⏱️ [04:15](https://youtu.be/X_p_v8HahQk?t=255) — [H2] First attempted fix failed due to ordering - Key idea: Initial workaround broke task completion ordering requirements - Evidence: "It means that the scheduling is ordered not the completion is ordered" #### ⏱️ [06:45](https://youtu.be/X_p_v8HahQk?t=405) — [H1] Andrew's suggested workaround approach - Key idea: Embed comments function inside reviews task to maintain ordering - Evidence: "Putting that inside of pull request review to make it to where it has to be in order" #### ⏱️ [08:20](https://youtu.be/X_p_v8HahQk?t=500) — [H2] Core and facade collection working well - Key idea: Issue is isolated to secondary collection workflows - Evidence: "Facade and core are kind of well-oiled machines at this point" #### ⏱️ [10:30](https://youtu.be/X_p_v8HahQk?t=630) — [H2] Celery chaining efficiency rationale - Key idea: Task dependencies prevent expensive database operations - Evidence: "It's really inefficient if the app starts to insert data into the database that doesn't have parent records" #### ⏱️ [14:00](https://youtu.be/X_p_v8HahQk?t=840) — [H1] Task never scheduling vs getting stuck - Key idea: Problem is scheduling failure, not execution failure - Evidence: "They're never get like that task is never getting scheduled. It's not that they're getting stuck" #### ⏱️ [18:15](https://youtu.be/X_p_v8HahQk?t=1095) — [H2] Red Hat instance isolation puzzling - Key idea: Only Red Hat instance experiencing this issue - Evidence: "The fact that it's only us is really strange, too" #### ⏱️ [25:40](https://youtu.be/X_p_v8HahQk?t=1540) — [H2] Testing query for other instances - Key idea: Need to verify if other instances have same problem - Evidence: "I can give you a good query or know if good is the right an easy query to figure that out" #### ⏱️ [30:20](https://youtu.be/X_p_v8HahQk?t=1820) — [H3] Sean's instance showing successful collection - Key idea: Sean's instance has functioning secondary collection with proper ordering - Evidence: "I've got a bunch of them... both are going" #### ⏱️ [35:45](https://youtu.be/X_p_v8HahQk?t=2145) — [H1] Screenshot evidence of scheduling failure - Key idea: Comments task shows "received" but never "started" - Evidence: "Pull request reviews comments succeed. And this isn't even started yet. It's only received" #### ⏱️ [42:30](https://youtu.be/X_p_v8HahQk?t=2550) — [H2] Data integrity maintained in workaround - Key idea: Andrew's solution preserves ordering and data consistency - Evidence: "I think it does hold our data the data integrity is intact" #### ⏱️ [47:10](https://youtu.be/X_p_v8HahQk?t=2830) — [H3] Release process documentation needed - Key idea: Adrian requesting documented release procedures for redundancy - Evidence: "Having documentation for like what what are the steps that a maintainer should go through to do a release" #### ⏱️ [49:00](https://youtu.be/X_p_v8HahQk?t=2940) — [H3] Current release process is artisan - Key idea: Release process currently exists only in Sean's knowledge - Evidence: "It's an artisan bespoke process right now very popular in Portland but not necessarily something you should do for open source software" ## 📚 References - [Celery Chain Documentation](https://docs.celeryproject.org/en/stable/userguide/canvas.html#chains) - [Augur Secondary Collection](https://github.com/chaoss/augur) - [Flower Monitoring Tool](https://flower.readthedocs.io/) ## 🗣️ Paraphrase - The Augur team is rushing to fix a critical bug where secondary data collection has been broken since December due to a celery task scheduling issue - Their Red Hat instance is the only one experiencing this problem where pull request review comments tasks never get scheduled despite other tasks succeeding - They're testing a workaround that embeds the comments collection directly into the reviews task to maintain proper ordering while preserving data integrity ## 📰 Newsletter **Augur Tackles Critical Collection Bug Before Chaos Con** The Augur team is working through a complex debugging session as they prepare for their Chaos Con release. They've identified a celery chaining bug that's been blocking secondary data collection since December, specifically preventing pull request review comments from being scheduled. What makes this particularly challenging is that only their Red Hat instance is experiencing this issue, while other community instances continue working normally. The team is testing a pragmatic workaround suggested by Andrew that maintains data integrity while they investigate the root cause. This highlights the importance of having multiple test instances and community feedback in identifying edge cases that might not surface in smaller deployments. Resources: [Augur Repository](https://github.com/chaoss/augur), [Celery Documentation](https://docs.celeryproject.org/), [CHAOSS Community](https://chaoss.community/) ## ✅ Action Items #### DevRel - Community management: Create query for other instances to check secondary collection status - Education: Document troubleshooting steps for celery chaining issues #### OSPO - Software Inventory: Track celery version dependencies across instances - Developer Support: Create debugging runbooks for collection failures #### Community - Contributors: Engage community to validate collection status across instances - Maintainers: Document release process for better redundancy ## 🧪 Hypotheses - The celery chaining bug is specific to certain repository sizes or complexity patterns - Different celery versions across instances may explain the isolation to Red Hat's deployment - The issue emerged from a dependency upgrade that occurred around December 2026 - Nested celery chains within groups have known stability issues in production environments ### 📊 Methodology - Query multiple community instances using the secondary collection status query - Compare celery versions and configuration across working vs failing instances - Test Andrew's workaround on isolated repositories before full deployment - Monitor flower dashboard for scheduling patterns after implementing fix ### ❓ Questions - Why is only the Red Hat instance experiencing this celery chaining issue? - What changed in December that could have triggered this scheduling failure? - Should the embedded function approach become the permanent solution or just a temporary workaround? ### 🔮 Future Research - Investigate celery version compatibility with nested chain patterns - Develop automated testing for task scheduling dependencies - Create monitoring alerts for collection pipeline failures ### 💡 Implications - Practice: Need better monitoring and alerting for collection pipeline health - Policy: Require documentation of release processes for all critical projects - Security: Collection failures could create blind spots in repository analysis - Education: Teams need training on celery debugging and task dependency patterns ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Flower dashboard screenshots, task flow diagrams - Improve: Celery chain architecture diagrams, task dependency visualizations #### 🔊 Aural learners - Present: Meeting recording with detailed discussion - Improve: Audio walkthrough of debugging process #### 🗣️ Verbal learners - Present: Technical terminology explained in context - Improve: Glossary of celery and augur-specific terms #### 🧩 Kinesthetic learners - Present: Hands-on debugging session, query testing - Improve: Step-by-step troubleshooting checklist #### 🧠 Logical learners - Present: Clear cause-and-effect analysis of scheduling failure - Improve: Decision tree for debugging collection issues #### 🧑‍🤝‍🧑 Social learners - Present: Collaborative problem-solving session - Improve: Community validation process for fixes #### 🧘‍♀️ Solitary learners - Present: Individual analysis of logs and monitoring data - Improve: Self-guided debugging documentation #### 🌿 Naturalistic learners - Present: Pattern recognition across different instances - Improve: Ecosystem thinking about dependency relationships ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Well-structured meeting with clear speaker identification - Improve: Transcript formatting with semantic markup #### Keyboard navigation - Present: Standard video controls accessible - Improve: Timestamped navigation for key topics #### Contrast and readability - Present: Clear audio quality for speech recognition - Improve: High-contrast transcript formatting #### Text resizing and reflow - Present: Transcript scales appropriately - Improve: Mobile-friendly timestamp navigation #### Media and captions - Present: Full transcript available - Improve: Time-synced captions, speaker identification #### Forms and interactions - Present: Standard video interface - Improve: Accessible bookmark and note-taking tools ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global open source community considerations - Improve: Multiple timezone accommodation for debugging sessions #### Inclusive language - Present: Technical terms explained clearly - Improve: Avoid insider jargon, provide context for newcomers #### Neurodiversity support - Present: Clear structure with defined agenda items - Improve: Written summary of action items, visual progress tracking #### Representation - Present: Collaborative decision-making across team members - Improve: Ensure diverse perspectives in technical discussions #### Psychological safety - Present: Open discussion of uncertainties and knowledge gaps - Improve: Explicit encouragement for questions from community members ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear logic from celery chaining principles to scheduling behavior - Inductive strength: Pattern recognition from similar celery issues in community - Causal reasoning: Systematic elimination of potential causes for scheduling failure - Analogical strength: Comparison with working instances to isolate variables #### Fallacies - False cause: Assuming timing correlation equals causation without deeper investigation - Hasty generalization: Concluding Red Hat instance is unique without broader community survey #### Tone of voice - Overall tone: Collaborative problem-solving with technical precision and appropriate urgency ### 🛡️ DARVO Analysis - Deny: No defensive patterns observed - Attack: No attacks on reporting or investigation - Reverse victim/offender: No victim-blaming present - Emotional cues: Professional frustration with technical challenges, appropriate for debugging context ## 🎯 Conclusion The Augur team is methodically addressing a critical celery chaining bug that threatens their Chaos Con release timeline. While they've identified a viable workaround that preserves data integrity, the isolation of this issue to a single instance suggests deeper investigation is needed. The collaborative debugging approach and community validation strategy demonstrate good open source practices for handling complex technical issues under time pressure. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad Purpose: Stage CEE units that become Literature notes. Split level: H3 ### [H1] Celery chaining creates task execution dependencies > [!quote] [Augur/8 Knot WG Meeting](https://youtu.be/X_p_v8HahQk) by Augur Team via YouTube. Published 2026-01-20. Accessed 2026-01-20. > "The chaining exists because it's really inefficient if the app starts to insert data into the database that doesn't have parent records." Technical rationale for why task ordering matters in data collection pipelines. [view highlight](https://youtu.be/X_p_v8HahQk?t=630#H1) #xref #articleseed ID:rw_augur_chaining ### [H2] Nested celery structures cause known scheduling bugs > [!quote] [Augur/8 Knot WG Meeting](https://youtu.be/X_p_v8HahQk) > "There's some known celery bugs where those kinds of structures don't work like those nested chains within groups." Evidence of celery limitations with complex task dependency structures. [view highlight](https://youtu.be/X_p_v8HahQk?t=840#H2) #xref ID:rw_celery_bugs --- ### 🔖 Atomic Split Pad (Zettel Candidates) Purpose: Promote reusable claims into Atomic notes. Split level: H4 #### [Z1] Task scheduling order differs from completion order - Scope: Celery chains ensure receive order but not completion sequencing - Pointer --- ## Full Transcript [00:01] Hello, welcome to the Auger 8 not auger [00:04] meeting for January 20th, 2026 [00:08] and um we have a very light agenda today [00:11] but open to other things. Welcome to [00:14] Johnson who's joined us and um Cali [00:18] Adrian um the one thing that Adrian had [00:22] thought of before Chaos Con and that's [00:24] we're kind of like in a high velocity [00:27] release process right now in preparation [00:30] for Chaos Con. [00:34] Um, [00:36] so that's what we're kind of we've got [00:39] kind of a light agenda, which is pretty [00:41] typical, and we're trying to get a [00:42] release out, which is also pretty [00:44] typical. Um, so Johnson, before we kind [00:48] of go on with our agenda and the poll [00:50] request reviews that we were pl we were [00:53] going to do here, do you have anything [00:55] that that you want to bring up? [01:06] Not really. Not really. All right. No, [01:12] >> maybe I'll have some questions at the [01:13] end, but um not for now. [01:15] >> All right, that sounds good. [01:26] So, right now I am staring at [01:31] the startup screen to see if Adrian's [01:34] branch works. [01:37] >> Yeah. Adrian, have you What's the state [01:39] of our like because we Yeah, I think [01:40] there's a lot of We're all like testing [01:42] different things actively right now. [01:46] Yeah, I I have yet to [01:48] um undo the state of that one repo. Um [01:53] >> Okay. [01:54] >> But I think the the I have an image that [01:56] should be built for the new the second [01:58] test. So I'll do that after this call. [02:00] Um I've been getting a little distracted [02:03] because I think um our department's [02:05] trying to put together a promo video and [02:06] I've been asked to record some people in [02:07] the Boston office. So I'm like [02:10] >> Slack messages and telling people to [02:12] throw it on my calendar and I'll deal [02:13] with it later. Yeah, that makes sense. [02:16] Um, [02:18] for some context, um, I it's probably [02:21] worth like more of a conversation in [02:23] like the maintainers call of like how do [02:25] we want to handle this long term with [02:28] the issue that comes from chaining. It [02:31] doesn't seem like there was that the bug [02:33] was from the augur side, but more of a [02:37] celery issue. Um, either way, need to [02:41] find a way around it. Um the first [02:44] attempt that Adrian tried out yesterday [02:48] um once I saw it this morning I was like [02:50] oh I get why this doesn't work. Um and [02:53] so by taking it out of the chain even [02:55] though it was ordered um it means that [02:57] the scheduling is ordered not the [03:00] completion is ordered. Um so wasn't a [03:04] good solution for the problem at hand. [03:07] Um, but Andrew had suggested just [03:10] putting the pull request comments [03:12] function inside of the pull or it's pull [03:16] request review comments and putting that [03:18] inside of pull request review um to make [03:23] it to where it has to be in order, but [03:24] it's in the same um seems like a pretty [03:28] good workaround, but probably is worth a [03:30] discussion of how to actually fix the [03:32] issue instead of patching the issue. Um, [03:35] but we're looking for a patch for right [03:37] now for the very least that we can keep [03:39] collection moving. Um, because this is [03:41] pretty [03:43] tendering pretty much all progress on [03:46] secondary collection. So, that's kind of [03:50] what we're working on on the Red Hat [03:54] side of house. Um, because for us, um, [03:58] facade and core are kind of welloiled [04:01] machines at this point. I mean, we still [04:03] have there's some smaller errors that we [04:06] want to work on, but um nothing like a [04:10] major hindrance like we are with [04:12] secondary right now. Um [04:17] yeah. [04:21] Do we want to look at that issue, [04:22] Adrian? like um I don't have a I haven't [04:26] up you know since we talked yesterday [04:27] afternoon I haven't gotten myself up to [04:29] speed with it entirely but is there [04:31] something that um you think we can look [04:33] at [04:34] >> uh with our secondary things? [04:36] >> Yeah. Um, I think it's mostly I'm I'm [04:39] mostly I think it's mostly dependent on [04:41] what our results of our second. So, the [04:43] first test was just me tinkering around [04:45] with sort of the ordering of the celery [04:48] sort of workflow um to try and see if [04:52] that worked and I don't it didn't seem [04:53] like that worked. Um, so I I tried [04:57] there's a suggestion from Andrew that he [04:59] made on Discord yesterday that I was [05:01] like, "Hey, that that's actually a that [05:03] that it helps increase our road map [05:05] because I didn't I didn't have any other [05:06] ideas after that first one." So, we're [05:08] trying that one now. I'll probably be [05:10] deploying that after this call and we'll [05:11] we'll probably know before the [05:12] maintainers call whether or not that [05:14] worked. Um, that might be a better [05:16] conversation like if we want to deep [05:18] dive into that, that might be better for [05:19] the maintainers call cuz we we'll [05:20] probably have like Andrew and some other [05:22] people who pretty much wrote the [05:24] collection process like there to help. [05:27] Yeah, I mean I c I certainly know that [05:28] the chaining is one of the strategies [05:30] that we used to insert to make the [05:33] dependencies between the tasks be [05:36] handled by the app cuz it's really [05:38] inefficient if the app starts to insert [05:42] data into the database that doesn't have [05:45] parent records because that basically [05:47] just offloads the sequence of collection [05:50] to um [05:53] it offloads the sequence of collection [05:54] to the database which gets extremely [05:56] expensive on a server. So that's why the [05:59] chaining exists. I don't doubt there is [06:01] some kind of celery chaining issue. Um [06:05] >> yeah, I'm not I'm not questioning why [06:06] they have to be run in a particular or [06:08] like I was I I was just asking because I [06:09] I didn't 100% know, but I I I understand [06:12] that they have to be in this done in the [06:13] right order. Um it's just I think the [06:16] fact that we had that chain nested [06:18] within a group of other tasks in Celery, [06:20] I think there's some known celery bugs [06:22] at least for based on a quick search. um [06:25] where like th those kinds of structures [06:28] don't s like I guess there's some salary [06:30] bugs with that and we were seeing that [06:32] the the pull request reviews like the [06:35] first part of that chain was running and [06:37] getting scheduled and completing but we [06:39] were not seeing anything even scheduled [06:40] for the second one. Um and so it's like [06:44] what what's going on here? Why is this [06:46] not scheduling? Because if the second [06:48] piece of that change is not scheduling, [06:49] that whole group is probably not [06:51] completing or that whole that whole line [06:53] is probably not completing and then the [06:54] whole group is not going and then it's [06:56] not going to get marked as success. So, [06:58] it's just kind of like I I do need I do [07:01] probably need a lot more context on like [07:02] how how scheduling works and how how all [07:05] that's kind of supposed to work in terms [07:07] of because I I know I think Callie's [07:10] been trying to look at Flower and see [07:12] like whether tasks are scheduled and [07:14] whether they're completing and I I think [07:16] I remember some conversations a while [07:17] ago about how we may be only scheduling [07:22] the number of tasks that like can be [07:23] processed right like by the number of [07:25] workers we have or something. So, we're [07:26] maybe not Those maybe aren't being [07:29] scheduled, but then maybe I I don't [07:31] there might be some kind of weird stuff [07:33] going on there that we could we could [07:34] look into, but that might be better for [07:36] the maintainers call. [07:38] >> Okay, that sounds good. [07:43] I don't have anything. [07:46] >> Um I did see your message on Slack. Do [07:48] you want to quickly jump back to your So [07:50] that that issue you were testing. So it [07:52] it seems like you said it worked, but [07:53] you ran into a different error. Um [07:55] you're [07:56] >> Yeah, but I don't think the error I [07:57] think the error is just like um [08:00] unrelated to anything. [08:02] >> Yeah. So that particular error when one [08:05] of the routes is crashing because the [08:07] like request reports isn't um like a [08:11] function doesn't exist. Um one of the [08:13] one of the commits or one of the PRs [08:14] we've gotten recently resolves that. Um [08:16] I think I I preemptively just sort of [08:18] added that to the current milestone so [08:20] you can find it easily if you want to [08:22] test that and review that. But [08:26] yeah. [08:26] >> All right, that sounds it's added to the [08:28] milestone now. [08:32] >> Um, [08:34] >> all right. So, this worked. I'm going to [08:55] Sorry. Something stupid that I was [08:57] doing. [09:02] Okay, now I have to fix that. [09:08] Sorry. [09:40] What was that? PR name. [09:42] >> Uh, the one with your issue 3516 I think [09:46] is the number. [10:00] They screwed up the drainage. [10:03] >> I can I can pull it low. I can pull it [10:05] on mine. And [10:06] >> yeah, pull it on yours. And [10:08] >> you're just trying to remove the commits [10:09] for that one extra file. [10:10] >> I'm trying to remove the commits for the [10:11] extra file, which was basically me just [10:13] making sure I had my GitHub configured, [10:15] which was stupid, but [10:18] that's where we are. [10:23] Okay, I'll just force push that branch [10:24] with the [10:26] previous version. [10:29] >> Meanwhile, it does look like things are [10:31] running [10:32] and I didn't get any errors. So, you [10:35] might think it's definitely [10:38] definitely fixes the bare metal issue. [10:42] Sweet. [10:45] I guess feel free to merge it. I don't I [10:48] don't think this [10:50] should affect I mean it it's yeah this [10:52] should this should be fine um if it I [10:55] mean it was passing all the tests I [10:56] think before so [10:58] >> hey I I it says I was requested for [11:01] review but I did the review I thought [11:05] >> it might have it might have um not [11:08] submitting it would help but um [11:10] >> I think also sometimes if if you force [11:12] push it will dismiss all the approving [11:15] reviews or thing. Uh, do you want me to [11:18] rebase it as well or [11:23] I guess it doesn't? [11:24] >> No, it should it should be fine. This [11:35] What do I want to do here? I haven't [11:36] seen this before, but it's probably a by [11:38] >> I think GitHub's been starting to like [11:39] kind of like force that. I don't You [11:41] don't super have to. I tend to prefer [11:43] rebase. Um, [11:45] >> all right. But [11:48] >> yeah, [11:51] >> but also if you click rebase, it's going [11:53] to have to rerun all the tests. So if [11:55] you're waiting for the tests to pass, [11:57] then [11:58] >> I think we're good. [12:01] They passed already. [12:09] >> Wait. [12:13] Um, and then what what was I I wasn't [12:16] sure how much I picked up of this from [12:17] from the various maintainers calls, but [12:18] like what are your kind of criteria for [12:20] when to release? Like it sounds like [12:22] you're kind of wanting the obviously to [12:23] be able to [12:24] >> bring up the main have it not have any [12:26] major issues. [12:29] >> Not issues is important. [12:35] Stability is important. So like all the [12:38] all the CR passes [12:41] uh CI passes [12:44] there are there's nothing on our road [12:46] map for that release that's missing that [12:49] we think we need and I think it sounds [12:52] like right now we do want to possibly [12:55] try to solve this issue with secondary [12:57] collection before we release. like it [13:00] seems like that falls under the category [13:02] of something that's really important to [13:04] fix that that we should also release [13:08] that happening on other people's [13:09] instances. [13:11] >> Um I I don't see how it wouldn't, but we [13:14] are the only instance that's reporting [13:16] this problem. [13:24] Yeah, because I also noticed I think [13:25] when I was trying the initial version, [13:27] >> what would you evaluate if I was [13:29] experiencing the problem? [13:31] >> Um, I can give you a good query or know [13:35] if good is the right an easy query to [13:38] figure that out. I'll put this [13:40] >> huge fan huge fan of easy. [13:43] >> Um because if you look at I'm gonna put [13:46] this in the thread that's about this [13:48] issue. Where is it? Um give me just a [13:54] second. [14:04] Um, I just send it the channel because [14:06] then you can see how long it's been [14:08] since you've had a You have to go to the [14:11] bottom of all your NLES, but the first [14:13] it orders your um like collection by the [14:18] um last collected on date for secondary. [14:21] And so you can kind of see if how long [14:25] has it been since you've had successful [14:29] secondary collection. Somewhere in [14:32] December we stopped having successful [14:36] secondary collection and I don't know [14:39] why. [14:41] That's also a big portion of it is like [14:42] why did this like happen? Was it a [14:46] upgrade? Did we upgrade some packages [14:49] that ended up like [14:51] unknowingly impacting this? [14:54] Um maybe were the ones that were [14:57] passmber. [14:58] >> What [14:59] >> did you say? September. [15:00] >> December. [15:01] >> You said December. [15:02] >> December. Okay. [15:05] Um yeah because the last time that we [15:07] have seen [15:10] any successful collection [15:14] um for secondary has been in December [15:17] but mostly [15:19] my theory is that it might be [15:20] repositories that don't have any PRs and [15:23] so it never even gets and so there might [15:26] be something that's like oh no PR [15:28] reviews and so you don't need to go to [15:31] PR commits. I'm not that's a that is a [15:35] completely uninvestigated theory. [15:38] >> So yeah, let me show you what I'm [15:41] seeing. So I've got nulls first. That's [15:44] the default behavior. [15:46] >> So um pardon me. This is kind of a a [15:49] wide query result, but the secondary [15:53] data last collected in this column. [15:56] Obviously, I'm getting secondary data. [16:02] That fact makes me ask the question [16:05] about whether or not the things that [16:08] your instance may be stuck on are in [16:10] fact very large repos [16:12] if [16:12] >> but they're not getting scheduled. Like [16:14] it's like they're never get like that [16:16] task is never getting scheduled. It's [16:18] not that they're getting stuck. All of [16:20] the other four tasks that are related to [16:22] secondary are completing are getting [16:25] scheduled and succeeding [16:28] and the like pull request review comment [16:33] is never being seen by our instance. And [16:37] so it'd be interesting to see if your [16:39] instance is scheduling if you can go to [16:42] flower. Um I was curious to see if you [16:48] control F [16:50] >> on um [16:55] I'll put this just in. [17:01] >> All right. I'm in I'm in flower now. [17:04] >> Just a collect. I just put it in persist [17:06] like I'm I'm just putting these in [17:07] Slack. So then you can see like they're [17:09] there. um [17:16] just to see if that task is ever [17:18] scheduling. It's the very last thing. I [17:20] already put it down there. [17:21] >> Collect pull request review comments. [17:23] >> Okay, hold on. Sorry. [17:25] >> Yeah, if you just search for that in the [17:26] tasks tab of [17:28] >> copy and paste. [17:29] >> Uh okay. [17:33] >> Oops. [17:35] >> Select. Yeah, [17:36] >> if I typo it, it's really nasty. [17:39] review [17:41] >> comments. Yeah, [17:43] >> I've got a bunch of them. [17:44] >> There's a lot that looks like there. Um [17:48] do like um do like secondary task. I'm [17:51] curious if you're seeing secondary [17:53] success. Um [17:55] >> I would assume he is. [17:57] >> Yeah. [17:57] >> Yeah. [18:01] >> So, I mean that I haven't looked into [18:02] this. Obviously the smell is that [18:05] there's some kind of scheduling issue on [18:07] secondary but so is any nothing is [18:10] collecting at all though. [18:11] >> No everything but that everything but [18:15] >> the comment task that you searched for [18:17] is the thing that as far as I can tell [18:19] seems to be blocking the success task [18:21] from running [18:22] >> because it seems like in your case both [18:23] are going. [18:24] >> Yeah. So ours are never even scheduling. [18:27] So it doesn't even give our instances [18:29] shot to have a sex a successful [18:34] um secondary collection because if it [18:35] never gets scheduled it can never [18:37] succeed. [18:37] >> Yeah. Yeah. So you're just you're saying [18:40] if I'm hearing correctly pull request [18:41] review comments never I don't if I made [18:44] the same query on your flower dashboard [18:46] I would see no successful completed pull [18:48] request review comment tasks in my [18:49] >> see nothing. It would just have a blank [18:51] response. [18:52] >> Nothing. [18:53] >> None would be none would be pending. [18:55] None would be like about to run just [18:58] nothing at all. [18:59] >> Yeah. [19:00] >> But other secondary tasks are running. [19:02] >> All of them. All the rest of them are [19:04] running [19:06] and succeeding. [19:09] >> How big are so [19:12] how big are the repos? So pull request [19:14] review comments is never even starting [19:16] or never succeeding. [19:17] >> It is or oh comments is never being [19:19] started. It is never being scheduled. So [19:21] it doesn't have anything to do with [19:23] size. the size this the size does not [19:25] matter if it never even gets it doesn't [19:28] get scheduled. [19:30] >> What's the task before what what's the [19:32] last task that you can see in the chain [19:34] that's scheduled Adrian? [19:36] >> I mean so all of the [19:38] >> right before the pull requests like the [19:39] pull requ the one that goes right before [19:41] the comments one is we're seeing that [19:43] succeeding we're just not seeing [19:45] >> so pull request reviews is succeeding. [19:48] >> Yes. [19:48] >> Yes. But then for some reason the the [19:50] task that's changed immediately after it [19:52] is somehow not running. That's sort of [19:54] what leads me to think that this is pro [19:56] I feel like from my just super quick [19:59] cursory search of open celery issues [20:01] because my various AI assistants were [20:04] saying like this is a common issue in [20:05] celery and I I went and looked at their [20:06] issues and and checked and it looks like [20:08] there are quite a few issues in celery [20:11] related to just various flavors of like [20:14] nesting things in other nesting like [20:17] chains inside groups or something like [20:19] that. Um, so I suspect that that could [20:23] be a very plausible cause. [20:26] >> Um, yeah, [20:28] >> the fact that it's only us is really [20:30] strange, too. I'm not sure what the heck [20:31] is happening there. [20:32] >> The fact that it's quote unquote really [20:33] you also is is a factor that just has me [20:39] has me [20:39] >> also you've got a bigger instance than [20:41] us. So like it's not even like this was [20:44] like some of my some of the tests like [20:45] the test cases we were running yesterday [20:47] were running f like I was seeing these [20:49] successes on my local my tiny local dev [20:53] instance [20:55] our prod is not happy it's not working a [20:59] like what you ran locally though yet I [21:04] yes it showed successes but that's like [21:07] that that was not a correct solution [21:11] Um because the way that you implemented [21:15] it yesterday did not require [21:18] um pull request reviews to finish before [21:20] pull request review comments. [21:23] >> It should have that that's [21:25] >> No, it didn't like because you you don't [21:27] have any training just it it requires it [21:29] to be scheduled beforehand, not that it [21:32] has to finish beforehand. [21:35] So it shows the scheduling order, but it [21:38] allowed for that pull request review [21:40] comments to start while pull request [21:43] reviews was still going. But for smaller [21:46] instances with a very for a very small [21:50] repo, um you saw the successes, but it [21:53] didn't but it didn't um require the [21:56] ordering to keep to be correct. That's [21:58] why we had the [21:59] >> I think that's wrong. the the the docs [22:02] I've been looking at for celery chains [22:04] say that that that it's chains and [22:06] celery allow you to link tasks together. [22:08] The linked task is called when the task [22:11] returned successfully. [22:12] >> You took away the chain. [22:14] >> No, I took the chain and I wrapped I [22:16] instead of having the chain inside the [22:18] group the chain was surrounding the [22:21] group. [22:23] >> I guess I can say in practicality on our [22:26] instance that's not what happened. [22:29] >> Yeah. I yeah I I know that but I also [22:32] our instance is the one anomaly in [22:35] >> I would be curious to see on your is [22:37] your instance still running [22:38] >> is your is your dashboard public? [22:42] >> Um no we don't put anything public. [22:44] >> Um you don't put any access to our [22:46] repository but you do have access to it. [22:48] Um, I'd be curious, Adrian, to look at [22:50] the finish times or the like to see if [22:54] the start time for pull request review [22:57] comments on your instance is [23:02] a actually after the finish time for the [23:05] related pull request reviews because if [23:08] it is then I'm like I'm happy to be [23:10] wrong but then that I'm happy to be [23:12] wrong [23:14] makes it more interesting that it's a [23:16] problem on our instance. I would be [23:18] curious to see what if we're running a [23:19] different celery version or something [23:22] like that, but the screenshot that I saw [23:26] from your instance showed them all to be [23:27] successfully colleed, but I didn't I [23:30] can't see if they the ordering was held [23:34] to the way that it's supposed to. [23:38] >> I can I'll I'm looking into that. Uh [23:41] collect poll request review. Okay, [23:44] so collect poll request reviews. [23:46] received [23:49] 1926 started 1926 succeeded 1926 this [23:53] was all yesterday UTC [23:56] and then the poll request review [23:58] comments it's all 1927 [24:02] for this is just for chaos auger repo I [24:04] can screen share if that helps but [24:06] >> yeah I just would look at each one [24:08] because it's like it's a harder thing [24:10] because if it's super quick I don't know [24:12] if it's going to end like yeah let's [24:14] look at each of the example examples [24:20] Um [24:21] because I believe you on that one. [24:23] >> So this is this is collect pull request [24:25] reviews for chaos augur. [24:28] Um everything's basically 1926 UTC. [24:33] Um so it basically started and finished [24:36] within 30 seconds. [24:38] >> Yeah. And so that's why I'm concerned is [24:40] that [24:40] >> like pull request review comments. Well, [24:42] I think this uh [24:45] >> because my thing is that it's I'm saying [24:47] is that for something that is more than [24:48] 30 like the in these longer [24:50] repositories, [24:52] it's requiring that it's like it the [24:54] scheduling order I agree with that's [24:58] happening but it's I think that it's [25:01] allowing it to start or at least we [25:04] could say on our instance it's allowing [25:05] the scheduling order the receive order [25:08] is correct but it is allowing the pull [25:12] pull request review comments to start [25:15] while pull request reviews is still [25:17] going. And so with these shorter like [25:20] with the 30 seconds ones um that it's [25:24] holding up um which is anticipated. [25:28] That's why I was going to say let's try [25:30] to go through a couple different [25:31] examples to see if that's holding for [25:34] all of them. But if you only have really [25:35] small repositories, even if this is a [25:37] problem, I don't know if it's going to [25:39] be shown. Um what's up Sean? [25:42] >> Do do we know that it's running this the [25:45] for the same repository? So is it [25:47] collect do we know that it's collecting [25:48] reviews and review comments for the same [25:51] repository at the same time? for the one [25:54] instance on our so we have we tried to [25:56] run there was a patch that Adrian tried [25:59] yesterday [26:00] um and we only saw one pull request [26:04] review comments get scheduled and that [26:07] one got scheduled and was run and [26:10] completed before its related pull [26:13] request reviews was finished. [26:23] >> All right. So, we have we have one that [26:25] >> any of these repositories sufficiently [26:27] large? I sort of [26:28] >> I don't I would just say let's just look [26:30] at like look at each one and like look [26:32] at some reposi I I don't it's not like [26:34] I'm not going to be able to like look at [26:35] the names of these random repositories [26:38] and know they're [26:39] >> the USD might be on the larger side. So [26:42] that's [26:50] >> you can show more tasks too at the top. [26:52] You can list a 100. [26:54] >> I was I saw it though. So this is uh Z [26:56] Galaxy B USD. [26:58] >> That's comment. It's pull request. It's [27:00] collect pull request. That's not what we [27:01] want. [27:02] >> Well, it was like 1410 and then this one [27:05] was [27:05] >> that's not reviews. [27:06] >> That was We want pull request reviews. [27:08] >> Reviews reviews. Pull request. Yeah. [27:11] Pull request reviews and then pull [27:12] requests. [27:16] >> Just show more tasks. Yeah. [27:18] >> Yeah. Oh, collect. Hey, there we go. [27:23] So, that's not the right one. Okay. [27:26] Reviews 1926. [27:31] So, that was that ended like 30 to 44, [27:34] 57 to 58. So, these are going pretty [27:37] quick. [27:38] >> Yeah. I mean, honestly [27:41] doesn't matter. We're I I don't know if [27:44] it matters with the patch that we're [27:46] coming that we're about to use for this, [27:49] but I would say that in the larger term, [27:53] I think it's beneficial for us to un [27:55] like to go down the rabbit hole to [27:57] understand why this is happening. [28:01] >> Agreed. [28:02] >> Um that like and again I could be wrong. [28:06] I don't know. Like it's you it's [28:08] pointing to something being a little bit [28:10] more complicated that it's not happening [28:12] on Sean's instance and it's happening on [28:14] ours. Um I would love our [28:16] overenthusiastic contributors. Um if [28:20] they want to be helpful right now, it [28:22] would be great for them to and I might [28:23] try to see if people will like if we can [28:26] write this out. This would be a lot more [28:28] useful for them to contribute would be [28:30] to for them to check this [28:32] >> to see if other people's Yeah. What I [28:35] think I saw in the data though is that [28:37] it did very rapidly move through pull [28:40] request reviews and in a very brief time [28:43] period after that do comments. [28:46] >> Yes. [28:46] >> And and it only matters from the [28:48] sequence matters only because that tells [28:49] me whether or not something that we [28:51] expect to be working in celery is. It [28:54] seems like celery is working the way we [28:56] expect it to. But there is still this [28:59] problem with your secondary collection. [29:01] But I would say that the problem that [29:03] we're seeing like with this solution [29:05] would not show up in these like these [29:08] are these tasks are finishing too fast [29:11] for the error that we saw to pimp up [29:13] because they're being received in the [29:15] correct order. They are being scheduled [29:17] in the correct they're like it's the [29:19] chaining is [29:21] >> like pull request reviews is always [29:23] coming before pull request comments. The [29:27] problem is is that pull request reviews [29:29] was still actively running when pull [29:32] request reviews comments was started and [29:34] because comments doesn't have any [29:36] comment like that it's it's able to [29:38] finish so fast because the I would [29:40] assume this is this is getting onto [29:42] assumption land but there's no there's [29:45] no pull request reviews for it to gather [29:47] comments for because it hasn't collected [29:49] the reviews. So pull request reviews [29:51] comments is going to finish really [29:52] quickly and then that's how you can see [29:54] that inversion of like the the [29:57] completion is not [29:59] >> yeah going [30:00] >> I need to ask Andrew about this. I can't [30:02] remember if we did this for the poll [30:04] request review task but in the case of [30:06] of many of the tasks certainly in core [30:09] and facade now we we have it so that [30:12] it's only going to go back and collect [30:14] things that it hasn't already collected. [30:16] It's also possible that it's collected [30:19] basically all the pull request reviews [30:21] previously and in this particular case [30:23] and that's why it's jumping through that [30:25] fast and then jumping through comments [30:27] fast because there's nothing new. [30:28] >> That's that's what I suspect from my [30:30] instance. Yeah. Like [30:31] >> from your instance. Yes. From ours like [30:33] even we were looking at this like hours [30:35] later, pull request reviews was still [30:38] going. Um I don't know if I wish I would [30:40] have taken more screenshots. Um, I don't [30:43] know if I ended up actually sending any [30:45] screenshots to you. Yeah, I took one. [30:47] Um, [30:48] >> how long how long are we letting it run? [30:50] And I'm only asking that because if [30:53] these are larger repositories, [30:55] it it just might be waiting for the [30:57] finish signal. And I have seen this [31:00] before where something I can see from a [31:03] data perspective is finished, but for [31:05] some reason it takes [31:07] >> Flower like two hours to to to recognize [31:10] it's finished. [31:12] So, it's actually even worse than I [31:13] thought. I'll I'll share my screen. [31:16] >> Excellent. [31:16] >> Uh that [31:19] for you can see pull request reviews, [31:22] comments, pull requests, reviews. [31:25] You had pull request, reviews, comments, [31:27] succeed. And this isn't even started [31:29] yet. It's only received. [31:31] >> So, that's possible if you've collected [31:34] the reviews before. Like if [31:37] >> this is these are nulls. [31:40] >> There are no reviews. is there are no [31:41] pull requests. I'm saying that like [31:43] >> it sounds like it might be possible that [31:44] like if because this has been we've been [31:46] running we've been trying to hit [31:48] secondary and try and figure this out [31:49] for a while and I I I imagine like the [31:54] the database would the would the [31:55] database clear out review like so say [31:59] pull request reviews runs it collects [32:02] all the reviews stores them in the [32:03] database but secondary doesn't fully [32:07] complete and get stamped as with a new [32:08] date if the instance gets restarted is [32:12] augur going to clear out those [32:16] entries in the review finishes, you're [32:19] good. Like if if you bring it down while [32:22] the task is still actively going, it [32:24] will clear it out. But if you're if you [32:27] if that singular task does finish, it [32:30] will hold. Um but [32:32] >> sometimes it's also a byproduct of when [32:34] how frequently your instance of the [32:36] Postgress database is doing checkpoints [32:39] as well. But you're right. [32:41] >> I should be fairly [32:42] >> Callie's right. But sometimes if I have [32:44] a checkpoint run, I will get the partial [32:46] collection. [32:49] >> But Callie's point, right? So it sounds [32:52] like this is this screenshot is [32:54] potentially expected then like this is [32:57] okay behavior because if if because [32:59] we've been seeing up until now collect [33:01] pull request reviews run successfully [33:03] and then we just never were seeing [33:04] comments getting scheduled [33:06] >> and now with this patch comments got [33:07] scheduled and it ran successfully based [33:10] on the the potentially based on the [33:12] reviews that were previously collected. [33:14] possibly theory. I guess I don't agree [33:17] with that theory personally, but I would [33:19] talk to like this would be something [33:20] where it's like I would opt to Andrew [33:23] because how would you even how would [33:24] Augur know that it has the pull request [33:27] reviews if the last collected on date is [33:30] null and it hasn't even started? There's [33:33] nothing there's nothing there for it to [33:34] check. [33:35] >> Like the database of the [33:37] >> what? the pull request reviews are in [33:40] the augur data whatever relevant auger [33:42] data table it could just look through [33:44] all like grab all the reviews for that [33:47] >> if if secondary data so we'd have this [33:50] we'd have to look at the way that the [33:51] queries are being run but as a principle [33:53] the secondary data collection if it's [33:55] date of success is null augur is going [33:58] to start assuming it hasn't collected [34:00] anything if it has and that that that [34:04] log entry in the the collection status [34:06] table is wrong um augur will just pass [34:09] over the ones that it's already [34:10] collected very quickly. So that [34:13] >> that's why I'm saying that there's no [34:15] way that Augur knows, hey, this data is [34:17] null, but there's data there until it [34:19] starts the pull request review task. [34:22] >> That's right. That that has to be the [34:24] case. You're you're right that that it [34:27] >> the pull request review task is already [34:28] finished in the last the previous run, [34:31] >> but it's not [34:32] >> Augur doesn't know that. Augur in [34:34] Augur's mind this has never happened [34:36] because in Augur the date is for pull [34:39] request the um last collected on date [34:42] for secondary is null. So in Augur's [34:44] mind this the these tasks have never run [34:48] in the lifetime of the of the instance [34:51] for these repositories. [34:53] >> That's right. [34:55] Once that date has a value, then Augur's [34:58] like, "Cool, I have data up until this [35:01] date, then I start from there." But but [35:04] whenever once this task is started, then [35:07] Augur is like, "Cool, I already have all [35:09] this date. I'm not going to recollect [35:10] recollect and then it goes." But it [35:12] can't it's not going to know that until [35:14] that task starts. [35:17] Does that make sense? [35:20] I think maybe I think we can I think we [35:21] can also discuss like maybe having [35:23] Andrew around I think would probably [35:24] help because I'm [35:25] >> Yeah, for sure. [35:25] >> I'm not fully familiar with like like [35:29] celery like scheduling versus [35:31] >> I think I think what you're seeing there [35:34] is a discontinuity Adrian between what [35:36] is in the collection status table and [35:38] what's running in celery. I've run into [35:40] this myself on longunning repos before [35:42] where the secondary data last collected [35:45] never gets updated for example but the [35:48] tasks all seem to have succeeded on on [35:51] celery and flower and I so sometimes I [35:55] have observed that if I wait many days [35:58] it will eventually get marked completed [35:59] other times I've observed that it [36:02] appears when something runs for longer [36:04] than some amount of time usually at [36:06] least a week if it's if it finishes is I [36:09] actually don't ever get to see that for [36:12] some reason. So I have I've seen these [36:14] I've seen anomalies like this before [36:17] >> if not this particular one and I think I [36:20] think they do arise from the the non [36:24] deterministic nature of celery. [36:28] So I think we're looking at the right [36:29] place. [36:30] >> We were like yesterday we before we [36:32] realized that there was a task not [36:35] scheduling. We were hoping it was a [36:37] similar issue that we were seeing and I [36:38] forget whether it was core or facade [36:40] where all the tasks were running but [36:42] because of I think it was like a because [36:44] of a scheduling issue the success task [36:46] was never going. So we were going to [36:48] manually add the success like tag and [36:51] put it in the date. But because we fi we [36:53] we we figured out that no it's actually [36:56] it's not success because this task is [36:59] never running. So one thing that I'm [37:02] happy about is that for like I think and [37:06] the workaround that Andrew has given [37:10] so like it's not I don't think this is a [37:12] good long-term solution. I don't I think [37:14] we should try to figure out what's going [37:15] on. But I think that it does hold our [37:18] data the data integ in integrity is [37:21] intact and I'm hoping that we can get [37:22] some of these repositories marked as for [37:25] an initial collection and we can kind of [37:27] like start to figure out the nuances [37:29] about this a little bit more to figure [37:31] out where this gap is. Um, but even as [37:35] we go through this more, the workaround [37:38] that that Andrew put forward feels more [37:41] like it's just it's it's a solid it's [37:43] it's a solid workaround [37:47] which I don't know I don't know Sean if [37:48] you are aware of what that is. I'm kind [37:51] of just like it's pretty much [37:52] >> I I am still catching up after being out [37:55] for 10 days so forgive me. I I don't [37:58] know. I was going to say it pretty [38:00] Andrew suggested to just put the pull [38:02] request review comments function call it [38:06] at the end of pull request reviews [38:09] >> that that would ensure that it gets [38:11] called [38:12] right [38:13] >> and it would it would ensure ordering [38:15] and it would ensure it gets called. Is [38:16] that the longterm and that's why I'm [38:18] like is that the long-term solution we [38:20] want here? No, that's not really what [38:23] we're trying to do. But it does solve [38:26] it. And maybe we can start to figure out [38:29] and it might be that we're just at a [38:30] really weird asynchronous point of our [38:32] like larger repository that is caused by [38:35] a nature of all these null dates that [38:38] really shouldn't that are like [38:40] arbitrarily null and they're really [38:42] large and rabbit's freaking out. Um, so [38:46] >> yeah, [38:46] >> it's gonna be interesting to bring that [38:48] to Andrew tonight, especially once we [38:50] test it and see how that goes. So, I [38:54] think I think we've like we beat the [38:56] horse very dead. Um, [38:59] >> I'm we want to Have you gotten a chance [39:01] to start it, Adrian, or we you've been [39:03] kind of [39:04] >> Not yet. I'll do that after this call. [39:06] Um, I did also have a quick question for [39:07] Sean as far as like the release process. [39:09] I know. I mean, maybe this isn't [39:10] necessarily like pre- chaoscon thing, [39:12] but like [39:13] >> I do eventually want to kind of learn [39:15] like what what your process is for [39:17] actually doing a release because it it [39:18] does [39:19] >> I feel like that would be helpful to [39:21] just have some redundancy in how many of [39:23] the core maintainers actually kind of [39:25] know the release process. I I think you [39:27] mentioned on Discord or something that [39:28] you had it doc I don't know if that was [39:31] No, I think that was that was something [39:32] else. But um [39:33] >> it was the install process that uh [39:35] >> yeah that was the install process but [39:36] like having documentation for like what [39:38] what are the steps that a maintainer [39:39] should go through to to do a release I [39:42] think would be helpful to just have so [39:43] that in case you end up like [39:45] unexpectedly busy like I could always do [39:48] a release. [39:48] >> It's an artisan bespoke process right [39:51] now very popular in Portland but not [39:53] necessarily uh something you should do [39:54] for open source software. Um so yeah we [39:58] should write that down. Um, I think [40:00] you're seeing a lot of the critical [40:02] signals which are obviously the CI has [40:05] to all work because I trust that now. I [40:07] didn't used to trust that that was [40:09] finding defects and really the bare [40:12] metal smoke test is the next thing. So [40:15] like that's almost constant where I'm [40:17] constantly running a bare metal smoke [40:18] test. Um, and I just update it whenever [40:21] we have a candidate for release. Uh, I [40:24] am almost always run trying to run off [40:26] of main which is kind of our dev branch. [40:29] uh so that I am not waiting until the [40:31] end to see what's working. So I have a [40:33] test instance working on that. But yes, [40:35] we should write these things down. I [40:36] agree. Um just to give you an idea [40:39] though, it's I don't want you to think [40:40] it's like some mysterious complicated [40:41] process. It's I think you've seen most [40:44] of the parts happen. You just haven't [40:46] seen the artisan bespoke checklist. Um [40:50] that uh just is in my mind, but it needs [40:53] to be elsewhere. So [40:56] >> yeah, if you if you get time at some [40:58] point, I mean, it might have to be after [40:59] Chaos Con at this point, but like [41:01] >> feel free to just if you if you happen [41:03] to write any of that down, like even [41:05] just throw it in an issue and I can I [41:06] can kind of take it from there. But [41:08] >> yeah, I think I think what I'll do is [41:10] while we're working on it tonight, I'll [41:12] just make a note of all of the things [41:13] I'm doing and that will be a a pretty [41:16] good start at that. [41:19] >> Yeah. If y'all I don't know why I [41:22] potentially would be if y'all want any [41:25] of my like point of view on this. I'm [41:28] can't like I'm free from 6:30 to 7. Um [41:31] but I don't know if I'm providing any [41:33] new perspective that at this point y'all [41:36] the two of y'all don't have. Um [41:39] I [41:39] >> I'll let Adrian make that call. I I I [41:43] don't know. I I [41:45] >> I think Cali I think the last invite I [41:47] sent you for for last week's meeting um [41:49] I think I made it like a recurring [41:51] invite so you're you're free to drop [41:52] into the maintainers calls. Um I don't I [41:54] think as far [41:56] >> saying do if you need it is a if you if [42:00] my perspective is helpful I will come [42:03] bring prioritize the thing that you [42:04] would like me to have put in a voice [42:06] within the third first 30 minutes. If [42:08] you were like, eh, either way, [42:11] >> we would never tell you that your [42:12] perspect we would never tell you that [42:13] your perspective is not helpful. [42:16] I think I think it's a question of if [42:18] it's, you know, [42:21] >> helpful. It might speed us along on this [42:24] path. [42:25] >> Like I said, we'll see. Just let me [42:27] know. I'm not [42:29] >> Yeah. The other thing you should [42:31] >> Andrew need anything for me? Need it [42:34] before seven. [42:35] >> Yeah. Okay. Yeah. Andrew and John are [42:37] often late as well, so showing up at [42:39] 5:30 or 6:30 your time might not even [42:42] matter. So, [42:44] >> um, but yeah, we're at time and I have [42:47] another meeting. So, [42:48] >> yeah, me, too. All right, sweet. I will [42:50] talk to you'all later. [42:51] >> Thanks, everyone. Have a great day. TS.  📄 **Full text:** ![[Full Document Contents/Articles/Augur8 Knot WG Meeting, Jan 20, 2026.md|Augur/8 Knot WG Meeting, Jan 20, 2026 by YouTube]] ## Highlights - bugs with that and we were seeing that the the pull request reviews like the first part of that chain was running and getting scheduled and completing but we were not seeing anything even scheduled for the second one. Um and so it's like what what's going on here? Why is this not scheduling? Because if the second piece of that change is not scheduling, that whole group is probably not completing or that whole that whole line is probably not completing and then the whole group is not going and then it's not going to get marked as success. So, it's just kind of like I I do need I do ([View Highlight](https://read.readwise.io/read/01kp9kj3d64qrff2j1kpx6wh4j)) - Tags: 👻 ai highlighted - >> What's the task before what what's the last task that you can see in the chain that's scheduled Adrian? >> I mean so all of the >> right before the pull requests like the pull requ the one that goes right before the comments one is we're seeing that succeeding we're just not seeing >> so pull request reviews is succeeding. >> Yes. >> Yes. But then for some reason the the task that's changed immediately after it is somehow not running. That's sort of what leads me to think that this is pro I feel like from my just super quick cursory search of open celery issues ([View Highlight](https://read.readwise.io/read/01kp9kj3gycehaedvs8nbtk3pf)) - Tags: 👻 ai highlighted - has to finish beforehand. So it shows the scheduling order, but it allowed for that pull request review comments to start while pull request reviews was still going. But for smaller instances with a very for a very small repo, um you saw the successes, but it didn't but it didn't um require the ordering to keep to be correct. That's why we had the >> I think that's wrong. the the the docs ([View Highlight](https://read.readwise.io/read/01kp9kj3m02d844zfgb32ay7q9)) - Tags: 👻 ai highlighted - Um so it basically started and finished within 30 seconds. >> Yeah. And so that's why I'm concerned is that >> like pull request review comments. Well, I think this uh >> because my thing is that it's I'm saying is that for something that is more than 30 like the in these longer repositories, it's requiring that it's like it the scheduling order I agree with that's happening but it's I think that it's ([View Highlight](https://read.readwise.io/read/01kp9kj3r36dvqfzred2hhtnex)) - Tags: 👻 ai highlighted - But I would say that the problem that we're seeing like with this solution would not show up in these like these are these tasks are finishing too fast for the error that we saw to pimp up because they're being received in the correct order. They are being scheduled in the correct they're like it's the chaining is >> like pull request reviews is always coming before pull request comments. The problem is is that pull request reviews was still actively running when pull ([View Highlight](https://read.readwise.io/read/01kp9kj3tep5e2m7r6xd5hf6dj)) - Tags: 👻 ai highlighted - instance. Yeah. Like >> from your instance. Yes. From ours like even we were looking at this like hours later, pull request reviews was still going. Um I don't know if I wish I would have taken more screenshots. Um, I don't know if I ended up actually sending any screenshots to you. Yeah, I took one. Um, >> how long how long are we letting it run? And I'm only asking that because if these are larger repositories, it it just might be waiting for the finish signal. And I have seen this ([View Highlight](https://read.readwise.io/read/01kp9kj427bw3hk66mgtefb39b)) - Tags: 👻 ai highlighted - before where something I can see from a data perspective is finished, but for some reason it takes >> Flower like two hours to to to recognize it's finished. So, it's actually even worse than I thought. I'll I'll share my screen. >> Excellent. >> Uh that for you can see pull request reviews, comments, pull requests, reviews. You had pull request, reviews, comments, succeed. And this isn't even started yet. It's only received. ([View Highlight](https://read.readwise.io/read/01kp9kj45jw20958zfh6tmh45p)) - Tags: 👻 ai highlighted