Github continues to Downfall ๐Ÿ”— **Source:** [Github continues to Downfall](https://www.youtube.com/watch?v=E3_95BZYIVs) > [!note]+ Document Notes > tags: [github-actions, microsoft, ci-cd] > author: ThePrimeTime > source: YouTube > reference: https://www.youtube.com/watch?v=E3_95BZYIVs > date created: 2024-12-19 14:30 > > # ๐Ÿท๏ธ Title: Github continues to Downfall > <!-- deep-read:sections start --> ## ๐Ÿงฐ Action Inbox - [H1] Review GitHub Actions runner code for inefficient sleep implementations โ€” 00:00-05:00 #github-actions - [H2] Monitor CI costs for zombie processes eating CPU cycles โ€” 08:30-12:00 #cost-optimization - [H3] Consider alternative CI platforms like Codeberg for critical projects โ€” 01:30-02:00 #platform-migration ## ๐Ÿงญ Table of Contents - [๐Ÿ“ Summary](#-summary) - [๐Ÿ“– Main Content](#-main-content) - [๐Ÿ—’๏ธ Notes](#๏ธ-notes) - [๐Ÿ“š References](#-references) - [๐Ÿ—ฃ๏ธ Paraphrase](#๏ธ-paraphrase) - [๐Ÿ“ฐ Newsletter](#-newsletter) - [โœ… Action Items](#-action-items) - [๐Ÿงช Hypotheses](#-hypotheses) - [๐Ÿ“Š Methodology](#-methodology) - [โ“ Questions](#-questions) - [๐Ÿ”ฎ Future Research](#-future-research) - [๐Ÿ’ก Implications](#-implications) - [๐Ÿง  Learning Styles Assessment](#-learning-styles-assessment) - [๐Ÿ“Š Accessibility Assessment](#-accessibility-assessment) - [๐ŸŒ DEIB Assessment](#-deib-assessment) - [โœ… Logical Validity](#-logical-validity) - [๐Ÿ›ก๏ธ DARVO Analysis](#-darvo-analysis) - [๐ŸŽฏ Conclusion](#-conclusion) ### ๐Ÿ”— References - [GitHub Actions Runner Repository](https://github.com/actions/runner) - [Zig Project Migration to Codeberg](https://codeberg.org) ## ๐Ÿ“ Summary - GitHub Actions runner contains a catastrophic bug in sleep functionality that causes infinite loops and wastes CPU resources - The bug has existed for years, costing companies thousands in unnecessary compute charges - Zig project migrated away from GitHub to Codeberg due to these persistent issues - Microsoft/GitHub appears negligent in addressing known critical bugs despite community fixes - The problematic code stems from Windows programmers writing bash without understanding Unix sleep utilities ## ๐Ÿ“– Main Content The video exposes a critical bug in GitHub Actions runners that has persisted for years, causing infinite loops and excessive CPU usage. The issue centers around a flawed sleep implementation that uses busy-waiting instead of proper system sleep calls. The problematic code evolved from a Stack Overflow solution for sleeping in Windows batch files, using ping commands as a timing mechanism. Over time, this became a bash function that polls the `$SECONDS` variable in a while loop, but with a fatal flaw: if the loop execution takes longer than expected, it can miss the exact target time and run forever. One documented case shows a runner consuming 5,135 hours of compute time (costing ~$2,400) due to this bug. Despite community members providing fixes, GitHub has been slow to merge solutions, leading some major projects like Zig to abandon the platform entirely. ## ๐Ÿ—’๏ธ Notes #### โฑ๏ธ [00:00](https://youtu.be/E3_95BZYIVs?t=0) โ€” [H1] Introduction to GitHub's code quality issues - Key idea: Criticism of GitHub Actions runner code that costs companies real money - Evidence: "It looks so bad it looks sinister... it's actually costing real companies real money" #### โฑ๏ธ [01:30](https://youtu.be/E3_95BZYIVs?t=90) โ€” [H2] Zig project abandons GitHub - Key idea: Major project migrated to Codeberg due to GitHub Actions bugs - Evidence: "Actions has inexcusable bugs while being completely neglected" #### โฑ๏ธ [02:45](https://youtu.be/E3_95BZYIVs?t=165) โ€” [H3] The problematic four lines of code - Key idea: Four lines of bash code described as "utterly deranged" by legendary programmer - Evidence: "It is also utterly deranged. I have never written four lines of code and have had anybody call those four lines of code utterly deranged" #### โฑ๏ธ [03:30](https://youtu.be/E3_95BZYIVs?t=210) โ€” [H2] Historical evolution of the sleep function - Key idea: Code originated from Windows programmers adapting Stack Overflow batch file solutions - Evidence: "This is Windows programmers programming bash... a solution from a Stack Overflow question 16 years ago" #### โฑ๏ธ [05:15](https://youtu.be/E3_95BZYIVs?t=315) โ€” [H3] Original safe_sleep implementations - Key idea: Early versions had multiple fallback mechanisms including CPU-eating echo loops - Evidence: "It echoes sleep into what appears to be a version of dev null 5,000 times" #### โฑ๏ธ [07:00](https://youtu.be/E3_95BZYIVs?t=420) โ€” [H1] The infinite loop bug mechanism - Key idea: $SECONDS variable polling can miss exact timing and run forever - Evidence: "If you don't perfectly land on 5,000, what ends up happening? You run forever" #### โฑ๏ธ [08:30](https://youtu.be/E3_95BZYIVs?t=510) โ€” [H2] CPU consumption impact - Key idea: Bug consumes entire CPU cores while waiting, reducing available compute - Evidence: "You could end up running half of your available compute waiting" #### โฑ๏ธ [09:45](https://youtu.be/E3_95BZYIVs?t=585) โ€” [H3] Timeline of the bug introduction - Key idea: Problematic code was introduced on February 22nd, 2022 - Evidence: "Somehow on February 22nd, 2022... we got rid of the old safe sleep" #### โฑ๏ธ [10:30](https://youtu.be/E3_95BZYIVs?t=630) โ€” [H2] Real-world cost example - Key idea: Single runner consumed 5,135 hours costing approximately $2,400 - Evidence: "That single dead sleeping process... cost about $2400" #### โฑ๏ธ [11:15](https://youtu.be/E3_95BZYIVs?t=675) โ€” [H3] Community fix rejection - Key idea: GitHub closed community fix after one month, took 1.5 years to merge - Evidence: "About a month later, it was automatically closed" #### โฑ๏ธ [12:30](https://youtu.be/E3_95BZYIVs?t=750) โ€” [H2] Additional code quality issues - Key idea: Other bugs exist with equally problematic code patterns - Evidence: "Failed to hash files... zero addressing of it" #### โฑ๏ธ [13:45](https://youtu.be/E3_95BZYIVs?t=825) โ€” [H3] Refactoring that introduced more bugs - Key idea: Recent refactoring made code more complex and introduced new issues - Evidence: "The refactor actually just made it more complicated and introduced more bugs" #### โฑ๏ธ [14:30](https://youtu.be/E3_95BZYIVs?t=870) โ€” [H1] Microsoft's apparent negligence - Key idea: Multi-billion dollar platform shows lack of care for critical infrastructure - Evidence: "This is like a many billion dollar business and they just don't care at all" ## ๐Ÿ“š References - [GitHub Actions Runner Source Code](https://github.com/actions/runner) - [Stack Overflow Batch File Sleep Solutions](https://stackoverflow.com) - [Zig Project on Codeberg](https://codeberg.org/ziglang) ## ๐Ÿ—ฃ๏ธ Paraphrase - GitHub's Actions runner contains a sleep function that can enter infinite loops, wasting CPU resources and costing users thousands of dollars in unnecessary compute charges - The problematic code originated from Windows developers adapting outdated Stack Overflow solutions for Unix environments without proper understanding of system utilities - Despite community members identifying and providing fixes for these critical bugs, GitHub/Microsoft has shown negligence in addressing them promptly, leading major projects to abandon the platform ## ๐Ÿ“ฐ Newsletter **GitHub Actions Bug Costing Developers Thousands in Wasted Compute** A critical bug in GitHub Actions runners has been silently draining CI budgets for years. The issue? A flawed sleep implementation that can enter infinite loops, consuming entire CPU cores while accomplishing nothing. The problematic code uses busy-waiting instead of proper system sleep calls, occasionally missing timing windows and running forever. One documented case consumed over 5,000 hours of compute time, costing approximately $2,400 in unnecessary charges. **Key Takeaway:** Review your GitHub Actions usage for unexpectedly long-running jobs and consider monitoring tools to catch zombie processes. **What You Can Do:** - Audit recent Action runs for unusually long execution times - Consider alternative CI platforms for critical workflows - Implement monitoring alerts for runaway processes This highlights the importance of proper code review and testing, even for basic utility functions. When building CI/CD pipelines, always implement proper timeouts and monitoring. Resources: [GitHub Actions Documentation](https://docs.github.com/en/actions), [CI/CD Best Practices](https://github.blog), [Alternative CI Platforms](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) ## โœ… Action Items #### DevRel - Developer advocacy: Create content warning about GitHub Actions sleep bugs, demonstrate proper CI monitoring - Education: Workshop on CI cost optimization and zombie process detection - Developer experience: Document alternative CI platforms and migration guides - Developer marketing: Share cost-saving tips for CI/CD workflows - Community management: Collect feedback on CI platform reliability experiences #### OSPO - Policy & Governance: Establish CI platform evaluation criteria and backup strategies - License Compliance: Review CI platform terms for cost liability clauses - Security Management: Audit CI runners for resource exhaustion vulnerabilities - Software Inventory: Catalog CI dependencies and platform lock-in risks - Developer Support: Training on CI debugging and cost monitoring tools - Community Engagement: Coordinate with CI platform vendors on reliability improvements - Strategic Alignment: KPIs for CI cost efficiency and reliability metrics #### Community - Maintainers: Provide templates for CI monitoring and cost alerts - Contributors: Onboarding includes CI best practices and budget awareness - Sponsors: Transparent reporting on CI costs and optimization efforts - Ecosystem: Cross-project coordination on CI platform experiences #### WordPress - Core: Audit GitHub Actions workflows for potential zombie processes - Core Performance: Monitor CI resource usage and execution times - Community: Share CI cost optimization strategies with meetup organizers - Training: Lesson plans on CI/CD platform selection and monitoring - Documentation: DevHub guidance on CI best practices and cost management - Meta: .org infrastructure CI reliability and cost monitoring ## ๐Ÿงช Hypotheses - GitHub's market dominance reduces incentive to fix non-critical infrastructure bugs quickly - Windows-centric development culture at Microsoft contributes to Unix/Linux platform bugs - CI platform costs are often hidden from developers, reducing pressure for efficiency optimization - Large-scale infrastructure bugs persist longer in proprietary platforms than open-source alternatives ### ๐Ÿ“Š Methodology - Comparative analysis of bug resolution times across different CI platforms - Cost analysis of zombie processes across different runner configurations - Survey of developer awareness regarding CI cost attribution and monitoring - Code quality assessment comparing Microsoft's open-source vs proprietary projects ### โ“ Questions - How many other critical infrastructure bugs exist in GitHub Actions that remain undiagnosed? - What percentage of GitHub Actions costs are attributed to inefficient or buggy runner code? - How do bug resolution times compare between GitHub's free and enterprise tiers? ### ๐Ÿ”ฎ Future Research - Comprehensive audit of GitHub Actions runner codebase for similar efficiency issues - Analysis of CI platform reliability and cost-effectiveness across the industry - Investigation of Microsoft's internal code review processes for open-source infrastructure ### ๐Ÿ’ก Implications - Practice: Implement CI monitoring and alerting to catch runaway processes early - Policy: Establish CI platform evaluation criteria including reliability and support responsiveness - Security: Resource exhaustion attacks via CI platforms may be more viable than previously considered - Education: Developer training must include CI cost awareness and monitoring techniques - Funding: CI costs may represent hidden technical debt that compounds over time without proper oversight ### ๐Ÿง  Learning Styles Assessment Reading level estimate: 10th Grade (Flesch-Kincaid) #### ๐Ÿ‘€ Visual learners - Present: Code examples, before/after comparisons, cost calculations - Improve: Architecture diagrams showing runner resource consumption, visual timeline of bug evolution #### ๐Ÿ”Š Aural learners - Present: Video format with narrated code explanation - Improve: Podcast version focusing on lessons learned, audio summary of key takeaways #### ๐Ÿ—ฃ๏ธ Verbal learners - Present: Clear analogies (comparing to real-world cost scenarios), defined technical terms - Improve: Story-driven explanation of how bugs evolve in large codebases #### ๐Ÿงฉ Kinesthetic learners - Present: Code examples that can be tested and modified - Improve: Hands-on lab for reproducing the bug, checklist for auditing CI workflows #### ๐Ÿง  Logical learners - Present: Timeline of bug introduction and fixes, cost calculations, cause-and-effect analysis - Improve: Decision tree for CI platform selection, systematic debugging methodology #### ๐Ÿง‘โ€๐Ÿคโ€๐Ÿง‘ Social learners - Present: Community discussion around platform migration decisions - Improve: Forum for sharing CI horror stories and solutions, peer review of CI configurations #### ๐Ÿง˜โ€โ™€๏ธ Solitary learners - Present: Self-contained code analysis and examples - Improve: Self-assessment worksheet for CI efficiency, personal CI cost tracking template #### ๐ŸŒฟ Naturalistic learners - Present: Connection between code efficiency and environmental impact of compute waste - Improve: Lifecycle analysis of CI resource consumption, sustainability implications of inefficient code ### ๐Ÿ“Š Accessibility Assessment #### Screen reader semantics - Present: Clear heading hierarchy, code blocks properly labeled - Improve: Alt text for code screenshots, better semantic markup for code explanations #### Keyboard navigation - Present: Video player supports keyboard controls - Improve: Timestamped navigation links, skip-to-code functionality #### Contrast and readability - Present: High contrast code examples in video - Improve: Transcript with proper formatting, adjustable text size for code blocks #### Text resizing and reflow - Present: YouTube player supports zoom - Improve: Responsive transcript layout, code examples that reflow properly #### Media and captions - Present: Auto-generated captions available - Improve: Professional captions with technical term accuracy, downloadable transcript #### Forms and interactions - Present: Standard YouTube interface - Improve: Custom controls with better labeling, accessible sharing options #### Standards reference - Video content meets WCAG 2.2 AA for media with transcript availability ### ๐ŸŒ DEIB Assessment #### Cultural breadth - Present: Technical content applicable globally - Improve: Examples from diverse CI usage scenarios, international cost perspectives #### Inclusive language - Present: Generally neutral technical language - Improve: Avoid dismissive language about programming skill levels, more supportive tone #### Neurodiversity support - Present: Clear structure with timestamped sections - Improve: Content warnings for rapid code switching, summary sections for key points #### Representation - Present: Focus on technical merit rather than personal characteristics - Improve: Acknowledge diverse backgrounds in development teams, credit community contributors #### Psychological safety - Present: Criticism directed at corporate practices rather than individuals - Improve: Emphasize learning opportunities rather than just criticism, acknowledge complexity of large system maintenance ### โœ… Logical Validity #### Strong reasoning - Deductive strength: Cost calculations based on documented compute rates and measured usage - Inductive strength: Pattern recognition from historical code evolution and bug introduction - Causal reasoning: Clear connection between code behavior and resource consumption - Analogical strength: Comparison to other infrastructure reliability issues #### Fallacies - Ad hominem: Some attacks on Microsoft's competence rather than focusing purely on technical issues - Hasty generalization: Extrapolating from one bug to overall platform quality without comprehensive analysis #### Tone of voice - Overall tone: Humorous but critical, occasionally inflammatory when discussing corporate responsibility ### ๐Ÿ›ก๏ธ DARVO Analysis - Deny: No significant denial patterns detected - Attack: Criticism directed at corporate practices rather than individuals making accusations - Reverse victim/offender: No victim/offender reversal present - Emotional cues and rhetoric: Uses humor and frustration to highlight technical incompetence, appropriate given context ## ๐ŸŽฏ --- ## Full Transcript [00:00] Now, I do not like making fun of<br>[00:01] people's code, okay? We've all been<br>[00:03] there. We've all written code under<br>[00:05] duress, shall we say, and it's not<br>[00:07] impressive. You know, caught absolute<br>[00:10] cinema. I've been caught on many a PR<br>[00:13] trying to sneak in some just truly<br>[00:14] heinous lines of code. So, therefore,<br>[00:16] I'm not going to make fun of some random<br>[00:19] person's code, except when that code<br>[00:21] belongs to one of the world's largest<br>[00:22] corporations. And for years, people have<br>[00:25] been begging them to change the code.<br>[00:27] And they have effectively done nothing<br>[00:29] ever, including addressing or merging<br>[00:31] fixes in which is actually costing real<br>[00:34] companies real money. It looks so bad it<br>[00:38] looks sinister. Now, I'm not saying that<br>[00:40] they're intentionally keeping this code<br>[00:42] bad to make money. Okay? I'm not I would<br>[00:45] never suggest that. What I am saying is<br>[00:48] that it just looks really bad. Okay? It<br>[00:49] looks real bad. But first, I'm at<br>[00:52] 977,000<br>[00:54] subs. Okay, I don't want to say this.<br>[00:56] Quit making me say this. Just press the<br>[00:57] button. I am less than 23,000 subs away<br>[01:00] from making it to the gold. Okay, I get<br>[01:02] the gold plaque. Just make it happen.<br>[01:04] Make my dreams come true. The code<br>[01:05] that's making everybody upset is, of<br>[01:07] course, GitHub action runners. Now,<br>[01:09] apparently, this has just been limping<br>[01:12] for almost a decade now. So many bugs.<br>[01:15] And the bug that we're going to be<br>[01:16] looking at today was so egregious that<br>[01:19] Zigg has decided to leave GitHub<br>[01:22] altogether because of it and moved to<br>[01:24] Codeberg. In fact, they say right here,<br>[01:26] most importantly, actions has<br>[01:28] inexcusable bugs while being completely<br>[01:30] neglected. After the CEO said to embrace<br>[01:33] AI or get out, it seems the lackis at<br>[01:35] Microsoft took the hint because GitHub<br>[01:36] actions started vibe scheduling,<br>[01:38] choosing jobs to run seemingly at random<br>[01:40] combined with other bugs and inability<br>[01:42] to manually intervene. This causes our<br>[01:44] CI system to get so backed up that not<br>[01:47] even master branch commits can get<br>[01:49] checked. Okay, so what is the big<br>[01:51] problem? What is actually causing this<br>[01:52] huge gigantic just feathers being<br>[01:55] ruffled thoroughly? Well, it comes down<br>[01:57] to these simple four lines of code right<br>[02:00] here. Now, these four lines of code, you<br>[02:02] may not like this, but this is not what<br>[02:03] peak engineering looks like. Okay, this<br>[02:05] is some disgusting code. In fact,<br>[02:08] Mattlad, one of the most legendary uh<br>[02:11] programmers of all time, I believe he<br>[02:13] works at antithesis. He says the<br>[02:14] following about that function. It has<br>[02:16] clear formatting. It's short. It uses<br>[02:18] fancy bash features. The second feature,<br>[02:20] it has the elegant colon as the while<br>[02:22] body. It really reads nice. It is also<br>[02:25] utterly deranged.<br>[02:28] I have never written four lines of code<br>[02:31] and have had anybody call those four<br>[02:33] lines of code utterly deranged. and I<br>[02:35] have written some deranged code. It is<br>[02:38] truly impressive. Before you can<br>[02:40] understand those four lines of code, we<br>[02:42] need to do a bit kind of a historical<br>[02:44] deep dive into Microsoft ActionRunner.<br>[02:46] Trust me, it's worth it cuz it's<br>[02:48] hilarious the transfer. So, it all<br>[02:49] starts with this. This is the first<br>[02:51] public commit known as the GitHub action<br>[02:54] runner. And in here, we find this<br>[02:56] beautiful piece of code. So, now what is<br>[02:58] going on here? As best as I can tell,<br>[03:00] this is Windows programmers programming<br>[03:03] bash. And the key feature is is this<br>[03:05] right here. It's actually a solution<br>[03:07] from a Stack Overflow question 16 years<br>[03:10] ago on how to sleep for 5 seconds in a<br>[03:13] batch file, a command file on Windows<br>[03:15] because timeout some sometimes just<br>[03:17] wasn't available and someone suggests<br>[03:19] using ping as a means to sleep for 5<br>[03:23] seconds. So let's look at this code. So<br>[03:25] apparently if it returns a four, the<br>[03:27] code needed to do some sort of sleep.<br>[03:29] Now for whatever reason they were on<br>[03:30] machines that did not have the sleep<br>[03:32] command. Now, I actually don't even<br>[03:33] believe this cuz there is extremely few<br>[03:35] edge cases from 2010 to 2020 where sleep<br>[03:38] was not available and likely Microsoft<br>[03:41] had none of them other than if you're<br>[03:43] using Windows during the weasel one<br>[03:45] days. That was kind of it. So, if you<br>[03:47] had sleep, it'd sleep for 5 seconds. If<br>[03:49] you didn't have sleep, but you had ping,<br>[03:51] it would sleep for 4 seconds cuz<br>[03:53] remember, ping sends off a packet every<br>[03:55] 1 second. So, with five packets being<br>[03:57] sent, that means at second zero you send<br>[03:59] one. Second one, you send your second.<br>[04:01] Second two, you send your third. Soda<br>[04:03] only sleeps for 4 seconds. But the truly<br>[04:05] deranged one is this right here. It<br>[04:06] echoes sleep into what appears to be a<br>[04:08] version of dev null 5,000 times.<br>[04:11] Assuming that echoing takes 1<br>[04:13] millisecond. Now, I showed you that<br>[04:15] because apparently that function was so<br>[04:17] useful, so dynamite that someone decided<br>[04:19] that they needed it as a top level<br>[04:21] function called safe sleep, which again<br>[04:24] calls sleep for 5 seconds, ping for 4<br>[04:26] seconds, or literally eats your CPU, and<br>[04:29] tries to run 5,000 times, echoing to<br>[04:32] devnull. This is some truly deranged<br>[04:34] code. But somehow this code gets worse<br>[04:36] because it finally gets to its final<br>[04:38] form, this one right here. And it<br>[04:40] remained this way for years, costing<br>[04:42] companies and people who pay for GitHub<br>[04:44] actions thousands upon tens of<br>[04:46] thousands, if not hundreds of thousands<br>[04:48] of dollars. This is single-handedly a<br>[04:50] line of code that has probably eaten<br>[04:52] more energy than AI itself. Okay, that<br>[04:55] was a bit of an exaggeration, but you<br>[04:57] but just just see. Okay, so to<br>[04:59] understand this code, first you got to<br>[05:01] understand what this does. Seconds is a<br>[05:03] special bash variable that increments as<br>[05:05] time goes on. So if I set it to zero, it<br>[05:08] will slowly become one, then two, then<br>[05:10] three as time goes on on the system. So<br>[05:12] it is a way to be able to sleep.<br>[05:15] Obviously, dollar one is the value<br>[05:17] that's passed into the script. And so<br>[05:19] what happens here is this is equivalent<br>[05:21] to doing effectively like a while date.<br>[05:23] Now minus start is greater than 5,000.<br>[05:26] To make it easier for you JS Andes out<br>[05:28] there, this is effectively what's going<br>[05:30] on. We take a start time and then we<br>[05:33] just slowly wait until that exceeds the<br>[05:35] amount of time being passed in. In this<br>[05:37] example, I just put 5,000, but in the<br>[05:39] case of this code, it's whatever is<br>[05:40] being passed in. Now, you're probably<br>[05:42] thinking, okay, so what's the big deal?<br>[05:44] What's the big problem? What if I change<br>[05:46] this code to look something like that?<br>[05:49] Would you be able to easily spot the<br>[05:51] problem then? Well, what happen if<br>[05:53] there's longer than a single millisecond<br>[05:55] here between the two to the point where<br>[05:58] it would just run forever? Right?<br>[06:00] Because if you don't perfectly land on<br>[06:02] 5,000, what ends up happening? You run<br>[06:04] forever. Well, this code did the exact<br>[06:06] same thing because you're on a machine<br>[06:08] that's running long and difficult jobs.<br>[06:10] And so what would end up happening? You<br>[06:12] could end up having more than 1 second<br>[06:14] wait in between this while loop actually<br>[06:16] executing. And the crazy part is is that<br>[06:19] if there is no wait and the while loop<br>[06:21] is running, it just eats an entire CPU<br>[06:23] just waiting for 5 seconds so that<br>[06:26] nothing else can use that CPU. Now,<br>[06:28] considering that the average size of a<br>[06:30] GitHub runner is only two CPUs, you<br>[06:32] could end up running half of your<br>[06:34] available compute waiting, which means<br>[06:37] that it would actually cause more of<br>[06:38] these to overrun over time. And that's<br>[06:41] exactly what it did. So, somehow on<br>[06:44] February 22nd, 2022, a lot of twos in<br>[06:47] that. Okay, I love how many twos there<br>[06:48] are. Somehow we got rid of the old safe<br>[06:50] sleep in which actually did at least<br>[06:53] sleep most of the time and used the<br>[06:56] system utility when available to have<br>[06:58] the brand new insane bugridden<br>[07:01] absolutely destroying people's CI<br>[07:03] environment code. I think the thing that<br>[07:05] hurts more is that in 2024 Mr. over here<br>[07:09] fixes it by simply making it do a less<br>[07:12] than or equal to operation as opposed to<br>[07:15] just simply having a not equals to. This<br>[07:18] would fix all the hanging forever<br>[07:19] because as he puts a picture right here,<br>[07:21] this single GitHub action runner ran for<br>[07:24] 5,135<br>[07:26] hours just running forever on one of his<br>[07:29] machines. Now, GitHub CI does cost based<br>[07:32] on CPU commit. The cheapest machine is8<br>[07:35] per CPU minute. That means that that<br>[07:37] single dead sleeping process which was<br>[07:39] burning the CPU at 100% cost about<br>[07:42] $2400.<br>[07:44] Absolutely<br>[07:46] beautiful. We're talking about cinema<br>[07:48] going on right now. Now, with this<br>[07:50] incredibly lovely fix that of course is<br>[07:52] going to be able to save the whole<br>[07:54] GitHub CI running thing, you you would<br>[07:56] assume that this would get merged in<br>[07:57] relatively quick, right? No, about a<br>[07:59] month later, it was automatically closed<br>[08:00] and then after this whole problem became<br>[08:03] an actual issue a year and a half later,<br>[08:07] they have decided, okay, it is good to<br>[08:09] merge in this gigantic fix and which<br>[08:11] fixes the whole zombie process of this<br>[08:13] forever running loop. Now, there's<br>[08:15] plenty of other issues. I did just want<br>[08:16] to highlight this for a quick second.<br>[08:17] Failed to hash files. It is due to this<br>[08:20] bug that is in this diff that was merged<br>[08:22] in two weeks ago and which has had zero<br>[08:24] addressing of it. But look at this code<br>[08:26] that they're dealing with. If for if<br>[08:30] like this code is disgusting so they<br>[08:32] refactored into something nice, right?<br>[08:34] Okay. So it does get own property names<br>[08:36] or it's a function which does for and if<br>[08:38] right here and then returns this own<br>[08:40] keys. Just absolutely insane code going<br>[08:43] on right here that returns another<br>[08:44] function. Then inside here does an if<br>[08:46] four and then an if again. It has the<br>[08:48] same code but multiple times. It is just<br>[08:50] absolutely horrifying code that somehow<br>[08:53] the refactor actually just made it more<br>[08:55] complicated and introduced more bugs.<br>[08:58] What is going on with GitHub actions?<br>[09:01] Who approves this type of code? This<br>[09:04] should not be approved. This is not<br>[09:05] okay. Anyways, I wanted to highlight<br>[09:07] some of this code because the fact that<br>[09:09] there has been over three years with<br>[09:11] this bug that has just been destroying<br>[09:14] CI jobs, causing so many problems in<br>[09:17] life, and costing thousands upon<br>[09:19] thousands of dollars for no reason other<br>[09:20] than a bad while loop is insane that it<br>[09:24] was not diagnosed earlier. GitHub<br>[09:26] actions are single-handedly<br>[09:29] one of the most insane platforms ever<br>[09:32] when you look at the code. And somehow<br>[09:34] this runs countless projects. You know,<br>[09:36] I think I've been a little too harsh on<br>[09:38] the old AI coding because when I look at<br>[09:40] code like this, it restores my faith in<br>[09:42] AI<br>[09:44] because there's just absolutely no way<br>[09:46] that an AI could be this creative to be<br>[09:49] able to create code this bad. Truly a<br>[09:52] work of art. The thing that kills me the<br>[09:54] most is the fact that this is like a<br>[09:56] many billion dollar business and they<br>[09:58] just don't care at all. Microsoft just<br>[10:01] does not care. And if they do care,<br>[10:03] they're really really selling us that<br>[10:05] they don't care. They're like acting<br>[10:06] like the cool kid that just doesn't<br>[10:07] care. I don't care about nothing. I'm<br>[10:10] just too cool. No, your product sucks.<br>[10:11] You should fix it. Okay. Literally,<br>[10:14] millions of people rely on this thing to<br>[10:15] be good. You should fix it. You should<br>[10:17] make that nice. That's all I have to<br>[10:18] say. It's another day that I get to make<br>[10:20] fun of Microsoft. So, it's a day I'm<br>[10:22] going to take advantage of. The name<br>[10:26] is I don't really like Microsoft, but<br>[10:28] also you should hit subscribe because<br>[10:29] we're almost at a million. Okay, I'm<br>[10:30] going to stop saying it afterwards. A<br>[10:32] gen. Hey, do you want to learn how to<br>[10:34] code? Do you want to become a better<br>[10:36] back-end engineer? Well, you got to<br>[10:37] check out boot.dev. Now, I personally<br>[10:39] have made a couple courses from them. I<br>[10:40] have live walkthroughs free available on<br>[10:43] YouTube of the whole course. Everything<br>[10:45] on boot.dev you can go through for free.<br>[10:48] But if you want the gamified experience,<br>[10:49] the tracking of your learning and all<br>[10:51] that, then you got to pay up the money.<br>[10:52] But hey, go check them out. It's<br>[10:54] awesome. Many content creators you know<br>[10:56] and you like make courses there.<br>[10:58] boot.dev/prime dev/prime for 25% off. <!-- deep-read:sections end -->