Github Continues to Downfall  Now, I do not like making fun of people's code, okay? We've all been there. We've all written code under duress, shall we say, and it's not impressive. You know, caught absolute cinema. I've been caught on many a PR trying to sneak in some just truly heinous lines of code. So, therefore, I'm not going to make fun of some random person's code, except when that code belongs to one of the world's largest corporations. And for years, people have been begging them to change the code. And they have effectively done nothing ever, including addressing or merging fixes in which is actually costing real companies real money. It looks so bad it looks sinister. Now, I'm not saying that they're intentionally keeping this code bad to make money. Okay? I'm not I would never suggest that. What I am saying is that it just looks really bad. Okay? It looks real bad. But first, I'm at 977,000 subs. Okay, I don't want to say this. Quit making me say this. Just press the button. I am less than 23,000 subs away from making it to the gold. Okay, I get the gold plaque. Just make it happen. Make my dreams come true. The code that's making everybody upset is, of course, GitHub action runners. Now, apparently, this has just been limping for almost a decade now. So many bugs. And the bug that we're going to be looking at today was so egregious that Zigg has decided to leave GitHub altogether because of it and moved to Codeberg. In fact, they say right here, most importantly, actions has inexcusable bugs while being completely neglected. After the CEO said to embrace AI or get out, it seems the lackis at Microsoft took the hint because GitHub actions started vibe scheduling, choosing jobs to run seemingly at random combined with other bugs and inability to manually intervene. This causes our CI system to get so backed up that not even master branch commits can get checked. Okay, so what is the big problem? What is actually causing this huge gigantic just feathers being ruffled thoroughly? Well, it comes down to these simple four lines of code right here. Now, these four lines of code, you may not like this, but this is not what peak engineering looks like. Okay, this is some disgusting code. In fact, Mattlad, one of the most legendary uh programmers of all time, I believe he works at antithesis. He says the following about that function. It has clear formatting. It's short. It uses fancy bash features. The second feature, it has the elegant colon as the while body. It really reads nice. It is also utterly deranged. I have never written four lines of code and have had anybody call those four lines of code utterly deranged. and I have written some deranged code. It is truly impressive. Before you can understand those four lines of code, we need to do a bit kind of a historical deep dive into Microsoft ActionRunner. Trust me, it's worth it cuz it's hilarious the transfer. So, it all starts with this. This is the first public commit known as the GitHub action runner. And in here, we find this beautiful piece of code. So, now what is going on here? As best as I can tell, this is Windows programmers programming bash. And the key feature is is this right here. It's actually a solution from a Stack Overflow question 16 years ago on how to sleep for 5 seconds in a batch file, a command file on Windows because timeout some sometimes just wasn't available and someone suggests using ping as a means to sleep for 5 seconds. So let's look at this code. So apparently if it returns a four, the code needed to do some sort of sleep. Now for whatever reason they were on machines that did not have the sleep command. Now, I actually don't even believe this cuz there is extremely few edge cases from 2010 to 2020 where sleep was not available and likely Microsoft had none of them other than if you're using Windows during the weasel one days. That was kind of it. So, if you had sleep, it'd sleep for 5 seconds. If you didn't have sleep, but you had ping, it would sleep for 4 seconds cuz remember, ping sends off a packet every 1 second. So, with five packets being sent, that means at second zero you send one. Second one, you send your second. Second two, you send your third. Soda only sleeps for 4 seconds. But the truly deranged one is this right here. It echoes sleep into what appears to be a version of dev null 5,000 times. Assuming that echoing takes 1 millisecond. Now, I showed you that because apparently that function was so useful, so dynamite that someone decided that they needed it as a top level function called safe sleep, which again calls sleep for 5 seconds, ping for 4 seconds, or literally eats your CPU, and tries to run 5,000 times, echoing to devnull. This is some truly deranged code. But somehow this code gets worse because it finally gets to its final form, this one right here. And it remained this way for years, costing companies and people who pay for GitHub actions thousands upon tens of thousands, if not hundreds of thousands of dollars. This is single-handedly a line of code that has probably eaten more energy than AI itself. Okay, that was a bit of an exaggeration, but you but just just see. Okay, so to understand this code, first you got to understand what this does. Seconds is a special bash variable that increments as time goes on. So if I set it to zero, it will slowly become one, then two, then three as time goes on on the system. So it is a way to be able to sleep. Obviously, dollar one is the value that's passed into the script. And so what happens here is this is equivalent to doing effectively like a while date. Now minus start is greater than 5,000. To make it easier for you JS Andes out there, this is effectively what's going on. We take a start time and then we just slowly wait until that exceeds the amount of time being passed in. In this example, I just put 5,000, but in the case of this code, it's whatever is being passed in. Now, you're probably thinking, okay, so what's the big deal? What's the big problem? What if I change this code to look something like that? Would you be able to easily spot the problem then? Well, what happen if there's longer than a single millisecond here between the two to the point where it would just run forever? Right? Because if you don't perfectly land on 5,000, what ends up happening? You run forever. Well, this code did the exact same thing because you're on a machine that's running long and difficult jobs. And so what would end up happening? You could end up having more than 1 second wait in between this while loop actually executing. And the crazy part is is that if there is no wait and the while loop is running, it just eats an entire CPU just waiting for 5 seconds so that nothing else can use that CPU. Now, considering that the average size of a GitHub runner is only two CPUs, you could end up running half of your available compute waiting, which means that it would actually cause more of these to overrun over time. And that's exactly what it did. So, somehow on February 22nd, 2022, a lot of twos in that. Okay, I love how many twos there are. Somehow we got rid of the old safe sleep in which actually did at least sleep most of the time and used the system utility when available to have the brand new insane bugridden absolutely destroying people's CI environment code. I think the thing that hurts more is that in 2024 Mr. over here fixes it by simply making it do a less than or equal to operation as opposed to just simply having a not equals to. This would fix all the hanging forever because as he puts a picture right here, this single GitHub action runner ran for 5,135 hours just running forever on one of his machines. Now, GitHub CI does cost based on CPU commit. The cheapest machine is8 per CPU minute. That means that that single dead sleeping process which was burning the CPU at 100% cost about $2400. Absolutely beautiful. We're talking about cinema going on right now. Now, with this incredibly lovely fix that of course is going to be able to save the whole GitHub CI running thing, you you would assume that this would get merged in relatively quick, right? No, about a month later, it was automatically closed and then after this whole problem became an actual issue a year and a half later, they have decided, okay, it is good to merge in this gigantic fix and which fixes the whole zombie process of this forever running loop. Now, there's plenty of other issues. I did just want to highlight this for a quick second. Failed to hash files. It is due to this bug that is in this diff that was merged in two weeks ago and which has had zero addressing of it. But look at this code that they're dealing with. If for if like this code is disgusting so they refactored into something nice, right? Okay. So it does get own property names or it's a function which does for and if right here and then returns this own keys. Just absolutely insane code going on right here that returns another function. Then inside here does an if four and then an if again. It has the same code but multiple times. It is just absolutely horrifying code that somehow the refactor actually just made it more complicated and introduced more bugs. What is going on with GitHub actions? Who approves this type of code? This should not be approved. This is not okay. Anyways, I wanted to highlight some of this code because the fact that there has been over three years with this bug that has just been destroying CI jobs, causing so many problems in life, and costing thousands upon thousands of dollars for no reason other than a bad while loop is insane that it was not diagnosed earlier. GitHub actions are single-handedly one of the most insane platforms ever when you look at the code. And somehow this runs countless projects. You know, I think I've been a little too harsh on the old AI coding because when I look at code like this, it restores my faith in AI because there's just absolutely no way that an AI could be this creative to be able to create code this bad. Truly a work of art. The thing that kills me the most is the fact that this is like a many billion dollar business and they just don't care at all. Microsoft just does not care. And if they do care, they're really really selling us that they don't care. They're like acting like the cool kid that just doesn't care. I don't care about nothing. I'm just too cool. No, your product sucks. You should fix it. Okay. Literally, millions of people rely on this thing to be good. You should fix it. You should make that nice. That's all I have to say. It's another day that I get to make fun of Microsoft. So, it's a day I'm going to take advantage of. The name is I don't really like Microsoft, but also you should hit subscribe because we're almost at a million. Okay, I'm going to stop saying it afterwards. A gen. Hey, do you want to learn how to code? Do you want to become a better back-end engineer? Well, you got to check out boot.dev. Now, I personally have made a couple courses from them. I have live walkthroughs free available on YouTube of the whole course. Everything on boot.dev you can go through for free. But if you want the gamified experience, the tracking of your learning and all that, then you got to pay up the money. But hey, go check them out. It's awesome. Many content creators you know and you like make courses there. boot.dev/prime dev/prime for 25% off., Github Continues to Downfall, YouTube, 01kc4dnxs5qthd5135p00wefsx.