Kaizen! Let It Crash  ## Document Notes ## Highlights - **Let Failures Be Localized** - Let-it-crash means design systems so small parts can fail without bringing the whole system down. - Delegate failure handling to supervisors or the platform so developers focus on unique app logic. Transcript: Jerod Santo Take us down the rabbit hole. The OOM, out of memory. Who remembers this book? Gerhard Lazu Erlang and Anger. Erlang and Anger. Stuff Goes Bad by Fred Hebert. Ferd.ca. Jerod Santo Now, I remember Learn Use Some Erlang for Great Good, but I do not remember this one in particular. So I'm not sure why the other one hit my radar because he wrote both of them, it seems. But when did this one come out? Gerhard Lazu Wow. So this one, if I look um i just switched to the browser 2016 2017 while he was still at heroku remember heroku those were the days so about 10 years ago and um fred i mean he just like if you Don't know his blog i mean it's just amazing i'll just click it very quickly uh just to have a look oh i think it's one of the best blogs out there there's so much goodness here so much but One of my favorites is uh cues and queuing and how cues don't protect from overload so cue cues don't fix overload and this is so relevant to today's conversation as well. But there's a lot of stuff in the Erlang ecosystem. And there's many, many things that Ferd wrote over the years that are so relevant to today. So if I click on download PDF, right, by the way, this is like a, it's amazing. This book is open source. You can download it open source freely available uh creative commons license and i'm going to make this a little bit bigger so we can see what's happening and if i search for let it crash It's page number one it's in the introduction page one page one and this idea of let it crash really comes from the Erlang ecosystem. It's very well renowned there because of how the Erlang VM works and how all the processes and the supervision trees just it was built this way. And we know a thing or two about Erlang, Jared, right? Because the application, Elixir, the Phoenix framework runs on the same principle. I know a thing and you know two. So that's how we get to a thing or two. And Adam, I'm sure he knows the big one, but we don't know whether he's going to share it. The point is, the point is when you think about let it crash, Jared, in your, like from your development experience with Erlang, with Elixir, Phoenix, is there any situation, any moment Where you could experience it and you realize, huh, that's nice. Jerod Santo When I let it crash. When you let it crash. Well, it's nice that the beam seems to handle a lot of the problems with letting it crash. You know, it just goes again or there's a supervision tree and things watching each other. And I don't have to think about it very much. I can't think of like an instance in development where i was like this is really useful but i'm sure you could come up with one yeah so you know when you write code we tend to write code very Gerhard Lazu Defensively um typically try a catch so you feel like you need to account for every single scenario and the let it crash philosophy is about not preventing failure learning from it what That means is you need to have a context where it's safe for things to crash and the overall system will still remain stable. So how can you build a resilient system? And really this is about resiliency, where the core of the system will remain running and the system as a whole will remain running, even though parts of it may experience failures. But those failures will not bring everything down and that's really important so fewer try catch blocks don't code defensively let it crash and separate the code that solves the problem From the code that fixes the failures and the more you can lean into the framework or the VM or whatever you have, the system, to deal with failures, the better off you are to focus on the Things that are unique to your application. ([Time 0:12:15](https://share.snipd.com/snip/62cc6d4b-9c15-4740-a3a7-e3a6634051b4)) - **Kill Heavy Processes, Not VMs** - Monitor process memory and kill offending processes rather than rebooting the VM when possible. - Use platform signals and process-level controls to recover faster and keep services available. Transcript: Gerhard Lazu And there's a certain property about the system, and this is Varnish specifically, that made these crashes pretty okay. And the property which I'm referring to is when you start the Varnish D, the daemon, Varnish itself runs as a thread and you have many, many threads that do different things. So when we had these out of memory crashes, all that happened, the thread was killed, which means that the system as a whole didn't crash. The VM didn't, the firecracker VM didn't crash. The application need to restart. It was just a thread that was using too much memory and it restarted within seconds. As in maybe two seconds everything was back to normal obviously the cache was cold but it was good i mean the vm and that's why the memory looked a bit interesting and that it doesn't release All the memory the vm doesn't restart there's there's not many hangs it restarts and it crashes really really quickly so that's a nice property well that confuses me So how does Fly know Jerod Santo About it then if it's just happening inside of Varnish? Gerhard Lazu So it's looking at the process. It has looking at the process ID, which process uses the most memory. And it's the same process that's asking for more memory. So it just basically will just send a signal to that process and kill that process. But that is just a thread that maps to a thread. So Varnish itself didn't crash. It's just a thread that maps to process ID that crashed. And then it was restarted by the varnish daemon. ([Time 0:17:05](https://share.snipd.com/snip/d73696df-9283-41ac-83df-ef3e2f3be0c2)) - **Fragmentation Blocks Big Objects** - Large, uncachable MP3s caused memory fragmentation and forced evictions in Varnish. - Fragmentation prevents fitting big objects even if total free memory exists. Transcript: Gerhard Lazu So what is many? Well, we have thousands at this point of MP3 files across all the podcasts, since the beginning of time. Large means anywhere from 30 to 40 megabytes to 100 plus megabytes. So that's, I mean, just think if you had to load a thousand files that take 100 megabytes, that's a lot of memory that you need to have available. And the problem is that once you store these large files, as we discovered, you get memory fragmentation. In that, imagine that you have all the memory available you keep storing all these files and at some point there's no more memory left so what do you do well you need to see what can evict From memory so that you can store the new file so imagine that you evict a few of those objects but maybe they aren't big enough and you haven't evicted them fast enough so then you have Like this big file that can't fit anywhere because the sizes, like the holes that you have in memory aren't big enough for this file to fit. And there's no defragmentation or nothing like that that runs in the background, which means that even though technically you kind of would have space in the memory for the specific Files, you may not. And then it's, it's, it's a, it can't be stored in memory. Now the thing in Varnish is actually called, I kid you not, N underscore L-R underscore nuked. So I think the connection to the nuke and to the book and to let it crash is right there. So L-R nuked, basically, it's like a forced eviction. So it's an event where an object has to be evicted from the cache just to make room for a new one because the storage is full. So you can see how many times this has happened. And that's like an important metric that if we look at, we can see we had too many of these events, right? Like many objects were being nuked from memory to make room for new objects, but sometimes they wouldn't fit. So how badly did it nuke? ([Time 0:24:05](https://share.snipd.com/snip/10188780-67fe-41d4-a616-83a4ca35a029)) - **Use File Cache For Large Files** - Move large objects to a file cache on disk and tune thread pools and workspace sizes in Varnish. - Add tests and split VCL logic to make cache behavior predictable and maintainable. Transcript: Gerhard Lazu I highlighted a few, so I'm going to look at this one. So it's not just that. You also need to tune, for example, thread pools. You need to tune the minimum, the maximum. You need to tune the workspace backend, like how many memory structures get allocated. You need to configure the new limit. And there's a couple more things that we had to go through just to make things stable now i just going to very quickly mention these things you can go and have a look at pull requests to see What else went into it so this was the one file the other one was the regions that's another thing not all regions would suffer from this so you don't want to allocate too much memory or Too much CPU to regions where maybe they don't get a lot of traffic. And you would think that this thing is easy. But oh man, I have a surprise for you. You can't mix and match sizes easily in Fly. So you can't say, create like application groups and this group will be like the small group and that group will be the big group and this is just one application it's it's not straightforward So you have to again this is how i solved it maybe someone listening to this will tell me hey you're wrong i would love to know that seriously so the way i solved it is we deploy in all the regions Right because you specify the size once so say my starting size is the large instance type it has a certain number of cores certain number of memory and by the way the disk is the same in All of them because that's like another another problem so we will sidebar that um or put a pin in that so when it comes to the initial deployment you deploy the one size across all the application Instances and then you go and need to check to see which instances should be scaled down so that you have the capacity but the regions that don't need the capacity you can just bring them Down and you do like a rolling deploy in that you replace one for one you have plenty of capacity to handle the traffic while instances are being rolled all that good stuff but we have hot Regions and we have cold regions and there's quite quite a few things here again if someone knows how to do this better i would love to hear about that and we have the toml we have the primary Region there's a couple of things here we'll come back to services and http services that's that's a fun one we'll leave that for a little bit later fly just we can see how we do the fly ctl Deploy we disable ha because we want only one instance per region we have 15 regions in total we specify the cpus the memory all the good stuff including environment variables oh that's Another thing we need to adjust the varnish size based on the memory the instance has right we need to say like hey varnish you get 70 and that's the other thing that this does same thing For the file size you can't take up the entire disk would tell you based on the disk that we provision how much space you should use from the disk that gets created. There's a scaling there. So that's another good one. I'm going through pull requests. Is there anything else? Oh, man, this was a pain. So recreating, like writing tests for this. Everything is tested in the sense that which requests would go or which, basically, which files would get cached in the file store and which files would be cached in the memory store. So how do you write the tests? Some varnish logging is included. You have to have anchors. There's quite a few things. So that's assets backend.vtc. And part of this, it was a huge refactoring. So if you look at the lines of code i wouldn't say it's that many 1 500 were added and 1470 were deleted so not much changed i mean the net is 30 new lines were added but there was like a huge Massive refactoring part of this uh so there's again again, this was, I think two, three days of like figuring it out, trying things, refactoring things. And if you think that an LLM can help you, well, you try this. And it takes longer to go through those iterations. ([Time 0:34:24](https://share.snipd.com/snip/bc0159b1-4536-4363-bb81-504179719c85)) - **Cache As A Front-Desk Shield** - A single Varnish instance can shield backends massively, serving terabytes while keeping backend hit rates high. - Observability reveals which regions are "hot" so you can right-size capacity regionally. Transcript: Gerhard Lazu We're looking at the Fly, the Grafana dashboard for our Fly application, the instance of the Pipedream, the current one. And we can see that SJC, San Jose, California is a red, nice big red circle, which means it has the most traffic. And also NRT, which which is Tokyo. Hmm. Adam Stacoviak Apparently. Gerhard Lazu We're big in Japan. Yeah. And Europe, there's quite a few. So if I'm going to pull this down a little bit, let's see. No, I wanted to go here. What about our new continent? Are we big there? The new continent? Australia? Adam Stacoviak Oh, there's a new one. There's a new, new one. Well, what's it called? Which is a new one? I don't know. There's a i heard i thought y'all would get the joke no over the holiday there was speculation there was a new continent being announced narnia maybe could have been narnia no no no so Gerhard Lazu The closet right now if even this loss even this list is basically if you think about it it kind of makes sense right it's us east us west europe but we have quite a few instances in europe We have four it's it's more geographically spread in europe and we have asia so these are like the big ones australia africa and south america they're not as busy they are like least busy Regions cool so which instance would you like us to have a look at so i have a queue right here sjc baby let's go let's go sjc baby all right let's see that so i'm running flyctl ssh console I'm using two flags dash s which is a short one for dash dash select it will prompt me which instance i want to select and then i have dash c capital c it's different than lowercase they do Different things i give it the command to run and it's varnish stat dash one which will give me all the statistics from varnish at a point point in time so since this was running i will select Sjc there you go and it will give me all the data which is like all the counters that varnish is incrementing is keeping track of different things of the origins uh backends the memory Pool the disk pool the lock counters there's so much stuff i'm really really impressed how many things varnish has so this is what we're going to do we because ai right we're going to copy All of this and we're going to ask ai what it thinks of this how about that okay it's just too much data here so let's be serious about it so question to you which is your favorite ai jared Jerod Santo Which one do you use oh i don't like any of them um i would probably start with claude and then i would go to grok and then i would go to chat gpt third okay so claude which one which version Gerhard Lazu Which model opus man give us the opus okay so we're looking at abacus.ai something i've been using for a long long time It allows you, I'm only paying $10 per month for it, not sponsored, You know, not affiliated in any way. It's just something that I've picked for myself and I can basically pick any model and I can just run this. So I have something prepared. So I'm going to drop this. It's all the data. And we're going to read through something that i prepared ahead of time you pre-prompted this i pre-prompted this exactly okay because engineering this prompt for exactly the problem Not really but that's a long prompt so we're going to read it and in the meantime adam will think about his favorite llm to try and i have mine so we'll try three llms to see what they say so I'm going to read the prompt now while everybody thinks, no, we should be using whatever LLM you should be using. You are a Varnish 7 expert. You need to prepare four distinct responses and be explicit about the person that you're addressing. One, a seasoned sysadmin that has been living and breathing infrastructure for the last 20 years. Be precise, think deeply, and approach the setup from a hardware perspective. Two, an Elixir application developer that embraces Erlang's let it crash concept. You need to give it straight, give it fast, and keep it relevant to their application. It's the app and the nightly backends. Assets and feeds are important, but less relevant. Cloudflare are too. Three, the business person that is selling this thing. They care about costs, efficiency, and simplicity. Keep it high level and relevant for someone that doesn't care about the tech, but cares about the outcomes. And four, the audience of a podcast where this is being discussed. Make it general, relatable, and fun. Make analogies, keep it light and engaging. I have fun too many times. We don't want to make it too fun. That's a lot of fun. So yeah, there's one too many funs. That's right. Now that you understand your audience, please analyze the following Varnestad output for the SJC. Look, I already knew that you would pick. How do you know? Jerod Santo I'd go for the big one. Gerhard Lazu I have no idea. Focus on things that work well, things that could be improved and anything else that you find interesting. And by the way, ignore the synthetic request. It will keep mentioning these like, I get so fed up with this. We have health checks that run every five seconds. So they are normal. Okay i'm going to copy this i'm going to run this and i'm also going to open a new window for adam so which which should we pick adam which is your favorite you mean model model yeah which Adam Stacoviak Model we just used it but uh i'd probably back up to like codex codex which is like gpt5 latest gpt5 5152 there you go so gpt codex my favorite one is gemini so i'm going to drop it and let's Gerhard Lazu See how do they compare you're in a different tab now so abacus can't do gemini uh it it might but i have like my own pro account so that's something else like i use i use quite a video i use Nano banana quite a few things transcripts it's all like part of the of the package sure so it can but that's what i prefer cool so claude opus 4.5 for the season for the season system this Is you this is me this is me exactly thank you for noticing who's who i'm following so what's working well rock solid stability so by the way the instance has been running for 5.4 days we Had like all these improvements uh shipped and we are able to observe how our busiest instance works and that's what this is basically that was uh the window moved cool so after four 5.4 Days zero child panics crashes zero threat failures this is important it means no threads died no threads had to be restarted everything is healthy on this instance it didn't crash So this instance didn't crash zero lock contention across all subsystems your cpu cache lines are happy ([Time 0:39:25](https://share.snipd.com/snip/45a017c0-a849-4449-acd4-16429dfc8ec8)) - **Communicate For Your Audience** - Tailor explanations to your audience: sysadmins want hardware detail; devs want app-focused fixes; execs want cost/outcome summaries. - Use relatable analogies when addressing a broad podcast audience. Transcript: Gerhard Lazu For adam so which which should we pick adam which is your favorite you mean model model yeah which model we just used it but uh i'd probably back up to like codex codex which is like gpt5 Jerod Santo Latest gpt5 5152 there you go so gpt codex my favorite one is gemini so i'm going to drop it and let's see how do they compare you're in a different tab now so abacus can't do gemini uh it it Gerhard Lazu Might but i have like my own pro account so that's something else like i use i use quite a video i use nano banana quite a few things transcripts it's all like part of the of the package sure So it can but that's what i prefer cool so claude opus 4.5 for the season for the season system this is you this is me this is me exactly thank you for noticing who's who i'm following so what's Working well rock solid stability so by the way the instance has been running for 5.4 days we had like all these improvements uh shipped and we are able to observe how our busiest instance Works and that's what this is basically that was uh the window moved cool so after four 5.4 days zero child panics crashes zero threat failures this is important it means no threads died No threads had to be restarted everything is healthy on this instance it didn't crash so this instance didn't crash zero lock contention across all subsystems your cpu cache lines Are happy excellent hit ratio 93 we like that we really like that we have back-end connection pooling with a two to one reuse ratio and memory pressure is minimal 132 lr use in in the last Five days lre nukes so very few objects had to be removed from memory thread pool property 300 threads zero queuing zero drops that's perfect areas to investigate disk storage allocator Failures we have disk c fails we are hitting storage fragmentation the disk is 97 97 full we have 48 gigabytes used that's how many mp3 files are stored by the way how many mp3 files total Jerod Santo Do you think we have size or file size size size well if we had a thousand episodes at 100 megs each which neither of those things are true that'd be 100 gigs right so 100 is too big but a thousand Adam Stacoviak Is too small i'm gonna say 80 gigs adam do you want to guess that math checks out i was gonna say like a terabyte but that's probably raw wave files versus not but all the files that we store Gerhard Lazu In R2, and this includes all the assets, but we know that the MP3 files are the biggest. It's close to 250 gigabytes. We may have some duplicates. I don't know. I haven't checked, but that's how much files we have in R2. Jerod Santo Yeah, well, we also have plus plus for the last couple of years, which means every episode has two files, not just one. So that makes sense. Gerhard Lazu So we should go higher. Now, we use this in every single region. So maybe we want to reduce number of regions. But I think- You need a third category called super hot. Super hot, yes. Which is like SJC in Tokyo, right? That's possible, yeah. There's four, which we know they're really, really hot, yeah. Yeah, but honestly, this is happening across multiple regions. It is. We'll get to some interesting things. So, okay, synthetic responses, grace hits, all good. For the Elixir developer, and I think this is you, Jared. Do you want to read it out? Jerod Santo Oh, well, the TLDR is varnished to do its job. Your app backend is well-protected. You want me to read the whole thing? If you want. I mean how it's shielded it's 95 shielded uh no failure zero back end failures that's because of you know my code doesn't really let it crash very often exactly your code is yeah it crashes Internally not externally that's right my thing is doing its thing It is generating some uncashable responses, but you know, we do have some that we just don't want to be cached. Ooh, unfetched failure, negligible. Yeah, I agree. You don't need to worry about that. And in the end it says, whoever wrote this is really good at what they do. I agree. That's exactly what it says. And congratulations on such a great hire. Gerhard Lazu Yeah, I agree. ([Time 0:44:48](https://share.snipd.com/snip/c2e634c4-0bb8-43ba-aa98-4b0e5c809d9e)) - **Connections vs Requests Misconfig Bite** - A misconfigured Fly.toml used 'connections' instead of 'requests', causing many long-running connections and hangs in Newark. - Changing the config and adding hourly checks fixed most issues and exposed HTTP/2 quirks. Transcript: Gerhard Lazu Oh, wow. This is like when all the fun begins. So you remember this, Jared? Jerod Santo Yes. MP3 requests intermittently hang in Newark, New Jersey. This was our good friend, John Spurlock, who's been on the show before and is a podcast nerd. In fact, he runs op3.dev and other podcast nerdery things and so he really knows his stuff and so when he reports issues uh you know i don't say did you try rebooting i take it seriously Gerhard Lazu So i shared it with you and he actually did some additional digging for us go ahead so in terms of you tested this i think you had issues as well so we've confirmed this for sure i did like Jerod Santo Certain times certain files actually it'd be all requests at certain times i assume that that was that particular pop as we could call them or pipe in the pipe dream was hanging and then It would go away and he actually had the same problem he had a friday night deploy of friends and he's trying to listen to it on friday couldn't get to it by saturday morning he can get to It so it's intermittent hanging very difficult to diagnose very difficult i assume to debug and then it just comes back to normal i thought it was maybe the out of memory thing like it's Just in some sort of fugue state until it reboots and then it works again but you go ahead that's what i thought that's why like a deep dive on this this was november end of november beginning Gerhard Lazu Of november so november i was just trying to figure out what on earth was going on just like you know like from the sides i didn't have too much time but if you look at this response there's There's quite a few things there uh this is like my initial one like an investigation trying to understand what's happening giving a couple of uh debug headers like a couple of extra Headers that the request can be made of sorry can be can be um run with so we just get a bit more details forcing regions as well so there's quite a few things there i was checking into that This is uh don mckinnon um he also had issues today um so he pasted some results so thank you thank you don for adding this this was this was helpful so this is long i'm still scrolling i'm Still scrolling. There we go. It's super helpful. I have confirmed. So I've confirmed that the requests have been hanging. You are getting the hangs this afternoon as well. This was only three weeks ago. So this has been going on for a while. I dug deeper and I found the problem. The problem was that in the fly config, we had the concurrency set to connections, not requests. So it's possible to configure an application. Again, you're configuring the fly proxy that sits in front of the application to limit how much traffic hits your application. Requests how many requests per second should the fly proxy forward to your application before it stops like because you want you don't want to get overloaded right so before it like Starts throttling it starts slowing clients down and then when you start that's when you start seeing fly edge errors connections you would use for something that has long running Connections, like a database, for example. In our case, it's not a database, right? It's an HTTP application. So requests would have been the right concurrency. I have no idea why I picked connections. It was the wrong one. But the effect was, as you can see here we had 2700 long running connections on that edge so on that region so this in this case it was i think orange one i think ewr right so ewr was getting Had like all these connections opened the clients weren't closing the connection, the proxy was full, no more connections could be forwarded to the application. Long-running connections, there are usually clients which are not doing the right thing. You shouldn't have that many long-running connections. So the problem was a misconfiguration on our side, which meant that connections like slow connections, long-running connections, were basically blocking other connections from Coming through. So that was a problem there. And I thought that was it, but there was more. ([Time 0:58:54](https://share.snipd.com/snip/4a0a0d9c-dd9b-484a-84c2-3b29c40e1d27)) - **HTTP/2 Can Break Proxy Body Forwarding** - HTTP/2 client-to-proxy behavior can break body forwarding while headers appear fine. - Forcing HTTP/1 fixed observed body-timeout issues in specific regions. Transcript: Gerhard Lazu So this is the last comment last week. We now have a check that runs every hour. And what was interesting, and I'll talk about the check as well, we had response bodies timing out in two regions. So 13 regions were fine. But even after this configuration, there were two regions. I had an EWR where when we were using HTTP2, and for some reason this is important, when we're using HTTP2 and the proxy, the fly proxy would see this, it would not forward the connection Correctly. As in, it would start, it would like serve the response, like we could see the headers coming back from our instances, what we wouldn't get is the body. So the body would always be like zero bytes served. And we could see this happening. We could see the connections that, by the way, they were opened. They shouldn't have been opened because the application changed. So these connections should have been dropped. There was something not quite right. My suspicion is with a fly proxy layer, because when we were forcing, when we were forcing HTTP 1, everything was working fine. And by the way, the fly proxy, when it talks to our varnish instance, it's using HTTP one. And you can see that in the headers. So the proxy to the varnish was fine, but the client to the proxy was not fine. And HTTP 2.0 is a very complex protocol. There's so many things which just don't work the way people would expect. So anyway, the issue fixed itself. That's the important thing. So, opening this. Not super satisfying. Yeah, that was very nice to see. And there was something, Mya Ilerus? How would you read this? Mya Ilerus. Ilerus. There you go. So, someone on the Fly community forum that was very helpful, they noticed that we had a misconfiguration in our Fly tunnel. And we were using services as well as HTTP service. And this is bad. By the way, this is very, very bad. So everything was happy. Like we could push this config. You know, the applications were running. Everything was fine. But because we had these two things together it was apparently creating some issues and all we did we were explicitly setting the idle timeout and the idle timeout that's the one where If after 60 seconds the connection isn't doing anything it will be forcefully terminated by the proxy so that part was important so anyway we made the change, we pushed the change, but Even before we pushed the change, the proxy started behaving. And now there's pull request 49 has like, we right-sized it. We, we, we made a few changes. I captured like all the details, the configuration, the commands, it's all there if you want to read it. But most importantly, now we have a check that runs against all regions every hour on the hour cicd it's using hurl and what i'm thinking is shall we try running that locally to see how It behaves because that's how we started it like i was starting um to do it locally so on the left hand side i'm back in the terminal. On the left-hand side, I am monitoring my internet connection. Remember that Christmas tree? This is related to that Christmas tree. So I'm at the top of the Christmas tree. I'm at the gateway, the core router. It's a Microtix CCR 2004. Pretty good. 10 gigabits per second maximum. Now, my internet connection isn't 10 gigabits but it's 2.5 which is plenty for this test so we every second is showing me how many packets and how many bits we're receiving and transmitting Okay and again we are recording everything's happening live so you can see jumping right as riverside we pushing more data to Riverside. Cool. So I'm going to run now, just check. And just check by default. It's one of the commands, the just command that we have in the pipely repository. And check all it does. It runs hurl with a couple of flags. It downloads an MP33 file it downloads feeds it basically connects to all the different backends and it sees how quickly it can get data back we're transferring about that was quick Those eight seconds i'm going to run it again as i run this pay attention to the left hand side it will go to 120 megabits per second so that's the mp3 file being downloaded so every single Time this runs a full mp3 file gets downloaded alongside a few other things okay i can open the reports we're not going to look into that because we're going to run something more interesting Now we do check all and what check all does it runs the same command against all the regions i'm at 2.3 gigabits per second we're downloading all the files we can see the response is coming Back ewur just sped by i had sped by so all the different endpoints are returning now i'm based in london obviously the further away you are so for example this was south america those Lax so a couple of instances are slower to respond and all this happens via headers. So you can, when you connect to fly, you can tell it, Hey, I want to connect to a specific region. And then that's what routes the request to that, to that, to that region. That's cool. And again, it's all captured in that pull request and you can see what it looks like. ([Time 1:03:26](https://share.snipd.com/snip/cb224757-fcda-4c4e-9006-2e0d4ae90d42)) - **Episode 456 Became A Download Juggernaut** - Episode 456 (OAuth) unexpectedly became massively popular and generated sustained megabytes-per-second downloads from Asia. - Thousands of distinct IPs repeatedly downloaded the MP3, causing tens of terabytes served per region. Transcript: Jerod Santo Oh, just getting rocked with downloads. Gerhard Lazu So episode 4456, OAuth, it's complicated. By the way, this was recorded in 2021. It was published again, August 2021. For some reason, it's been downloaded a lot in recent months. It has over 1 million downloads this is the most popular episode on the changelog ever the most downloaded episode it's crazy it's crazy so oh so you guys looked into this we did yes we We dug into this okay i didn't know you guys were doing this so we just had a quick look to understand what is happening here um so we have honeycomb open up remember every single request Which comes through the pipe dream through pipely every single request we sent to honeycomb we're able to look at it this is the last 60 days and i have filtering done in such a way so that I'm only looking at this one file how many times has this file been downloaded in the last two months and you can see the peaks right you can see and by the way this is gigabytes so and this Is the period is four hours so we are peaking at about a hundred actually it was like this peak was here we had 200 almost 300 300 400 anyway close to 400 gigabytes in a four-hour period it's Just too much i think so like i know this i know this is a great episode great conversation but i remember that conversation it was good like who is downloading this file 400 i know times Or actually more than 400 times every four hours consistently for months on end and a fan a super fan so we can see like a different regions now this is spread across the entire world it's Not just one region this is really really big i think if there was a ddos attack i think this would class this one and uh like in the last six months in sorry, in the last two months, 60 days, We served 30 terabytes in San Jose, California alone. In Tokyo, we served 15 terabytes. This is a big number. And if you look in this column, the distinct IPs, the client IPs, we had over 10,000 IPs downloading this file. So this is not one or two IPs. This is thousands and thousands of IPs which keep downloading this file over and over and over again. So I don't know how we would block 10,000 IPs. Right. That would be, the VCL would be crazy. Jerod Santo Well, that episode was starring aaron parecki who is a very talented person and he is the co-founder of indyweb camp and a big fan of the indyweb as well as oauth obviously so my hunch is Aaron's very interested in being the most downloaded episode ever and he and he controls a fleet of machines from all around the world and he points them wherever he wishes and he thinks You know i'm gonna do i'm gonna get the number one spot on these guys download charts and so i'm thinking aaron parecki is you know the man with the mask on we pull a mask off and it's him this Whole time what do you think you're hard i i think that we need to speak. Gerhard Lazu See, I don't want to say the specific language. I think we need to go to Asia. I think we need to visit a couple of cities in Asia. Okay. Find the IPs, which are responsible for this, because this is a crazy amount of traffic. Asia, it just so happens if we look at, so Asia is basically the continent where we are getting the most downloads from because of this one episode and this is actually traffic being served This is not like head requests or get requests these are bytes being sent to thousands and thousands of machines in asia every single hour so whoever is doing this please stop. Jerod Santo It's on a cycle. So we need to like knock on doors. We need to go over there and knock on some doors. And say, excuse me, is this IP address at this home? And then they might say yes and say, would you please stop? And what's going on over here? What could they possibly benefit from this? Like, what could they be getting? Gerhard Lazu Maybe, maybe we're the speed test someone is using us to speed test their connection who knows yeah maybe that's the only thing i i can imagine but that's a lot ([Time 1:14:11](https://share.snipd.com/snip/d1e66963-a5ed-409a-bdb3-e2105361a214)) - **Throttle MP3s Before Blocking Regions** - Throttle or apply VMOD-based IP limits for MP3 endpoints to protect bandwidth without blocking entire regions. - Start by throttling specific URLs and monitoring false positives before blocking netblocks. Transcript: Gerhard Lazu Getting? Maybe, maybe we're the speed test someone is using us to speed test their connection who knows yeah maybe that's the only thing i i can imagine but that's a lot of ip addresses it is and It's across multiple regions which multiple data centers yes so multiple regions fly regions are IPs, yes. They're all coming from Asia, by the way. Again, I don't want to mention any names because there's no bad guys here, right? We just want to assume that someone left the oven on. It's like the blinker on when you're driving. I was like, hey, you're not turning. It's time to turn that blinker off. So the way I can see us mitigating this and this is a hard problem because of the number right of ips which are hitting is we can basically start blocking entire net blocks entire network Blocks unfortunately some genuine listeners might be caught in this and basically a changelog will not be available or at least the mp3s will not be available to a portion of users the Other one is obviously we can we and we should right this is like the next problem we should enable some throttling because there's more stuff happening here so we don't have any sort Of throttling we assume fairness we're assuming goodwill we're assuming decency and we're not seeing that here that's the internet so to be honest like whoever is doing this and it's Not llm so i had look we have we have that problem as well but in this case it's not llm this is something completely different so my hope is by someone that listens to this episode maybe We put this in the intro whoever's downloading episode four five six please stop because we'll need to take the next step i know it's a bit of a cat and a mouse game but that's what will need To happen because we need to pay for this bandwidth this is only varnish right this is this is only the cache layer this is happening this is only the cache layer yes yep and so what mechanisms Adam Stacoviak Are in varnish to to do throttling or rate limiting or just anything like that whatsoever? Gerhard Lazu There's VMODs, which are basically modules that Varnish loads that just give it extra functionality. One such VMOD, and I've looked at this, it is free and open source, is the VMOD throttle. Now that means that we need to start keeping track of IPs and it will use a bit more memory. That's okay, we have more memory. Then we can need to start basically applying limits to how many download specific ips can do and we can limit it to mp3 files only so if we have a bot or if we have for example like i don't know An rss aggregator or something like that we can we we're okay serving those requests because again that's what varnish is meant to do the problem here is that we're serving a lot of bytes For MP3s, the same MP3 that cannot be real traffic. Adam Stacoviak Yeah. I mean, even in this case, you can like tie it potentially just to this MP3, like you just said, which is not an all MP3 scenario. Like if you request this MP3 with this kind of like request signature of X per whatever, I mean, I didn't examine the actual signature of the request but that's how i'd probably investigate It is begin to isolate yeah does that require us to write a lot of defensive code against that kind of scenario i don't think so it's just some configuration we just need to add more configuration Gerhard Lazu And back to jared's point we're just chasing now new problems that we didn't even think we would have um but we have what looks to me like an actor that's not very um i want to say this in a Nice way an unfriendly actor that is not very happy uh and they you know are very angrily downloading our mp3 over and over and over again thousands of times across thousands of ips and This is not cool because ultimately we end up paying for this bandwidth that is not helping anyone but that's one it's not the only one so we have one more so you can see here for example Um this is the last last seven days we have seven terabytes that were transferred in the last seven days seven terabytes maybe that's more than that it needs to be more and did actually Geoco does not exist okay i was expecting to see more than that anyway asia is the one that we can see like like that pattern but we also have like in Europe, sometimes we have these spikes And it's like this spike, which I wanted to focus on. We know that someone in Frankfurt that connects to Frankfurt downloaded the static favicon 170,000 times in the span of, I don't know, like an hour or two. So they downloaded this like two, three hours. So it gets, you know, requests like this that are putting stress on these instances. I mean, what potentially, that was like a pass request as well, which means it like went through the cache, which means that they must have had like a cookie set up or something like that, That basically was preventing the cache from working in this case, which again, that's how it's supposed to work. So anyway i think i think that was unfortunately not the best thing that we could have ended on but it's a thing and it's something food for thought like more work to be done there's many Things that we didn't get to talk to we didn't have time for for example we didn't talk about the nightly by the way nightly now is being served by by the pipe dream as well um and the reason Why we had to do this because that sometimes will get scraped would get hit really heavily um it's a very small app it's nginx but um if i open it so let's just click on that one and that's Pull request 46 before it was basically topping up at 141 requests per second now it's 1300 so it's almost like a 10x in order of magnitude faster the latency went way way down so and it's Just the only thing we had to do is basically put varnish in front of it nice well that's nice yeah that's one more thing there and you can go and have a look how it works. It'd be like a benchmark here, a small benchmark here. That's it. We have, we have last one for the road, but before we do that, anything else that we want to talk about before I share one last thought? Adam Stacoviak I suppose what we do, you know, if we know these downloads are happening, we're here on the podcast, just politely asking to stop. We just let it keep happening. Gerhard Lazu Well, we could, we could set up some sort of throttling. I think it'll be the easiest thing now, which will impact everyone. I don't want to start blocking again, IP ranges, net blocks, because we don't know who's going to be caught there. They may change to other IP blocks. So that's entirely possible. We don't know how this will work. We can't block an entire country, an entire continent, especially if it's a big one. I don't think that's reasonable. So really throttling is, I think, the fairest thing. And then we can throttle MP3 specifically. Because we do have, for example, I ([Time 1:18:30](https://share.snipd.com/snip/05827b53-07c6-4766-abec-b3089cd23d2b)) - **Use Observability To Find Scrapers** - Use observability (Honeycomb) to group by user-agent and filter by MP3 URLs to find scraping clients. - Flag outliers with thresholds (e.g., >100 downloads) to identify abusive agents quickly. Transcript: Gerhard Lazu It's honeycomb yeah you can filter by user agent for example and you can see that like there'll be on for example say um now i don't want to show any ip's or anything like that so that's why I'm not going to screen share that. But once we start digging into that, you can say group by client, by agent, and you can say filter by MP3s. So like URL contains MP3. And that will be able to group. And you can say, oh, and by the way, only show me where there's more than, for example, 100 downloads. And then you'll start seeing like the outliers, which are the clients that are downloading certain MP3s or MP3s in general excessively. ([Time 1:26:20](https://share.snipd.com/snip/e7a58544-cf85-4a94-a30e-f346903276ef)) - **Traffic Patterns Changed By Bots And LLMs** - The rise of LLMs and bots changes traffic patterns, creating unexpected high-volume spikes and odd request behavior. - Platforms must adopt defensive, fair protections as default to handle non-human traffic. Transcript: Gerhard Lazu And then you start having to set up like some form of protection for all sorts of clients that are just doing the wrong thing. You need like a defensive layer by default. Exactly, yeah. And something that would be fair to regular clients like for example when i want to do like a benchmark i mean sure it's me and i wouldn't want other people to do that because i'm testing The system making sure like the real world the production system everywhere in the world is working correctly and i'm aware of what that means and how it costs and by the way my ips are Removed from all like the stats because otherwise you see like those massive benchmarks. So we account for that. But we can't account for all these like weird clients. It's a challenge. ([Time 1:33:19](https://share.snipd.com/snip/685c26ab-f793-4d5b-a1eb-735f60018b93)) - **Home Network As Kaizen Project** - Gerhard iterates his home network annually, pushing latency down to 2.4ms and scaling WAN capacity toward 100Gbps readiness. - He treats hardware tuning as a personal Kaizen project for performance and resilience. Transcript: Gerhard Lazu My one last thing yeah so keep it short we'll keep it fun i mentioned about the christmas tree i mentioned about the various things which i had going um over the holidays so make it to work Club that's the place you're there both of you are there so you can join whenever you want next thursday yeah next thursday i'm going to talk about the 100 gigabit wan the 100 gigabit wan So why would i need such a thing smoking it's smoking for sure so i thought the ccr 2004 has like four cpus has like multiple 10 gigabit sfp plus ports it even has 24 sfp 20 sorry sfp 28 ports But it doesn't have a switch chip and people that know about a little bit about hardware you want to switch chip to the hardware offloading l3 and even l4 so after i bought the ccr 2004 it Was like almost like a christmas present i thought surely this will be enough for the rest of my life and no i had to get the flagship so i'll be talking about that um the land the setup quite A few things coming up and um it just goes to show how much i enjoy the hardware side of things as well the networking side of things like i shaved two milliseconds of my wan it's a basic like Like little things like like that you know like i was already good it was like already like sub five milliseconds but i wanted like sub three milliseconds it's now 2.4 milliseconds so The next and what it means like why would i do this so first of all i'm all about improving and every winter i improve uh the network in this specific instance i wanted the pages just to be Snappier like things to load a lot quicker to handle a bit more traffic but also to not have an impact like i was running that benchmark like 2.5 look i'm going to do another one right now Let's see i have a speed test i have speed test right here speed test uh london let's go for this one so we're recording we're streaming right and i'm just pulling 2.5 2.6 gigabits down And there's no interruption on my network right so it's just my bread and butter you know that's how i work and by the way if you see any buffering or any any slowing down let me know i see Adam a bit more pixelated maybe you can see me pixelated too i don't know but uh yeah i just pulled six gigabytes three down three up and it's just just what i do every day which i work with This stuff and yeah i enjoy it and by the way this is this is a the slower gateway router so i'm getting the proper one set up and i'll talk about that um and there's so many things there like Vlaning is quite a thing i have a new ipv4 block by the way so some would say that i'm preparing for hosting something and maybe i am i don't know we'll see how that how that works but i just Realized that my home connection obviously i couldn't serve all the mp3s that were being dialed like that would really cripple my connection if that was happening but i'm at 2.5 gigabits The next one will be five gigabits and the hardware can do it and then five gigabits i mean that's like a decent server and you can do five gigabits all day every day sorry yeah gigabits Yeah gigabits per second That's pretty decent. Jerod Santo So I'm just waiting for more internet. I was going to say you're going to have a hundred gigabit WAN, but you're not going to, you're not going to have a connection for it. Right. Right. Gerhard Lazu So very few places in the world have that. So if I was in Switzerland, I would get 25 gigabits. Now would you move? Would you move for this? Of course. The only reason. The move is, yeah, it's the 25 gigabit connection. But I know that a hundred gig is coming. So we'll see. Are they either ship it by the time I move or I come, I move and then they ship it. So it's one or the other. The important thing is I have the router to handle that. You'll be ready. You will be ready. I'll be ready. Exactly. So I'm a prepper. I'm prepping for that. Prepping for good internet. And this is just like, and interestingly, five years ago, when I got like the previous router, I did the same thing. It's like a forum post. It's like a follow up. So I just did a follow up recently at this milestone. ([Time 1:34:56](https://share.snipd.com/snip/4f092d6d-f3a9-459b-ba2d-c75a85a53bbc))