Kaizen! Pipely Is LIVE  ## Metadata - Author: [[The Changelog: Software Development, Open Source]] - Full Title: Kaizen! Pipely Is LIVE - Category: #podcasts - URL: https://share.snipd.com/episode/f54d6080-d2fb-4f4c-973f-30d7f53f75a3 ## Document Notes ## Highlights - **Origin: Kaizen As A Public Improvement Ritual** - Kaizen began as a ritual to improve publicly every ten Ship It episodes. - The hosts used it to share progress and inspire others by iterating in public. Transcript: Jerod Santo It was a long time ago in Japan. Yeah. I don't know. How did our Kaizen begin? 1,300. Gerhard Lazu That was a year. I wasn't there for that. Something like that. It began with this crazy idea that let's improve, but in a consistent way, so that every 10 Ship It episodes, we'll talk about the improvements that we drive in the context of changelog. Share them publicly, and that was our reminder that, hey, 10 episodes, we'll be talking about everything that we've done. Some called it navel gazing. I don't think that's what it was. That was me, probably. Jared. It was good fun. And, yeah, it's been four years. It's been four years. It's been four years. Jerod Santo It's been longer than four years. Has it? A decade. Gerhard Lazu We've been Kaizen-ing a decade. We just started calling it Kaizen. We've been calling it as a Kaizen. So Kaizen 1 was about four years ago. Adam Stacoviak Kaizen as it is materialized roughly four years ago. But the relationship, the reason why I bring that up is I want to say one thing. It's like these guys are my ride or die people. Gerhard's amazing. Jared, you know you're amazing. But like the magic and the beauty that's come from this relationship has just been tremendous. And to be here and share that with you all and to share Kaizen 20, this navel-gazing approach to our platform, this constant attention to detail of improvement. ([Time 0:02:23](https://share.snipd.com/snip/72208eac-18be-4c02-86e2-ca81494c736c)) - **Pipely Started As A Pipe Dream With Friends** - Pipely started about 18 months earlier as a pipe dream to fix cache problems. - The team built it with friends and shared the process publicly to help others learn. Transcript: Gerhard Lazu Pipely was, I think, 18 months ago, roughly. The idea was, shall we do this thing? I mean, we've been talking about it long enough. Shall we actually try doing something about it? And the beginning was, as all beginnings, like, can we do this? How long will it take? What will it take? Do we know what even needs to happen? And that's how the conversation started. And many of you that have listened to those conversations remember us, right, how we were pondering, should we, should we not? Are we crazy? Three wise men, the question mark was really the emphasis, like, are we wise doing this? We have no idea. And then along the journey, the best part was the friends that joined us. So it turned out there was not such a crazy idea. And just the idea of improving something in public so that others see how we do it. And it's our approach. And maybe it will inspire others. And I think that worked really well. And here we are today with friends. And that's, again, that's the focus. ([Time 0:04:31](https://share.snipd.com/snip/5b7a2f3c-3f66-4086-801c-0d954bde4a38)) - **We Use Open Source Varnish, Not Fastly's Fork** - The team clarified they run the open source Varnish cache, not Fastly's Varnish Plus fork. - Building on open source was intentional because it fits their values and tooling needs. Transcript: Gerhard Lazu So, Andrew O, he wanted to set the record straight. No, Allgreen, sorry. Allgreen, May 13th, he wanted to set the record straight about Varnish, which is the technology that we use, the relationship to Fastly and the relationship to Varnish Enterprise. And it's all here. So, Fastly is not running Varnish Plus, which is a Varnish software product, and they have their own fork. So that is important. It's similar, but not quite the same. What we use is nothing from Varnish Software, not Varnish Plus. We're using the Varnish cache, the open source project, as anyone can consume it. So we have built, everything that we built is on open source technologies. And that is important to us because that is in our DNA. So there's some history here that's important. So we are building on Varnish cache open source. ([Time 0:12:49](https://share.snipd.com/snip/43c39549-2a61-406f-a09c-58236aa38172)) - **Low Cache Hit Rate Was Causing Slow Pages** - The homepage had a 17.93% cache hit ratio, causing many slow backend requests. - Low hit rates increase latency and failure surface area, so improving caching was essential. Transcript: Gerhard Lazu It's a bad thing. Okay. Color, psychology of color is very important. So what is this? The 17.93% is the cash hit ratio on our current production CDN, which is low. That's really low. It means that less than 20% of the requests get served really fast, 80% plus are slow. And while it doesn't really impact us, I mean, it's all good for us. It impacts you. When you load something, it takes a while to load. And it shouldn't be that way. Right? Things should be instant. Things should be very, very smooth. And when something goes wrong in the backend, for example, if 80% of the requests, they have to go back to the backend, it means that they can fail. So the chances of something failing are fairly high. There's something wrong in the back. And then that's the other thing, which I keep thinking about a lot. Yeah. Jerod Santo And it's worth noting that our data is unidirectional. I mean, we rarely change anything once we publish. Adam Stacoviak Episode one is still episode one. Yeah. 19 years later. Jerod Santo You know, we might put the wrong audio in the wrong episode, or we might have to edit something. Or you ship the entire wrong audio, like I've done recently. Adam did recently. So we make mistakes, and when you make a mistake, you want to be able to quickly rectify it, purge everything, and get back to where you were. But generally speaking, you put an MP3 up on a CDN, and then you deliver that same MP3 in perpetuity. Yeah. And so this number is abysmal. That's terrible. We should never have misses. Yeah. And that's what I've been saying for 10 years. Right, 10 years. Why do we keep having cache misses? Okay. Gerhard Lazu And the problem is that in this case, it's mostly the website. So the changelog website appears slow in a lot of cases, and that's not great. So that was the problem that we were trying, or that was the thing that we were trying to improve. That's how this started. All right. ([Time 0:17:16](https://share.snipd.com/snip/78effbdc-5c27-4027-9a10-6e9052a5132e)) - **Night Crash: Varnish Ran Out Of Memory** - One Pipely instance crashed overnight because Varnish ran out of memory and crashed the app. - The team live-edited the Varnish memory config and deployed a fix on stage. Transcript: Gerhard Lazu So this happened about six hours ago or seven hours ago. This was 1 a.m. Last morning. Something went wrong. So things will continue going wrong. You will never really get there. It's all about the mindset of, can we do it a little bit better? And again, we are figuring stuff out. So this was yesterday, last night. So this is a question for the audience. Who would like to see us improve this specific crash live yeah yeah all right let's do it cool so um what do you think happened here just like let's do like a very quick um understanding Of what the problem is what do you think happened here can you describe the architecture of pipely Like PipeDream in terms of what's working out there? I can. So it's Varnish instances. I mean, Varnish is really at the heart of it. There's a couple of other components around it, but Varnish is the heart of it. Varnish makes requests to backends. The backends in this case would be assets, for example. We store static assets. We were mentioning MP3 files, PNG files, JavaScript, CSS, that kind of stuff, which rarely changes. Then there's a feeds back end. Feeds stores generated feeds for users, plus plus members, and shows, various shows. Jerod Santo Every show has its own feed, and then you can also create your own custom feed. So there's something like on the order of 600 to 800 feeds I would say. Eight of which are way more important than the others because they are publicly consumed by all the podcast indexes. Gerhard Lazu And those feeds get the most requests from all the platforms, the podcasting platforms that consume the changelog episodes and they distribute them to their audiences. Or to your audiences but through that platform. Correct. Our audiences. Our audiences, yes. Our audiences, for sure. And what this is, basically, we get these instances that are distributed around the world so that the delivery of that content gets accelerated. The one thing which I haven't mentioned is the applications, the application, the website, which is an important one. That's where many users go to, for example, look at the homepage, look at news, things like that. And that is the one which is most sensitive to latency because, as I mentioned, it's only in one location close to the database. So we need to accelerate delivery of that website to users which are around the world, including Australia, South Africa, South America, all over the world. We have a very diverse audience. And we want those users to have just as good experience as anyone else that's maybe in the North America. Jerod Santo So in this case, one of our ten? Yes. Ten instances of the Pipely application crashed. Gerhard Lazu Mistybird 4931. Yeah, Mistybird, that's the one. Mistybird crashed. That's what happened. Exactly. Okay, so now to your question to the audience was... Why do we think the application crashed? Why do you think it crashed? And anyone can answer except Matt, Nabil, and James. Jerod Santo Because they already know why it crashed? It ran out of memory. Why did it? Yes, of course. Thank you. I appreciate that answer. Gerhard Lazu Someone's paying attention, but why did it run out of memory? Why do you think it ran out of memory? Jerod Santo Because there wasn't enough memory. Gerhard Lazu Right. I love some trolling. Seriously now. You're going to make me say it, right? Okay. So as more content gets cached in memory, the problem is there's like a configuration which I wish it was easier to make, but you have to manually adjust how much memory you give to Varnish Out of the total memory available. So there's a dance that you need to make so that you know how much is enough so that when more memory gets allocated, the thing doesn't fall over. I wish this particular thing was easier, maybe something that we can improve, but for now you have to fine tune it and find, if you have four gigabytes of memory total, how much should Varnish be allowed to use? And if you think it's four gigabytes, it's way too much. So, that's what we're going to do now. I'm going to switch to some live coding and show what happened. So, actually, you know what? Let me, maybe let me try this. How's the font? Can everybody see the font? Can everybody see what's there? I'll make it a little bit bigger. A little bit bigger. Okay, so this is the change. And if I'm going to undo this, you'll see what it was before. So we, I thought, let me take responsibility of this, I thought that 800 megabytes is going to be enough. This was the case when the application had 2 gigabytes. So when the instance had 2 gigabytes, 800 megabytes of headroom was enough so the application wouldn't crash because of out-of issues. And apparently, when you have 4 gigabytes, 800 megabytes is not enough. So what happened is 33%. 33% should be enough for this to work. And again, you have to specify this explicitly. I'm sure that we'll improve this at some point. This is what this looks like. ([Time 0:24:33](https://share.snipd.com/snip/733002f8-4a40-4a5c-80e2-a0fc2dd92472)) - **Shorten Deploy Time To Tighten Feedback Loops** - Shorten the time from change to production to tighten your feedback loop. - Faster deploys let you validate fixes quickly and reduce the cost of mistakes. Transcript: Gerhard Lazu So the tag went out, and what we're going to see now is, actually this one right here, it's going to go to the actions. So this is a live, this one right here, 100RC4, and it's going to push this change into production across all instances. It's going to roll them live. We are deploying this. So why is it significant? Sorry. Jerod Santo Let me set the record straight. Sure. Bill Gates in 1996 said, I've said some stupid things and some wrong things, but not that. No one involved in computers would ever say that, that a certain amount of memory is enough. Right. So I guess I take it back. Is that true? I don't know. Is that factually correct? Gerhard Lazu I scroll further. What is your source of information? Jerod Santo Yes. So there you go. He may or may not have said that. So let's have... We're watching it roll out. Gerhard Lazu I see. So now we're seeing like a live roll out, and we'll see, so publish and deploy tag, is doing all the validation, gone, gone, it's moving on, it's going through the changes. Errors, errors, errors. Yeah, it's saying errors, but it's going to resolve itself. It just takes a while. Done. So there you go. Jerod Santo How long does it normally take, Gerhard? Gerhard Lazu Well, last time I think it took two and a half minutes. And you thought this was a good idea. Jerod Santo Of course, why not? Gerhard Lazu Look, we're already creating the new machines. Look at that. That's going. So this is something that I think is important to get right early on. And that thing is, how long does it take you to push a change into production? So how long does it take you to make a change and see that change roll live? And this is something that we've been working on for quite a few years on this thing. We do push to production. We own our production. We don't develop in production. That would be crazy, even for us. But it's like all like this mentality of if I'm going to make a change, how long will it take me to see that change happen? And if you can shorten that time, you're in a good place. Adam Stacoviak Ten years ago, it took 20 minutes or so. It was long. It was a long time. It was long enough that you go do something else. And now it's two and a half minutes maybe. Gerhard Lazu And this is a global CDN. ([Time 0:32:13](https://share.snipd.com/snip/7007c6a8-a6e7-42fe-9897-a1eac8503c8c)) - **Canary 20% First To Validate New CDN** - Roll features out gradually with canary traffic rather than switching everything at once. - The team routed ~20% of traffic to Pipely first to validate behavior safely. Transcript: Gerhard Lazu Team. Jerod Santo We did it. We barely launched it. We launched one out of five. One out of five, yeah. Yeah, so roughly 20% of requests go through our PipeDream at this point. Gerhard Lazu So whenever you go to changelog.com, that's it. One out of five requests hit the new instance. And we're able to see how well does it behave with 20% of traffic. And it's working. No one complained. Adam didn't even notice. And that's like one of the best things, right, like to roll things out. You do it in such a way so that, I mean, it is a big thing to us and to people that understand and know what goes behind it. ([Time 0:39:20](https://share.snipd.com/snip/77d501cf-265d-49dd-b696-9b7986c6ce8b)) - **Tiny Go Sidecar Handles TLS For Varnish** - Varnish cannot terminate HTTPS backends, so Nabil wrote a small Go sidecar to terminate TLS. - The tiny 'TLS exterminator' presents unencrypted requests to Varnish and simplified the architecture. Transcript: Gerhard Lazu So if the backend is talking HTTPS, Varnish cannot use it out of the box. Varnish Enterprise and other products can, but the open source Varnish cannot. We did a bit of digging and we realized, that's how we learned about Paul Henningkamp, PHK for short, and he was always against including SSL anywhere near Varnish because it would complicate Things too much. SSL is really, really complicated. So with Nabil's help, we wrote, I don't know, like he wrote 50, 60 lines of Go code that intercepts all the requests going to those backends, terminates SSL for Varnish, and presents The request unencrypted. A really simple, elegant solution, one of the almost like sidecars that sits next to Varnish, and helps it terminate requests which need SSL. Jerod Santo So I'm not sure if that's... Is that you justice, Nabil? Genius. Absolutely genius. Let's hear it for him. Nabil. Good job, Nabil. And his SSL termination. ([Time 0:44:20](https://share.snipd.com/snip/cd3c75f7-fb7d-4e1c-9c10-d75866b546a7)) - **Use Remote Dagger Engine For Reproducible Tests** - Use a remote Dagger engine to run CI tasks in a near-production Linux container environment. - Remote engines make tests and builds faster and more representative than local Mac VMs. Transcript: Gerhard Lazu Can we talk about how we're using Dagger? All right, okay. All right. I can show you about that. Yeah, show them. Cool. So there is one very tiny change that I have here, which is using dash dash cloud. This is an experimental feature that exists in the Dagger CLI. The reason why we do this is because we need a remote engine, a remote Dagger engine, so that everything that runs in the context of Dagger is quicker. Over WiFi, it would be over like my tethered connection, be very, very slow. So how we use Dagger in this case, let's go here. We have a couple of commands. The one I think that's the most interesting one, for example, for running tests. So let me show you what that looks like. I'm just doing just test. You can see there it does like the dash dash cloud. Let's see how fast it goes. It starts an engine in the cloud, it connects to it. So we need to set up Varnish. We need to connect Varnish to all the back ends, the TLS exterminator, and you want to do it in a way that is as close to production as possible. So it's almost like we want to run the system as it runs in production, but we want to do it locally. Okay, in this case, it's a remote engine, but from the perspective of how it gets put together, it's literally just the same context of running containers. And because everything runs in containers, we get the exact reproducibility that we get in Fly. So, we get the same configuration, the same Linux subsystem, the same kernel, or like a very similar kernel, and what that means, I mean, in this case, it's even using Firecracker behind The scenes, so it's very close to Fly. So, we are able to test the system most accurately, and we're even able to run the system most accurately, as it would do in production. And that part is hard because whatever you do locally, if I was running, for example, on the Mac, everything would be different. Even if I would have a VM, things would be different. So it's like that containers, the container image, the interaction between all the components, and when we ship something in production, it's as close as possible to that image. Even down like to the Go version. So right now, for example, what's happening here, we're pulling down, I mean, some of it is cached. We're pulling down various dependencies and you can see they're all Linux ones. So Linux, Linux, and again, a VM on a Mac, it gets you close, but it's ([Time 0:48:16](https://share.snipd.com/snip/6e07113f-f1b0-4b58-9b94-ed7f30b4b794)) - **Full-Request Telemetry Reveals Bot And Regional Hotspots** - The team pipes every request into Honeycomb to slice traffic by endpoint and data center. - That visibility let them spot unusual bot patterns and regional hotspots quickly. Transcript: Gerhard Lazu Thank you. Yeah, thank you. Thank you very much. So this is, we're looking at Honeycomb, and one of the integrations that we have in Pipedream is we are sending every single request to Honeycomb and some requests to S3. So, like, we see what's happening with these instances. So what that means is we are able to, for example, if I come back to the boards, if I come back to the boards again, let's hope the tethering works. It will be a little bit slow. Okay, so let's go PipeDream requests, and I think maybe PipeDream content, but let's go requests. We can see the get, okay? And now we can slice them and dice them. The 404s and the 500s. So maybe let's do gets, okay? So let's go to the method. And the only thing that we need to do, so this is one hour ago. So these are all the requests in the last hour. And we can say, give me the URL, which is what we've been asking. So group our URL. What this is going to show us is the get but also the number of requests so what you see here is that the most popular request is podcast feed which got 203 requests in the last hour and this Is global so we can do one more we can say let remove that, and let me remove request, and let me do data center. So we can see which data centers hit the most. So let's go that. And we can see that Frankfurt, 71 requests went to podcast feed. Chicago, the next one, 47. San Jose, California. So these are the most requested URLs. ([Time 0:56:31](https://share.snipd.com/snip/fff6113c-2ecb-4542-a81c-6d16238e2690)) - **We Switched Production By Changing DNS Live** - The team flipped production traffic by deleting DNS A records live on stage. - They relied on short TTLs so traffic shifted incrementally to Pipely during the event. Transcript: Jerod Santo We'll put it right there. Gerhard Lazu So, what we're going to do now, we are going to shift all the traffic to PipeDream. Right? So, all the production traffic is going to go to PipeDream. Shall we do it? Get your podcast to do it. That's why we're here. All right. This is the moment. All that we have to do, this is how simple it is. It's always DNS. No scripting. It is. No automation. It is DNS. It all comes down to that. We are going to delete these one by one. Jerod Santo These are the A records which are pointing to. Pause. Let me get my own recording of you doing this. Go for it. Gerhard Lazu Okay. Kaizen 20 in Denver. Let's do that. Jerod Santo This is going to be anticlimactic because you're going to delete those and then we'll be like, they're deleted. Gerhard Lazu Now, if this one goes down, so let's see if the system can handle all the load, right? Like, what can happen? So after this, the only requirement is we walk away, okay? We do this thing and we go for lunch or something like that. That's what's next is lunch, I think. All right. So, boom. We have four more to go. Jerod Santo Okay. Gerhard Lazu Three more to go. Sorry. Yeah, after this one, three more to go. It's not deleting. It's not deleting. It's slow, right? It's my tether. Shout out to DNSimple, big fans. Thank you very much. One more, one more. Cool. And the DNS is like one by one. So more and more requests are going to go to this one, which is the pipe dream. The pipe dream. The pipe dream, there you go. Jerod Santo All right. So delete. So we're at one out of three. Now half our traffic. Gerhard Lazu Yeah, And now 100%. We have one more. So we're 50-50. Jerod Santo What's this one? That was it. Gerhard Lazu That's CDN. That's CDN. We have to do the same thing for CDN as well. You've got to delete the CDN ones too. Jerod Santo There you go. All right. So this is app requests now. Is this goodbye to Fastly? Gerhard Lazu This is it. This is the farewell to Fastly? Adam Stacoviak This is it. This is a triumphant moment and a morium at the same time. Only if it works. Gerhard Lazu Only if it works, yeah. I may need to add these back. Jerod Santo We still love you, Fastly. Gerhard Lazu If this crashes and burns, we'll have to go back for sure. Jerod Santo The good news is they have no idea what we're doing. Yeah, exactly. Gerhard Lazu And this is not a live live show, so we're not streaming this. That's right. So it's okay. That's right. It's in this room. I'm with friends. If it doesn't work out, we edit this part out. So that's okay. Adam Stacoviak He always says that we never do. Okay? Cool. We ship it all. Gerhard Lazu There you go. One more to go. I think this is one and one more to go. Jerod Santo Now, how long is it? We're at 60 second TTL. 60 second TTL. Gerhard Lazu This should be fast, yes. All right. Jerod Santo If you can get your phone out and hit our website, please. Gerhard Lazu Let us know how it goes. But that's it. We only have these two. Let's generate some traffic. Where's the real time dashboard showing the activity? Real time dashboard, of course. Let's do that. Let's do last 15 minutes. Let's see everything crash and burn. Oh my gosh. No. What's the worst that could happen? Okay. Jerod Santo The worst that could happen is all our fly-ins run out of memory. Gerhard Lazu Well, there's no peaks and valleys. Let's go back to here. Let's go to the memory. There we go. Every one minute, let's maybe go to the last one hour. So, let's see. ([Time 1:01:42](https://share.snipd.com/snip/55995380-35df-4627-bd20-73e001a5054a)) - **Homepage Cache Hit Rate Jumped To 99.1%** - After the rollout the homepage cache improved dramatically to about a 99.1% hit rate. - That single metric jump shows how much user latency and backend load caching can save. Transcript: Gerhard Lazu Look at that. Are we going back to that? So let's go back to that. 17% cash hits. Last 24 hours. This is the homepage, right? That's the one that we were focusing at. Sure. So we're looking at the homepage. Last one day, we had 3,700 hits. Yes. And we had 33 misses. I think that's better. That's better. It is better. Jerod Santo It is better. We Kaizened it. It's better. We did it. Gerhard Lazu No way. No way. Do the math on that. Let's see. 3, 7, 1, 5 plus 33. That's 99.1% hit rate. Adam Stacoviak That's a lot of nines. Gerhard Lazu Yeah. Two. Jerod Santo To be exact. That's two nines. That's way more nines than we did. Gerhard Lazu More that we had. It was like 70%. So this is much better. Obviously this will take a while for all the traffic to start shifting through. It is DNS. ([Time 1:06:20](https://share.snipd.com/snip/4df3c7da-abdb-4cba-bd79-d3845a0a2977))