Kaizen! The post Pipely launch mop-up show with @gerhardlazu 🔗 **Source:** [Kaizen! The post Pipely launch mop-up show with @gerhardlazu](https://www.youtube.com/watch?v=hYahjUGoy5Y) > [!note]+ Document Notes > tags: [kaizen, pipely, podcast-production] > author: The Changelog Podcast > source: YouTube > reference: https://www.youtube.com/watch?v=hYahjUGoy5Y > date created: 2024-11-27 15:23 > > # 🏷️ Title: Kaizen! The post Pipely launch mop-up show with Gerhard Lazu > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Follow up on audience feedback for 2026 live event location and format — survey needed #community - [H2] Double memory allocation on Pipely to improve MP3/news cache ratios — config change #performance - [H3] Plan pairing session to investigate cache eviction behavior — technical deep-dive #devrel ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Pipely GitHub Repository](https://github.com/thechangelog/pipely) - [Varnish Cash Rename Announcement](https://varnishcache.org) ## 📝 Summary - The Changelog team debriefs their successful Pipely CDN launch at a live Denver event - Despite initial underprovisioning causing CPU spikes, blue-green deployment strategy enabled quick rollback - New CDN dramatically improved performance: cache hit ratio from 70% to 89.5%, homepage response times 863x faster - Team commits to planning another live event in 2026, considering Austin as alternate to Denver - Gerhard's new role at Loophole Labs involves live migration technology requiring 100 Gbps networking - Technical discussion of cache optimization opportunities for MP3s and news content ## 📖 Main Content The episode covers the post-launch analysis of Pipely, The Changelog's custom CDN solution built to replace Fastly. The launch occurred during a live recording in Denver, but within hours they discovered severe performance issues due to underprovisioned instances running at 100% CPU utilization. The team's blue-green deployment strategy proved crucial - they maintained parallel infrastructure allowing rapid DNS-based rollback when the new CDN couldn't handle full traffic load. This architectural decision prevented extended downtime and provided learning opportunities for proper scaling. Performance improvements from the migration were substantial across all metrics. Cache hit ratios improved from 70.5% to 89.5%, with feeds achieving 99.5% cache hits. Response times showed dramatic improvements, with homepage requests serving 863 times faster for 50% of users due to better caching (from 18.8% to 98.5% cache hit ratio). The discussion also covers planning for future live events, with the team considering timing, location, and format options. Austin emerges as a potential alternative to Denver, with optimal timing suggested for March-May or October-November to avoid extreme weather. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=0) — [H1] Kaizen Introduction and Context - Key idea: Post-Denver launch debrief after 18 months of building Pipely - Evidence: "After 18 months of building Pipley in the open with friends, we shipped it on stage in Denver" #### ⏱️ [02:15](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=135) — [H2] Immediate Post-Launch Crisis - Key idea: CPU utilization spiked to 100% within hours of full deployment - Evidence: "Some instances were just getting overwhelmed and users hitting those pops, they started experiencing a slow and unresponsive changelock.com" #### ⏱️ [04:30](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=270) — [H3] Blue-Green Deployment Saves the Day - Key idea: Parallel infrastructure enabled rapid rollback via DNS changes - Evidence: "Because we blue greened, there was nothing that could not be undone rather quickly" #### ⏱️ [07:45](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=465) — [H2] Hiking During the Incident - Key idea: Team went hiking unaware of performance issues due to blue-green safety net - Evidence: "Instead of dealing with a changelaw.com incident, we went hiking" #### ⏱️ [12:20](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=740) — [H1] Release Candidate Evolution - Key idea: Multiple RCs addressed scaling and functionality issues post-launch - Evidence: "RC5 hand of orange JSON response failing on startup and bump the instance size to performance" #### ⏱️ [15:30](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=930) — [H3] Upload Functionality Restored - Key idea: MP3 uploads were broken initially but fixed in subsequent releases - Evidence: "Uploads were failing. MP3 uploads were failing and that was pull request 39" #### ⏱️ [18:00](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=1080) — [H1] Performance Metrics Comparison - Key idea: Cache hit ratio improved from 70.5% to 89.5% overall - Evidence: "We went from 70% to 90%. That was a big jump" #### ⏱️ [22:45](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=1365) — [H2] Feed Performance Excellence - Key idea: RSS feeds achieved 99.5% cache hit ratio, critical for podcast syndication - Evidence: "Feeds. If you look at all the feeds, we are at 99.5% cash hit ratio" #### ⏱️ [25:10](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=1510) — [H2] Homepage Transformation - Key idea: Homepage cache hit ratio jumped from 18.8% to 98.5% - Evidence: "The homepage before the hit ratio was 18.8%... Today it's 98.5%" #### ⏱️ [28:30](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=1710) — [H1] Response Time Improvements - Key idea: Homepage responses 863 times faster for 50% of users - Evidence: "For 50% of the users, the homepage is 860 times quicker" #### ⏱️ [32:00](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=1920) — [H2] Future Event Planning Discussion - Key idea: Team commits to planning 2026 live event with date/location by end of year - Evidence: "By the next Kaizen we will have a date and a city" #### ⏱️ [35:15](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=2115) — [H3] Austin as Alternative Venue - Key idea: Austin suggested as alternative to Denver with March-May or October-November timing - Evidence: "The best month they say to come here is March through May or October through November" #### ⏱️ [38:45](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=2325) — [H2] Varnish to Vinyl Transition - Key idea: Varnish Cache renaming to Vinyl Cache due to legal disputes - Evidence: "Varnish end 8.0 the opensource varnish 8.0 will be the last one that will be called varnish cache" #### ⏱️ [41:20](https://www.youtube.com/watch?v=hYahjUGoy5Y?t=2480) — [H1] Gerhard's New Role at Loophole Labs - Key idea: Working on live migration technology requiring 100 Gbps networking - Evidence: "Live migration... memory, the disk, the connections. How do you live migrate connections from one host to another?" ## 📚 References - [Pipely Open Source Project](https://github.com/thechangelog/pipely) - [Varnish Cache Rename Blog Post](https://varnishcache.org/docs/trunk/whats-new/changes-trunk.html) - [Fly.io Platform](https://fly.io) - [Fastly CDN Service](https://fastly.com) ## 🗣️ Paraphrase - The team successfully launched their custom CDN called Pipely after 18 months of development, but immediately faced scaling issues that required quick rollback and remediation - Performance improvements were dramatic once properly scaled, with cache hit ratios improving from 70% to 90% and response times improving by orders of magnitude - The experience reinforced the importance of blue-green deployments and proper capacity planning for production systems ## 📰 Newsletter **Pipely CDN Launch: Lessons in Blue-Green Deployments and Performance Optimization** The Changelog team recently shared their post-mortem from launching Pipely, their custom CDN solution, in a live Denver event. While the initial deployment faced scaling challenges with CPU utilization hitting 100%, their blue-green deployment strategy enabled rapid recovery and ultimately delivered impressive performance gains. **Key Takeaways:** - Cache hit ratios improved from 70% to 89.5% overall - Homepage response times became 863x faster for most users - RSS feeds achieved 99.5% cache hit ratio, critical for podcast distribution - Blue-green deployments saved the day when initial scaling proved insufficient The team is now planning their next live event for 2026, considering Austin as an alternative to Denver. They're also exploring cache optimization opportunities for MP3s and news content. **Resources:** [Pipely GitHub](https://github.com/thechangelog/pipely), [The Changelog](https://changelog.com) ## ✅ Action Items #### DevRel - Community engagement: Collect feedback on 2026 live event preferences via surveys and comments - Technical content: Document blue-green deployment best practices from Pipely experience - Education: Create case study on CDN performance optimization and cache hit ratio improvements #### OSPO - Software inventory: Update SBOM for Pipely 1.0 release and vinyl cache transition - Community engagement: Support Varnish/Vinyl cache transition communication #### Community - Event planning: Establish planning committee for 2026 live event with deadline for date/location by end of year - Audience research: Survey community preferences for event format, location, and timing #### WordPress - Performance: Apply Pipely cache optimization learnings to WordPress CDN strategies - Documentation: Create performance optimization guides based on cache hit ratio improvements ## 🧪 Hypotheses - Increasing memory allocation will improve MP3 and news content cache hit ratios beyond current 87.5% and 83% respectively - Large object eviction is causing memory usage fluctuations and degrading cache performance over time - Austin location for live events will attract different audience demographics while maintaining engagement levels - Blue-green deployment strategies can be standardized and applied to other infrastructure changes ### 📊 Methodology - Memory allocation experiment: Double current memory limits and monitor cache performance over 2-week period - Object size analysis: Profile cached objects to identify large items causing eviction patterns - Audience survey: Deploy geographic and preference survey to current community members ### ❓ Questions - What specific object sizes and TTL patterns are causing memory pressure in the cache layer? - How can we optimize cache warming strategies to maintain consistent performance? - What event format would best serve both local and remote community members? ### 🔮 Future Research - Live migration technology applications for podcast infrastructure and content delivery - Community event scaling strategies for growing technical audiences - Cache optimization patterns for mixed content types (audio, text, images) ### 💡 Implications - Practice: Blue-green deployments are essential for high-stakes infrastructure changes - Performance: Cache hit ratios have outsized impact on user experience and infrastructure costs - Community: Live events create strong engagement but require careful planning and audience research - Education: Technical post-mortems provide valuable learning opportunities for broader community ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade Level 12 #### 👀 Visual learners - Present: Performance graphs, network diagrams, before/after comparisons - Improve: Architecture diagrams showing blue-green deployment flow, cache layer visualizations #### 🔊 Aural learners - Present: Full podcast episode with detailed discussion and explanations - Improve: Audio highlights reel focusing on key technical insights #### 🗣️ Verbal learners - Present: Clear definitions of CDN, cache hit ratios, blue-green deployments - Improve: Glossary section with technical terms and analogies #### 🧩 Kinesthetic learners - Present: Step-by-step deployment process, hands-on troubleshooting experience - Improve: Interactive cache optimization checklist, deployment decision tree #### 🧠 Logical learners - Present: Performance metrics, before/after comparisons, quantified improvements - Improve: Decision framework for scaling infrastructure, ROI calculations for CDN migration #### 🧑‍🤝‍🧑 Social learners - Present: Team discussion format, collaborative problem-solving approach - Improve: Community feedback mechanisms, peer review processes for deployment decisions #### 🧘‍♀️ Solitary learners - Present: Detailed technical analysis, individual metric exploration - Improve: Self-assessment questions for deployment readiness, personal reflection prompts #### 🌿 Naturalistic learners - Present: Systems thinking approach to infrastructure, ecosystem impacts of performance changes - Improve: Environmental considerations of CDN efficiency, sustainability metrics ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive technical terms - Improve: Alt text for performance graphs, detailed descriptions of visual metrics #### Keyboard navigation - Present: Standard video controls and timestamps - Improve: Keyboard shortcuts for key topic navigation #### Contrast and readability - Present: High contrast video presentation - Improve: Transcript formatting with clear speaker identification #### Text resizing and reflow - Present: Responsive transcript layout - Improve: Technical diagram scalability at high zoom levels #### Media and captions - Present: Full transcript available - Improve: Timestamped captions with technical term pronunciation guides ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global CDN performance considerations, international audience impact - Improve: Discussion of geographic access patterns, regional performance variations #### Inclusive language - Present: Technical jargon explained in accessible terms - Improve: Plain language summaries of complex infrastructure concepts #### Neurodiversity support - Present: Structured discussion format, clear topic transitions - Improve: Visual topic markers, consistent terminology usage #### Representation - Present: Technical leadership perspectives from different backgrounds - Improve: Diverse case studies, varied infrastructure use cases ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Performance metrics clearly demonstrate cache optimization benefits - Inductive strength: Blue-green deployment success supports broader infrastructure strategy - Causal reasoning: Clear connection between cache hit ratios and response time improvements - Analogical strength: Rattlesnake hiking analogy effectively illustrates production risk management #### Fallacies - Survivorship bias: Focus on successful aspects may downplay persistent challenges - Post hoc reasoning: Some performance improvements may have multiple contributing factors #### Tone of voice - Overall tone: Collaborative and reflective, with appropriate technical depth and humor ### 🛡️ DARVO Analysis - Deny: No defensive patterns observed - Attack: No attacks on external parties or critics - Reverse victim/offender: Not applicable to technical discussion - Emotional cues: Positive, constructive tone focused on learning and improvement ## 🎯 <!-- deep-read:sections end --> ## 📺 Video  --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=hYahjUGoy5Y)* **[00:00]** All right, Kaizen Gard, we are here to Kaizen, the first Kaizen after our onstage Kaizen. >> Yeah. >> How's it going? How's your life? Your life has changed since then. >> It has. Yes. Yes. Uh a new job that started in early September. Actually, September brought in that change. It was a good change. Uh so that's as you know, new jobs are always exciting. There's always like so many things to do and you know, you have to meet everyone on board uh do things properly. So that was that was really the whirlwind. I really enjoy **[00:32]** that very much. Um well obviously before that even we had like our family holiday which was amazing. I always you know love going outdoors and spending the proper outdoors you know like not the um well the mountains. [laughter] >> That's what I baser outdoors for me it's mountains and lakes and you know so that just goes together. >> Yeah. Which is why Denver was really really nice. I really enjoyed that. you know, it was close to what I enjoy. Um, but otherwise, I mean, how how is it **[01:03]** October already? That that's what I don't know, >> right? >> It was July and then now it's October. How did that happen? >> Well, there's this thing called time [laughter] >> every 24 hours. >> It just keeps going day. >> You I'm with you on that. I don't know how it's October. I feel like uh it was just January and I feel like it was a very fast year. >> Yeah. this year for some reason. It it was going slow and then it got really really fast. >> It's kind of how life goes, right? It starts off slow and you're like 10 years **[01:34]** old wishing you were an adult, you know, and you're like, when am I going to be able to drive? When am I going to be able to >> drink legally? I don't know what the 10-year-olds want to do anymore, [laughter] but back when we were kids. And then you get there and you're like, "Slow down, life." And then now you're all a sudden you're in your 40s and you're like, "Holy cow, >> I am Daisy, man." >> A vapor. Totally. Tell us about the new job. What are you what are you doing now? You spent years at Dagger. >> Uh yeah, it was almost four years. It was time for a change. The new companies called Loophole Labs. They're focusing **[02:05]** on infrastructure primitives. Some really interesting things that revolve around live migration. That is uh definitely the central piece. >> Mh. >> And when I say live migration, I mean the memory, the disk, the connections. How do you live migrate connections from one host to another? I find that really interesting. >> What's most interesting is like the size. So if you have 64 GB to migrate, how do you how do you do that >> in milliseconds? I mean, what does your connection even need to look like? >> Migrating between physical hosts. **[02:36]** >> Between physical hosts. >> Really? >> Yep. 64 gigs. >> Mhm. >> You might need like a 100 gigabit home lab to do that. >> Exactly. That's [laughter] coming up. Yes, >> that is coming up indeed. So that just sent me down that rabbit hole like, "Okay, so >> I need a new home lab basically because 10 GB is just not enough. So what would 100 GB look like?" >> Yeah. Yeah. >> I know what it looks like. >> I've been watching your channel, dude. [laughter] >> All right. What did you think? >> I'm loving it. I I love seeing the You got lots of engagement. You know, that **[03:08]** video blew up. The 100 Gbit Home Lab. Uh I love the production quality. I know all the I know you sweat the details. It's fun to watch and uh the narrative is cool and yeah, I'm just happy for you. It's going well. >> Yeah. No, I'm just I'm just so pleased and it's part of the day sky in as well. >> Oh, it is. >> Everything connects. Everything connects continuous connection. Adam, have you seen any of Ghard's recent videos? >> Not his recents, but I've seen them before. Not this 100 gigabit. So, I'm **[03:39]** I'm a little jealous. >> You'll like this one cuz it's kind of up your wheelhouse. Although you know his his home lab is better than yours. I mean that's the sad part. I mean happy for him but sad for you. >> So I think this is going to be um almost like a challenge like how can we improve the home labs too Linux Ubuntu Arch a couple of things are going to come up networks because they're really interesting GPUs like how do you how do you run all this stuff in a way that does not break the bank because that's the other consideration. I'm not a data **[04:10]** center. I wish I was, but I'm not. [laughter] I'm very sensitive to noise, so I can't have like fans blaring and like, you know, one UU, twoU servers, the really shrill ones. So, um, I'm just No, the Noctua person that goes for whisper quiet everything, even fanless if it's possible. I tried Fandless for 100 Gbit and it runs really hot. >> That's the one thing which I was not expecting just how hot these things run. 400 Gbit is even more crazy and 800. **[04:40]** Wow. So that's really like the next frontier. That that's what I'm looking at. 400 800 and beyond. And all this to service some workloads that have very sensitive latencies in terms of throughput latency. I mean how do you run a remote GPU? I mean that's crazy. What I mean what I mean by that the GPU is in a rack. You're on your laptop >> and you're running against the GPU. So you have an Nvidia GPU on your laptop. How does that even work? So like the software on your laptop is believes there's a GPU available to it but it's **[05:11]** over the network. >> The kernel kernel extension. >> Mhm. And it presented as a local device but it's actually remote device. So it intercepts all the calls and it has it has to have very low latency and very fast network to carry all this stuff around and it makes it appear as if it was local. >> That's GPUs in MacBook Pros. I never thought I would see the day. >> That's crazy. [laughter] >> Yeah, I know. So that just that's like a um a preview into the new job and what's coming. It's only been like a month. I can't believe it's already been a month, **[05:41]** but it all connects. >> Really all connects. And that's really the important part. >> So all roads lead to change log. Isn't that what Adam keeps saying? That's right. All road leads to change log. >> Yes, sir. >> Many ways to get to change log. >> Indeed. >> Do you have a uh visual aid for us, this Kaizen? Do you have a a deck? You bring your deck? >> Oh yes. always, you know, always. >> So, we're going to start screen sharing. October 16th, I'm going to try an intro. See if you like it. >> Okay. **[06:11]** >> I'm trying could I'm going to try and do the job easier [laughter] for like the intro. >> Okay. >> It's October 16th, 2025, and you're listening and watching Kaizen 21 where Adam, Jared, and Ghart do some mopping. Not moping. Mopping. Double P. Mopping. mopping, not mopping. >> So, yeah. So, stick with us to the end so that you can rate our mopping performance. >> That's the plan. [laughter] That's the **[06:42]** plan. >> I'm so good at mopping. >> I'm going to find out how good I am. >> Let's do that. Well, the audience gets to rate us. So, at the end of this, >> this is all get leading to some performance rating. Some mopping performance rating. >> I'm getting nervous already. Here we go. [laughter] Launching Pipley together was one of my 2025 highlights. I mean, seriously, it was just so amazing. >> After 18 months of building Pipley in the open with friends, we shipped it on **[07:12]** stage in Denver and it was so awesome. Seriously, such a great feeling. The audience was clapping. Jared and Adam were smiling. I was so proud of what we have achieved. really really proud. Really, really proud. But do you remember what happened a few hours right after we did the stage bit? >> Um, hiking. >> Yes. >> Um, >> in between that, before that or **[07:43]** something else? >> Lunch. Yes. Yes. That's right. >> Adam, help me out here. >> The selfie. >> Adam, what do you remember? >> A selfie. >> Yeah. Well, this is not a selfie [laughter] technically. >> What are you trying to get? Take us where you This is a That's a gangster selfie right there. >> This is like a gangster selfie. I really like it. I think it's one of my favorite pictures. >> Not a selfie, but that's a great picture right there. >> Yeah. A Kaizen for next time. An improvement suggestion for next time. >> Emo night. Mo Night Brooklyn. >> I think we have to just >> Oh, that's the other For those not **[08:14]** watching underneath the marquee, which we made sure that they put the Change Dog podcast on their marquee. And underneath there, apparently that night is going to be emo night Brooklyn, which is a weird thing to have in Denver, but >> yeah. Uh, and it reads as if the change look podcast, that's what's happening. >> It's our emo night. There should be like >> Yeah, [laughter] >> there should be like a line between the two. >> All right. So, we wrapped it up at the venue. We went to lunch and after we went to a hike. >> That's right. >> But right after lunch, as I was getting **[08:45]** ready for the hike, I thought to myself, let me check the fly.io biometrics. Oh my god, now I know where you're going with this. [laughter] >> All right, so for those who are listening, we are looking at a point in time snapshot of the fly.io app graphfana dashboard for pipe dream. So for the CDN that we just launched, we're just looking at that screenshot. So [sighs] I think I was mentioning Piper and Pipley. Do you still remember Jerry the **[09:16]** difference between the two? >> I do. So, Pipley is the software. It's the open source project that allows us to run pipedream, which is our instance of Pipley's running on flies network that actually serves as our CDN. >> Correct. >> That's right. >> Yeah. >> Yes, that is correct. Yes. And Pipedeream. >> Pipedream is I I think I just told you the pipe dream was Sorry. Yes, you did. [laughter] >> There's another trick question. >> Let me pump that up for you here. >> You just want me to say it cleaner. >> Yeah. Pip is our pipely instances **[09:48]** running as a distributed network around the world on flies network. >> That's it. >> Yes. >> Okay. >> Pipely the generic one. Pipe dream the specific one just for us >> cuz it was our pipe dream. >> It was it was Yes. Exactly for everybody. It's not just for us. >> Exactly. That's Yes. That's where we're going with this. That's right. >> So we're looking at this Graphfana dashboard and clockwise we see pops by traffic in North America. That is the top left. network IO in the top right >> and CPU and memory utilization at the **[10:18]** bottom. What stands out to you? >> CPU utilization stands out to me cuz at about 14 >> 37 >> it went nuts. >> That's it. Yeah. Yeah. So that's exactly what happened. So >> 100% nuts. >> Yeah. Yeah. Yeah. [laughter] >> It was crazy. >> There's the title. 100% nuts. >> 100% nuts. Like CPU just freaked out. So after we updated DNS on stage, once DNS propagated and more traffic started reaching our shiny new CDN, some **[10:49]** instances were just getting overwhelmed and users hitting those uh pops, they started experiencing a slow and unresponsive changelock.com. We're going to pretend that there's a bad boys [laughter] bad boys uh meme here. Okay, >> we bad boys it. [laughter] >> So >> Oh my god. Basically, >> what's the bad boys meme? I don't even know if I know this. >> What's You know, like uh when when like stuff blows up and they just walk away without >> walking in front of Exactly. stuff and stuff blows up behind them. So, that's **[11:21]** exactly what what I did. I have to take responsibility for this. >> You exploded everything. >> Pretty much they were on fire. >> You went to launch. Yeah. >> So, honestly, I underprovisioned. >> Okay. >> So, our CDN was running on these tiny instances and just didn't get very far. So, we're looking at an impossibly tiny bike that someone is actually riding. That's what we're seeing right now. >> I can't even believe they could ride that bike. By the way, we're we're watching a video. I don't Is this on Okay, it doesn't matter. There's a There's a video with a bike and a dude **[11:51]** on a very very small bike. And >> yeah, >> it's impossible. >> If you're watching this, if you're watching this, maybe this will make it part of the B-roll. Who knows? We'll see. We'll see what the editor will decide. But, uh, I underprovisioned. I mean, that's really what happened. >> Okay. And I know that you've heard me say this many times over the years in the kaizen. I'll say it again. Always blue green. Always. And in this case, it uh means that it in this case it meant **[12:21]** that all previous infrastructure remained in place. We just deleted some DNS entries on stage in Denver. And because we took that approach, really all I had to do was to add the DNS records back and confirm that the traffic was coming back to life. Uh everything was healthy again. So it just took minutes. It took maybe 10 15 minutes for everything to propagate up to 30, but that was just like the edges and everything was back to normal. And because we blue greened, there was nothing that could not be undone rather **[12:54]** quickly. explain the concept of blue green for those of us who aren't sres. >> So blue green in in this in this context means whenever you create whenever you introduce a change try to introduce it alongside what you already have what you're already running because if you run the new system in parallel it means that it's very easy to go back to the previous system. In this case one is blue and one is green. So rather than doing an in place replaceing in place whatever you're running doing **[13:26]** an in place upgrade or taking things down and you know putting the new things up. You don't want to do that. You want to run two of the same setup and then either gradually which is what we did. We were gradually migrating traffic. That's why we knew that 20% was good, 30% was good. So again we're not noobs. We've done this before. But in this case, I just could not estimate how much traffic it will hit the new instances and especially bot traffic. >> Mhm. >> I mean, there was like a lot of bot **[13:57]** traffic. I don't know whether it was LLM scraping. I don't know whether I don't know what exactly it was, but there was a lot of traffic hitting specific instances and they were just like falling over. They didn't have enough memory. They didn't have enough CPU. So in this case because all the previous infrastructure was in place updating the DNS record and in this case adding some DNS records was enough for the previous IPs to start propagating and then everything continued working as before. So the actual mechanism that you used in order to do a blue green with a CDN was **[14:31]** running both CDNs concurrently and using DNS to just point different directions. And so your roll back was literally just to go delete the new DNS entries or a few of them. We had options. >> So what we So when when we were on stage, we had uh we were serving I think five IPs. >> Yeah. >> Um one IP was a new CDN and the four IPs was was the previous CDN, which meant that 20% of the requests based on how DNS would resolve would hit the new CDN, **[15:02]** >> right? And then everything else would go to the existing CDN. Now I think when we were stage I think we were maybe one in three. So we're about 33%. It was more than one because that's how we started like 20%. >> We took it to 100%. >> We took it exactly. So on to 100%. We went to lunch. >> I checked the numbers like crap this thing is on fire. Right. >> So like it it's literally like blowing up. So all I had to do is add back the previous IPs so that when DNS >> fast they would serve some of our **[15:33]** requests and pipely would serve others. >> Exactly. Exactly. So we went back to about 33%. >> So they couldn't handle 100%. With the the underprovisioning that you had done with our fly VMs. >> That's correct. Yes. They were too small. They didn't have enough memory, not enough CPU, and there were too few of them. There were certain hotspots that needed more than one instance. And that's what we did. >> Okay. Mop it up. Ghart, mop it up. >> Exactly. How's mopping in that case? Well, uh, instead of dealing with a changelaw.com incident, we went hiking. **[16:06]** >> That's right. >> Like nobody knew that this thing had happened, right? >> And this is video, so it's actually plague. So, we were relaxed. Uh, we enjoyed great conversations. Uh, Jared was looking for great drone footage to shoot for news. >> That's right. We were in the amazing red rocks area which is just where Jared is pointing very soon. He's coming up to pointing it. He was like, "See you looking around. Where can I fly this thing?" >> Right. >> And look at that there. Let's go there. That's where the red rocks area. Did you **[16:37]** use any of the footage that you shot, Jared? >> Um, I did create one episode that featured B-roll from Denver. >> I'm not sure how much or if any of that was actual drone footage. Uh, I had to fly the drone all the way across the valley to get it over to Red Rocks, which was really kind of boring. Even if you speed it up, it's kind of just like there's a highway and I'm not I can't remember. It's been a few week a few months. I know we made a Denver version of Change Log News with the B-roll from Denver, but I combined a bunch of stuff **[17:08]** and I probably slipped a little bit in, but I also am not the best. I'm getting better at driving the drone. In fact, I have a show coming up next week. By the time this publishes, it'll be this week's change log news where I'm following around the tractors as they uh as they harvest. I did the soybean harvest and now I got the corn harvest and I'm I'm getting good like I'm flying it like 10 ft above the tractor and you know keeping it center frame and stuff. So >> I wasn't very good at flying it then because it was pretty new. But uh a **[17:39]** little bit probably snuck in but not enough to to be exciting. I mean it it was great to to know that um to basically see how news is being put together where you just like literally take this drone out, fly it around and then you decide whether it was something worth using or not. So >> basically >> seeing that production. So I think we had like two in one in that point >> and um that was really cool. On this hike, I would have missed this rattlesnake **[18:10]** >> was it not for Matt Matt Johnson that almost walked into it. >> You don't want to walk into a rattlesnake, by the way. >> No, he don't want to do. [laughter] No. No. So, yeah, but uh it was my first time seeing a rattlesnake up close and um it made it a happy memory because Matt was quick to react, but things could have turned out very very differently. Now, you have to imagine this is summer. We were in shorts. We were engrossed in our nerdy talk, right? And nearly walked into this rattlesnake. **[18:41]** So, it was very it was very close calls. It's the exact same thing as shipping an underprovision system into production. >> [laughter] >> Almost the same. Almost the same. >> Same shorts even. Probably. >> Yeah. Yeah. Yeah. >> Yeah. >> So, [laughter] yeah. Exactly. Exactly. Same shorts. >> Yeah. Short short ones again. >> Oh, yeah. That was that was a close one. So, uh this is the analogy to what we avoided both in real life, right, but also on stage. >> So, now the nice thing about a rattlesnake is, you know, God gave him those rattles and those rattles are very **[19:13]** useful because a rattlesnake doesn't want to be messed with. And so when you get close to a rattlesnake and don't know it, they'll they'll let you know that you're close. And that rattle that that shake is saying, "Get away from me." >> However, I don't think we got any notifications, you know, to draw the analogy. Our fly machines didn't say anything, did they? Like I didn't know there's no rattle on that side of the equation, was there? >> Well, I did receive some emails that instances were running out of memory and crashing, but it was taking happening after a while. So that was maybe the equivalent of that. Right. >> But in this case, because we were so **[19:45]** engrossed in the conversations, we never heard the rattles. >> There you go. Okay. Fair. >> There were drones buzzing around. [laughter] >> You know, it was a bit crazy. So, yeah, we did not pay attention to that. >> All right. >> Uh, so if you're watching this next piece, uh, and if you don't like meat, look away. >> Oh, [laughter] boy. >> Oh, I know what this is going to be. Just just just listen. I'll begin to play it for you. **[20:15]** >> Okay. >> Adam, would you le would you like to describe what is happening? >> Well, we asked very politely, can you show us the kitchen and how things are done? And they said yes. And then they took about 20 minutes to prepare a tour and they allow us to come in there and hold exactly how the chefs prepare and prep the meat to come out to the very awesome Fog Chow. uh patrons, which is kind of cool. And so there's Jared >> uh holding it like a beast about to eat **[20:46]** it, but not because it's somebody else's. >> Yeah, that's not my food. >> That's not your food. But you just have some versions of that. >> It's not the snake. >> No, it's not the snake. >> It's not the snake. [laughter] >> That would have been a propo, wouldn't it? >> Yeah, this is [clears throat] not the snake. >> Yeah. Oh man, that was really good. Really, really good. So, yeah, >> that was a good thing. That was like >> Yeah. The funny thing about that one too though is that uh you never get what you don't ask for. That's my if it's one advice I give anybody in the whole entire world, it's like you don't get what you don't ask for. So ask and you **[21:18]** might get it. And what Adam's referring to is the tour of the kitchen at Fog Chow. >> Exactly. >> Cuz I didn't I wasn't going to ask. I was ready to pay a check and go home. And I was like, we had very kind uh weight staff and they were discussing with us talking about, did you know the actual chefs are the ones that deliver the food? I was like, I had no idea about that. So, they're giving us an insider look and Adam just said, "Can we see the kitchen? >> Could you give us a tour?" >> And then he's like, "Yes, I can." And >> they're all kind of aruck because normally we're used to hearing no. But **[21:49]** yeah, don't uh if you don't ask, you don't receive. So, thank you, Adam, for getting us this sweet shot of the change log t-shirt on me while I >> eat that rattlesnake. Or no, that's a pork chop, I think. I don't know what that is. >> That's a joke. >> Looks like maybe I'm going to guess. Who can guess this? Uh, I'm thinking ribeye. Is that ribeye? >> Sirloin. >> Looks like maybe sirloin. That's >> Yeah, it's a little too >> That's what I would have picked if I if they gave me the choice would be the sirloin because I was to die for. >> Potentially picana. Is that pana? **[22:21]** [sighs] I don't know. You're the connoisseur. >> Looks like pecan. Yeah, I can see the fat cap there on the left hand side. >> I remember that. >> So good. >> And with fruit. I mean, that's how I like my meat with fruit. And it was just amazing. It was like one of the amazing meals I had in the US ever. Like seriously, it's just like above above them all. Just so so good. >> The thing which I wanted to convey is that even though we had a short trip, it was only two days. It was a very rich trip. Many things happened. It wasn't just about tech. It was literally **[22:52]** friends coming together and spending a bit of time and it just felt so good, so natural. Um, it flew by. I mean, we we were joking about how fast time flies by. But that those two days were like even I remember like when the when the uh when the border agent was asking me, "So, how long are you staying for?" And I said, "Two days." He was, "Really? Two days? Where are you coming from?" I said, "The UK?" Really? Can I see? Like he just would not believe me that I'm flying in for two days. >> Mhm. >> It was so worth it. I'm happy to hear that because that's a long trip for you. >> It was really good. It was really, **[23:22]** really good. So, this was uh the end of my Denver trip and uh it is the favorite thing that we did together ever. So, uh [snorts] it was really good and this is obviously leading to something. Uh what are your thoughts on doing this again next year, 2026? >> 100%. >> And I I I'll give you I'll give you a couple of moments to think about that. Uh moment as they are. >> So there is a reason for 100%. >> Jared Jared Jared, you answer. **[23:53]** >> Oh, I'm for it. I'm already I'm for it. I mean, Adam was like, "Let's do it four times a year or something." I was. So, we're talking about We're talking about how many times, not >> I'd say two to three. Two to three. Four is too much. Two's for sure. Three might be pushing it, but I'd love to do more live shows like this. And twice a year is enough in my opinion. three, four, maybe >> two for sure. >> Honestly, I'd be very happy if we if we do this again, like at least like a repeat. That would be very nice. Um it **[24:24]** doesn't have to be the same place. But at least once a year would be a good habit, I think, to start having um because so many things happen around and it's not just the show, it's everything else. And um I don't know how many people want to join us. Uh, now >> here's an opportunity to comment. Here's an opportunity to comment. Yeah, >> you do have to be close to Denver or willing to travel and we had many people travel which is kind of cool. >> Now you know that I like to do things well in advance. Um, for example, most **[24:55]** of my next year's holidays already booked. The only one which I couldn't book is actually end of October because flights only become available like a like like a year before. So I need to wait a few more weeks. >> You're such a planner. Holy cow. >> Wow. >> Oh yeah, that's why I'm asking. So >> that's why he's asking, Adam. He wants to get something on the calendar, >> right? So, I think I think uh it would be a good time like in the next >> What's your what's your opinion on Denver Ghart? Is like that our place now or is it more like that was fun. Let's go somewhere else and let a different subset of our audience and friends have an easier time getting there. Right. We **[25:26]** know Denver is not accessible for everybody. Uh Austin obviously would be another good choice because it's close to Adam and it's close to lots of people that we know. I'm not sure if you got directs to Austin Gearhard, but >> I do. Yeah, >> you do. So, thoughts on a new location versus right back to the mountains? >> I'm I'm I'm all up for it. I've been to Austin. It's a really nice place. Um I I had a good time there. Uh the Colorado River, which is a different Colorado River, was was very nice to uh kayak on. That was like a good experience. For example, um and yeah, I mean there's so **[26:00]** many things that we could do. The more interesting question would be who would like to join to see where is like where do we have like the most loyal fans so that you know we we're there for them as well and they can join us and we can you know do something together. Um and do we want to do any interviews? What does that look like? It will there be a conference before or after? So how do we structure it so that it makes sense on all accounts? What about holidays? Because all of us have holidays and other conferences. though. **[26:31]** >> Mhm. >> What would make sense? >> Well, I think thank you for getting the conversation started. We're definitely in to do it again and I think it's time now in October to start discussing the details of what that might look like. I don't think we have to wait until next summer. Uh but I haven't seen your calendar, Gard. As you know, I'm going to be crossing the pond in May with my family, taking my daughter and wife over to France and Italy um and to celebrate her **[27:03]** graduation. So, that's going to be a big trip for my family. But, um I don't know what yours looks like. I don't know what Adams look like. So, let's get talking and uh get some stuff figured out to our listeners. You know, comment, let us know where should we go, when should we do it, and what should it look like? What would you like to be a part of? Should it be just like this last one? You know, an interview episode, a Kaizen episode, and then, you know, festivities around that. Should it be more? Should it be less? Please let us know in the comments. >> I think if you ask Adam, he'll say change lock con. **[27:36]** The first change lock conference ever. Go, go, go. >> Maybe. I kind of liked it just as it was, though, honestly. I I really I wouldn't mind having like uh some trusted not like demos, but like some some showand tell, you know? I think there's a lot of like pontification from the stage. I'd love to have some showand tell type stuff if that was a thing. And maybe that's demos. I'm thinking like Oxide with their racks and stuff like that. You know, that's kind of showand tell. But uh you I don't know. I don't **[28:06]** know. I I really just enjoyed Exactly as it was. Honestly, I think a lot of it was really good. I think it was just short enough and just sweet enough that it was doable and repeatable and it was just an interview and just Kaizen on stage and just hanging out with friends and it was more community than it was like brands or you know what's new in tech kind of thing. I think it was just just the right kind of feels honestly. So, I'm not sure. I **[28:37]** haven't thought a lot about the the design of it enough to to want to change it much. >> Okay. Well, we got the conversation started. We got some thoughts. We can put it on the back burner and come back to it. Uh maybe between now and the next Kaisen um see if we've reached any conclusions or any any suggestions which are firming up. But um this was good. I think we should set a goal and say by the next Kaizen we will have a date and a city and that way we can at least say the date, start booking flights or whatever **[29:07]** we have to do and then the details of what we're going to do while we're there. We can figure those out from there. But at least, you know, in the next quarter we should know for sure by the end of the year what we're doing where and when. >> I like that. Okay. I like that very much actually. Very. While we're on that subject, I did have have uh my friend here that could tell me a couple details about Austin at least. >> Which is your friend? >> Claude. You know, the latest the latest model. >> They're actually suggesting Sonet 45 right now over Opus 41. They're calling **[29:38]** it Legacy. They're calling Opus 41 Legacy. That's kind of funny. I was just like brand new a month ago and now it's Legacy all of a sudden. Anyways, it's hot here in Austin. So, the best month they say to come here is March through May or October through November. And we're obviously in October now, so we can't do it next month. So I think if it was Austin, I would agree with this that it's before summer, not during summer. So March through May, somewhere in there if it's Austin. >> Yeah, I think that's a good time. **[30:09]** >> All right, let's plow forward. >> So we're looking at all the different steps that we had to take between being on stage at Denver. Do you know which RC we were there just looking at this list? It's it's a list on on the pipel we're looking at the readme all the various release candidates of 1.0 before going to 1.0. I thought it would happen on stage it didn't or soon after it didn't but it did happen now. So we are beyond and we are running on 1.0. If you look **[30:40]** at 1.0 RC4 limit varnish memory to 66%. And that's the one commit which I pushed that was on stage. There was next one uh RC5 hand of orange JSON response failing on startup and bump the instance size to performance. That was the scale up that needed to happen. So really we didn't need much in terms of resources. One CPU, a performance CPU and 8 GB of RAM. That was enough. And then we could send 50% **[31:11]** of the traffic and we we we were on that 50% for quite some time. So RC7 that was like the LA RC6 uh more locations uh back in timeout. Now the one thing which was failing and this this was discovered after I think we were rooting more and more traffic is that uploads were failing. MP3 uploads were failing and that was pull request 39. Do they work now Adam? >> I think so. Yeah. I mean I've been uploading. The >> answer is yes. **[31:42]** >> Yeah. >> Nice. Great. >> I can't say I can't say no. I can't say I guess I can say yes. Yes it does. >> Yeah. Very nice. They work. Yes, >> they work. Yes. >> Great. So, that was like the one thing which >> I had Fastly Hardcoded for a while. >> Yeah. >> Mhm. >> Because it wasn't working and we needed to keep uploading MP3s and then when you when you asked us to test it, I removed that from my Etsy hosts. >> That's right. >> And I have had no issues and didn't report back because I was saving it for for Kaizen to tell you. Yes, you fixed it. Thank you. >> Nice. Great. That's what I wanted to **[32:12]** hear. So, >> you probably hoping I report back, but I didn't say anything at all. I did test it though. Well, as long as you don't have the uh hard-coded IPs, that's was trying to uh check those out. >> Now, it's still a good time to remove them from everywhere and uh because everyone else is going to pipe. So, everyone's using the new CDN and uploads are also working. So, there should be no more issues and 100% of the traffic is being served with 1.0. So, 1.0 was tagged on yesterday. This is only **[32:43]** yesterday. >> Okay. However, however, the traffic has been served through the new instance on I think it was on the 5th. Yes, on the 5th of October, everything switched across all the traffic. We're looking now at the screenshot from Honeycomb, which is showing the requests going to Fastly. And we can see that after October 5th, they dropped and there's a few. There will always be a few um hard-coded IPs, **[33:14]** whatever the case may be. Uh it's not human traffic that for sure. It's just that that sounds wrong. It's not people hitting the website. >> It's most likely. [laughter] >> Yeah, exactly. It's it's it's bots. Um so we have been running 100% on the new system for more than 10 days now. And I wanted to be like why did it take so long? Because it was like that was July. Yeah. >> I wanted to be certain that everything worked fine. Like after the last time I went extra extra extra cautious. **[33:44]** >> I needed all the metrics. There were the summer holidays. Um I joined the new startup. I was a bit busy and I had to build the most insane home lab ever which took me a while but that's going to come come a bit later. So this is what it looked like. Now, do we remember now that everything is said and done, why did we need to build our own CDN? What's the reason behind it? >> Frustration, frustration. Okay, that's a good one. >> Plus the hacker spirit, plus >> our cash hit ratio was out of out of our **[34:16]** own hands. We wanted it in our own hands. >> Yeah. Yeah. It was like the previous screenshot. So it this is the moment I turned off all traffic from like forever in this case from Fastly. It was only a few days but you can see that in those few days we had 155,000 cash hits sorry cash misses 155,000 cash misses and we had 370,000 cash hits. So the ratio does not look **[34:47]** right. that green line, the cash hits. There were days when there were more or like periods, not days, there were periods up to maybe half an hour, an hour when there were more misses than hits. And you do not expect a CDN to behave that way. And by the way, this is across both changelock.com and CDNchange.com. So it includes both the static assets, everything. It's just just a small window, but it just shows the problem. Now, as a percentage, that translates to 70.5%. **[35:20]** So, 70.5% cash hits, and that is really not great. Okay, I know you've been expecting this. So, let's see what do you think is our current cash hit versus miss ratio. This is across all requests. So, now that we switched across, we had 10 days to measure this on the new system. What do you think is the cash hit versus miss ratio? Now you're giving us four choices. This is a multiple choice question. >> Only one. >> 85% is A, B is 89%, C is 95% and D is **[35:54]** 99%. Adam, what are you thinking? >> I'm locking in C. 95%. Okay, >> I'm going for the gusto. I'm going 99%. >> Whoa. [laughter] Straight indeed. >> I'd love to be wrong, but if I'm right, I'm going to be a I'm so >> 89.5%. >> So close. Both of us were wrong. >> That's the answer. Yeah. >> How good would that have been, though? >> Yeah. Well, >> well, you know, some stuff is fresh. It just is. >> Yeah. I mean, we we we can improve on **[36:24]** this. We can I mean, it's it's not >> Now, the important thing is we went from 70% to 90%. >> Mhm. >> Okay. That was a big jump. >> Now, in our hands now, we can actually affect it, which is the last one. Before it was just like we could only >> complain. Now, we can actually do stuff. And even that after a few years of complaining, you just become tired of complaining. >> Yeah. We grew weary. >> We just we just build our own. Okay. So, >> can we do better? **[36:55]** >> Can we do better than 89.5%. But so I think everyone is thinking this, but really what I think we should be thinking is do we need to do better? Do we need to do better than 89%. Okay. So, let's have a >> question. >> Feeds. If you look at all the feeds, we are at 99.5% cash hit ratio. Before they were at 96.8%. So what are feeds for our listeners and watchers that know don't know who wants **[37:25]** to answer that? What are they? Yes, what are they? These are XML files that represent the current state of our podcast syndication, our produ our our episodes that we're shipping and have shipped. And so they're hit often by robots who are scraping feeds in order to update their podcast indexes and let people know which episodes are available. [sighs] And they should be at 99.5% because they only change when we publish a new episode, which is at this point in **[37:56]** our lives, three times a week, on a Monday, on a Wednesday, and on a Friday. And every other request, every other day and time is the same exact content. >> That's it. So I would say that this is possibly the most important or or yeah I would say the most important thing to serve because if we don't serve fees correctly how do you know what content change log has? How do you know when content updates and this is like worldwide so I think I think this is pretty good and improving on 99.5% I **[38:28]** don't think we should do it. >> No >> the homepage before the hit ratio was 18.8%. Gosh, I understand that. [laughter] >> I like this was my my my biggest issue for as long as I can remember. >> Today it's 98.5%. >> And to our listener who's probably out there thinking you guys were certainly doing it wrong. We spent years trying to do it differently. like go back and listen to all the kaizens of us trying to change the way that we actually **[38:58]** configure and respond and our headers and our I mean we tried and we ended up with 18.8%. >> That's it page and so here we are. >> That was so bad. So bad. Now MP3s I would say the second most important thing was 86%. Now it's 87.5%. Maybe this can be better. Maybe this can be improved. um we just need basically more memory or store them on disk do a bunch of things because by the way caching is in memory **[39:28]** >> and I still need to understand why memory once it gets filled it doesn't remain filled so I've seen this weird behavior where after like a few hours memory starts dropping but why because there's no pressure on memory why are object objects getting evicted >> my assumption is that we're storing some large objects and if there are smaller objects objects that need to be stored in memory. The larger objects get get evicted which means that the cash drops but I would need to understand that a **[39:59]** little bit better. Still MP3s aren't worse than they were before. >> News. I know this is something that's very important to Jared. It was it was 52.6% cached before. Now it's 83%. >> So improvements across the board. Now, we could improve it and I think we I especially news and MP3s, I would like to to look into that, but I think news is like top of my list, but is there anything else that you think that we **[40:30]** should pay attention to or in terms of the cashit ratio, any other resources? >> No, I mean MP3s are static assets. You could go look at other static assets, images, etc. But I just don't think we want to squeeze this radish too hard. I agree that news and probably taking a lowhanging fruit pass on the MP3 end points and seeing what we could do there. We'll probably be bear would bear some good fruit, but even those I wouldn't like spend hours and hours **[41:01]** trying to make them much better. >> What I'm thinking, I would just basically double up the memory and see how the changes things, which is just a config setting. It would take me maybe a minute. >> That that would be my first action item. >> Okay. We'll try. >> Yeah. I imagine news could be very similar to feeds because once news is published it's similar to the feed. It's not really it's changing once a week and once it's published it likely never changes. It never has changed right **[41:32]** >> now that we've moved all all of our comments and everything exist elsewhere. I mean they're in Zulip. They're on YouTube. So there's no comment feed there. There's no reason for anything to really be >> new >> except on publish. Yeah. Okay. So, [clears throat] I'd say news of feed pushing that to the boundary because it doesn't change much. >> I'd love to explore that uh when you do the MP3 exploration of large objects getting pushed out. I'd love to just sit on your shoulder, I suppose, or as a fly on the wall kind of thing. Just explore that with you because I'm super curious **[42:03]** about what makes that cash get purged out of the memory myself. >> Yeah. Well, pairing up is something that I'm getting better and better every day. recorded and published pairing sessions. So Jared has the experience, not Adam. So >> uh yeah, that sounds great. Now it gets better. >> Oh my god. >> Okay. [laughter] It gets better. >> What does all of it? >> Uh do you recognize this seed? >> Um okay. So that **[42:35]** >> this is Johnny Monic, right? >> No. >> Say again. >> Johnny Nonmonic. Is this uh N? >> No. That's uh is that Hugh Jackman? >> Yes. >> This is not uh >> Yeah, this is not Wolverine. >> Swordfish. >> This is Swordfish. [laughter] >> Oh, no. I'm not sure. >> I'm very nervous right now, Gard. [laughter] >> That's okay. It's fine. It's It's recorded. >> Swordfish broke the show previously. >> Okay. >> I'm ready for it. [laughter] Sorry. >> But that doesn't mean that I'm ready for **[43:05]** it. You're going to play this live for me right now on camera. Okay. >> I will. Yes. So, this clip is going to blow you away. Okay. >> I'm going to watch it. [snorts] Okay. >> I'm hoping this is Sora. >> Okay. Halle B. John Travolta Hugh. Jackman has a gun to his head. He He's typing. He has to hack something in a certain amount of time. Was it 30 seconds? >> 45 seconds. 45 seconds. >> 45. >> Look at his fingers. >> Types very, very furiously. **[43:36]** >> He's typing furiously. Oh, access. He gets very disappointed when that [laughter] happens. >> He gets very disappointed. >> Yes. >> What Axis gets, he died. All right. >> So, um, that was that was a moment of fun. [laughter] >> That was for what? >> It was it was it was it's things get better. Okay. Things get better. >> That's a moment of disappointment. >> Exactly. Even if you're under pressure [laughter] and you have to deliver, >> things will be better. **[44:07]** >> So, pipe dream gets better. Okay, >> now we looked at the cache it ratio. What I would like to look at next is the response time in seconds. >> Okay, >> these are all the feed requests before and after. So the P50 for all the feed uh requests used to be two milliseconds. And you would think, wow, that's pretty good. Well, the new system is like half a millisecond. So it's it's a four time improvement. The P75 **[44:39]** is 13 times better. So for 75 of users, 75% of the users, the feed responses get served 13 times as quickly as they were before. The P19599 like it gets progressively better, which means that the requests are served much quicker, at least four times as quick as they were before. And you might be thinking, hey, that's **[45:09]** bots. What about humans? So, what about the homepage? For 50% of the users, the homepage is 860 times quicker. >> That's nearly three orders of magnitude quicker. That's a crazy amount quicker. >> Mhm. >> Now, obviously, it is the fact that it was not cached. like only 18% of the requests were cached, >> right? >> But the page is instant. Like now it's **[45:40]** instant. It's 0.0000 like three zeros 3 seconds. That's like a third of a millisecond. >> That's nice. >> So >> you're welcome humans. >> You're welcome humans. Now what does that look like? I think 863 times is really difficult to imagine. So I'm going to I'm going to play something for you to see what it means. So, what we have here is 1 second at the top. >> Mhm. >> That's how long it takes. No, hang on. I'm not playing it. I should be playing **[46:11]** it. There you go. Now I'm playing it. Okay. While 833 seconds at the bottom is still loading. And it will continue loading for so long that we're not going to wait 15 minutes for this thing to load. Okay? >> We're not going to wait that. So that's the difference between how fast the homepage loads now versus how it used to load before. This is for the majority of the users. So the cach it ratio the connection there was that everything was slow and there's nothing **[46:44]** we could do about it. And I think slow is relative right because when you think when you're talking about milliseconds um I think there's like about 50 or maybe 100 milliseconds when things are nearly instant. >> Mhm. But in our case, the homepage was taking 150 millconds to get served and the more like the tail latencies were really crazy. Like the tail latency was over a second for the homepage to serve. That was a long time. By the way, this thing is still going and it's not even like 10% there. >> What's the What's the rationale behind **[47:14]** this video? Ex explain to me how this is supposed to >> explain things. So the top one shows you how quickly it takes for 1 second to >> finish, right? So the response, the 1 second response, it just visualizes it how quickly that gets served. >> The bottom one shows you how long the previous CD and how long in comparison 83 863 seconds is. So we had we have now like things are **[47:46]** loading in a second or the equivalent of a second and before things were taking 863 seconds to load. They were >> the same request. They were >> longer >> relative not absolutely >> relative. >> Okay. >> Exactly. >> So the 1 second it represents >> like loading speed which is like a millisecond but we can't visualize that because it's too fast. >> Exactly. and 863 seconds. That's how much slower it used to be. So, it's a relative >> relative to 1 second versus relative to **[48:18]** milliseconds. >> Right. I understand. >> That way we can actually visualize it. And now it's at 2 point. Oh, he reset it. Okay. I was going to say backwards. >> It's it's like loading again because it finished, >> but yeah. >> So, it's like 15 minutes versus 1 second and then reduce that down to milliseconds and we're basically that much faster. >> Exactly. 15. Yeah. waiting 15 minutes versus waiting a second. That's exactly the speed difference between what we used to have and what we have now, >> right? >> It's just really fast. And that was [laughter] that was really really fast. **[48:49]** >> That was a CDN thing. That was not that was traversing DNS into CDN getting a cache hit or miss serving, you know, rehydrating getting new from cache. That's where all the time was spent. So in in this case, these were the responses from the varnish perspective. So if I go back here and we're looking at this table. >> Mhm. >> So [clears throat] uh this is the homepage, the response time. How long does it take to serve the homepage from Varnish's perspective? **[49:19]** Whether it's in the cache, whether it needs to go to the application and request it and then eventually finish serving the actual response. So before P50 259 milliseconds that's how long it used to take. >> Yeah. >> And now it takes a third of a millisecond. >> What changed specifically with varnish then? >> Well caching most of the like if we go back to the table if you go back here now 98.5% of the requests are served **[49:51]** from cache. We don't need to go back. I mean the the homepage is almost always in cache. We we very rarely have to go back to the application to fulfill a request. Before only 18% or 19% of the requests could be served from cache. >> Yeah. >> So varnish had to go to the application. The application had to serve the response so that varnish could serve the response back to the user to the end user. >> You keep saying varnish. Don't you mean vinyl? >> Yes, I do. [laughter] **[50:21]** I do. >> Not that you but there's a >> vinyl. Yeah, vinyl is coming up in >> curve curve ball in January. So, it's not it's not here yet. >> For those that know, they're renaming the varnish software. >> They are >> because of legal disputes to vinyl cache. So, Varnish Cache, the open source project will be renamed to Vinyl Cache. >> That's correct. >> Whereas Varnish Software the company will continue as Varnish Software the company. >> I'm going to jump ahead. I mean, this is like uh yeah, Jared is coming from the from the future now. I am. So, um, PHK, **[50:52]** uh, Paul Henningamp, he posted on September 15th. This is on varnishcash.org. Uh, it was just about a month ago. He wrote the 20 years old, >> uh, 20 years old and it it it is time to get serious, sir. That is the title of the blog post. And he talks about open source. Yeah, the open source varnish cache rename. >> Some legal disputes indeed. You can go and read it up, but basically varnish end 8.0 0 the opensource varnish 8.0 **[51:22]** will be the the last one that will be called varnish cache. So from March next year and this I think this is very interesting. It will be vinyl name of varnish. The open source varnish will get renamed to vinyl. >> Okay. Now I know who we have to have as a guest on our interview in Austin. PHK >> launching the vinyl cash >> live on stage. He I think he lives over on your side of the ocean >> or Berlin. Yeah, because he is >> he probably wouldn't come to Austin, but [gasps] >> we can try. **[51:54]** >> So that's Yeah. >> All right. So [sighs] speaking of speed, this one's for Adam. >> Uh you know that I joined Loophole Labs. That's it. So what we do, it requires a really fast low latency network. And by that I mean at least 100 Gbits, okay? It needs to have sub 1 millcond latency. So about two months ago, I started building a new 100 GB home lab. Um, and I know that Adam has been asking me for a really long time to to do a video on my **[52:25]** home lab. So Jared already watched it a few weeks ago. This is live. And the easiest way is to go to yt.makeitwork.tv. That will take you to YouTube and there you can go and watch it. And I take you through the entire journey why I had to build it. A couple of interesting things. Yeah, it's How How did you How I mean, what did you think about it, Jared? The portions that you managed to watch? >> Yeah, like I said earlier, I thought it was really good. I It's cool. You get >> What did it miss? What did it miss? How do How would I improve it? Maybe that's **[52:56]** a better question. What would have made the video better? >> What would have made the video better? I don't know. in terms of like content that was missing. I feel like the the way that you do your summarized intro is compelling, but sometimes it like jump cuts so fast or it it it's sometimes hard for me to track exactly. So, it's like a a hard thing to give feedback on. I'd have to give you specifics for you to be actually know what you're But you do. even though I'm **[53:27]** like compelled to continue watching and I do enjoy it, there are times where I'm confused. >> Um, and so I think as you continue to refine that because I know it's a it's a style that you're doing and you've gotten better at it since cuz I've seen some of your videos from a year agoish and definitely getting there where I'm like I look forward to it, but I also sometimes am not sure what's going on. I'm not sure if that's on purpose. Maybe it is to keep me intrigued as there are all kinds of techniques to keep people watching on social media. But um I think you can continue to refine the narrative **[53:58]** that kicks off from the beginning in in making it more cohesive or just you know laying more breadcrumbs perhaps for the person who's not initiated because you're so deep into what happened and you know the whole story from front to back and you're telling it and that's great but I I think that's one thing that could be improved. Um, nice and further about stuff that's missing, but because I wasn't there for what miss what you know what got dropped. >> That's great. That That's great. So, yeah. Okay, that's excellent feedback. **[54:29]** All right. Well, Adam, when you get a chance to watch it, if you get a chance to watch it, I would love uh uh to get some feedback from you, too, into what in terms of what would make it better for you. >> For sure. >> So, as you watch it, anything that could be improved because the next one is coming up. So, how fast is the pipe dream? That's what we're going to answer now. How fast is the pipe dream on this 100 GB home lab? Because you remember we talked before uh we're looking at um how much like when **[55:02]** we benchmark these things, fly.io itself is limiting us to how much bandwidth we can push, right? We wouldn't want to be pushing tens of gigabytes or hundreds of gigabytes. That would be crazy because that's cost someone money. So running benchmarks like that is is is is is not great. But also um the WAN there's like a limit there. But in this case the limit is 100 Gbit. We know that. So this is the new home lab. This is what it looks like. It's running Auntu 2404. Um I think the most interesting thing is **[55:33]** the CPUs about it. So it has the Thread Ripper 9970X. And the reason for the Thread Ripper is because I need a lot of PCIe lanes. So I couldn't use like a regular consumer grade CPU. Mhm. >> Uh I needed uh 16 lanes for the mother for the uh GPU, the RTX 4080 which is there at the bottom. And I need another 16 lanes for the network card. Even though it's a PCI 3.0, if you give it for example eight lanes of in this case PCI 5, it can only use eight lanes of **[56:04]** PCI 3. So you would like basically half its it its speed. And eight lanes of PCI 3.0 And zero means 64 Gbits. >> So really I need a full 16 lanes. Doesn't matter. I mean needs to be at least PCI3 to achieve its full speed. And the card is like the the thing that you see there's like some green LED lights. It's right below the fan for the CPU. All right. So that's one. This is the server. We're going to be running Pipe on this host. And this is the other **[56:36]** part of the home lab. This is my older machine. It's about 3 years now. It's a Ryzen 75800X. It has 16 threads, not 16 core, 16 threads. Um, a small GPU. GeForce GT730 is a fanless one. It's the one at the bottom. And again, the reason why I had to do this is because I needed the 16 lanes which are right under again that CPU cooler for the network card. If I was for example to put an NVME in a specific slot or if I was going to fill **[57:07]** any other PCIe slots, the first slot because it shares bandwidth, the first PCI slot would be limited to 8x which again would create that 16 uh sorry that 64 Gbits limit. So I needed to give this full 16 lanes and that's where this limitation came from. Now this is the star of the show. This is what that card looked like. uh those cards look like um you can see I mean there's like a DAC cable uh obviously you need two now they have two modules which is interesting because the card itself in terms of the **[57:39]** PCIe bandwidth will max out at 128 Gbits so it can't go beyond that from a PCIe slot perspective even though each module could do 100 Gbits >> really in combination you can only push 128 theoretical maximum and these are all cards. It's a It's an Nvidia Melanox Connect X5. So, they've been around for many years now. Uh I think 2020 at this point. So, they're like about 5-year-old cards. >> What's the price range on these? Are they expensive? **[58:09]** >> Uh they they can be. Yes. These specific ones, I got them off eBay and I paid for both of them £500, which I think it's about $800, I think. 7 $800 roughly. Um now in the US, I think you can get them for even cheaper because you get more hardware, right? data center hardware that just gets, you know, sold for um a good price. So, refurbished stuff or in this case, it's just used. It's not refurbished. And a DAC cable. The cable can be a bit expensive. You can pay **[58:40]** anywhere between like close to $100. I think this one was about 60. And then you need two. Um and the reason why you need two is here because I'm configuring the two modules in a bond in a network bond. It's an LACP bond. And in this case, what it means is that I'm basically having two cables and I'm creating a single virtual connection that uses both um both modules. So again, the theoretical maximum is 128 GBs, but in reality it's more like 112. **[59:12]** I was not able to push it beyond that. So, HTTP stat. What I'm showing here is that if I do an HTTP stat from the client, H22, by the way, H22 is the client. That's the year home lab 2022 and W25 uh the workstation 25, but it's also home lab, but it's also workstation. So, it's it's a it's a combined thing because of the the many cores that it has and it has a multi multi-roll that host. All right. So from the client from H22, I'm basically going **[59:43]** to that private IP uh 1025 10141 and on port 9,000 the pipe dream is running and you can see here the the the response is going to the app. It's just basically a pipe dream running in a container locally as it would run on fly. >> Mhm. >> All right. So let's see what this baby can do. I'm using Oh. I'm running 64 clients and I'm sending a million requests to the homepage. **[60:14]** Let's see what it will do. >> That was it. That was 1 million requests sent to the homepage. >> Oh my gosh. >> So, it took less than 5 seconds to complete. We pushed 225,000 requests per second. In terms of data, uh, we transferred 12.5 gigabytes in 5 seconds. >> Wow. >> Uh, we reached 2.8 gigabytes per second. **[60:44]** And that's over just over 22 Gbits per second. So, if you were to guess, what would you say the bottleneck is in this case? You just had to guess. It's not the network. We know it's not the network. >> It's not the network. I mean, it has to be >> maybe container. >> The CPU. >> Yeah, it's actually Yeah. So the container I mean it it doesn't have it has doesn't have any overhead. It uses the it uses it binds to the local network. So there's like no uh no matting no bridging um nothing happening **[61:16]** from a networking perspective it binds to a port on the local network. So when we go to 9,000 it's the port local 9,000 on the host. Now I'm wondering what happens if you fetch the master feed because the master feed is really big. It's 13 megabytes in size. So, it's about a 100 times the size of the homepage. So, we're going to do something very similar. The difference is we're only going to run a 100,000 requests, not a million requests. And this is what it looks **[61:47]** like. Okay. It's taking a little bit longer. Mhm. What do we see? What do we see here? So here we see um the CPU and the network for the client H22. And what stands out is that many of the cores are 90 80 close to 90% usage. We can see the network throughput is 4.59 GB per second **[62:19]** which in this case is 36 Gbits. So we're getting close to 40 Gbits per second. But in this case, we can definitely see that it is the CPU that seems to be the bottleneck, but the CPU on the client. So, this is where we're running the benchmark from. So, I'm wondering what does the CPU look like on the host on the big one, >> the thread ripper. >> The thread ripper. >> It looks pretty quiet from this. >> Pretty quiet. Pretty chill. **[62:50]** >> That's it. >> Yeah. I think the peak that there's one core which is 11% but otherwise everything is like less than even 5%. >> So most of the cores are chilling. So this confirms that we have the bandwidth. I mean we went from 20 to about 40 GB gigabits per second from 20 Gbits to 40 Gbits per second. And we can see the CPU is fine. So Pyream could serve more from the instance perspective. uh but the client where oh runs the one that benchmarks seems to be **[63:21]** approaching a limit even so we are able to send a 100,000 requests and I think it takes I know about 40 seconds something like that roughly we can see again the host uh the client which is running really really hot 36 constant so that's nice and constant and we'll see it now at the end finish 100,000 requests it transferred about 40 something gigabytes this is where like this in terms traffic. It would cost a few dollars just this benchmark. Uh we transferred 200 GB of data for this one **[63:55]** and um the peak the request per second was 2,200. That's pretty good. That's pretty good. >> Mhm. >> So I'm wondering what would happen. There's like one more thing which I would like to do. What would happen if we move the client from the host with a slower CPUs to the host with a faster CPUs? Right? So we do like a swap around. We run Pipeam on the slower host, but we run the client that benchmarks things on the faster host. So what would that look like? All right. So same 100,000 **[64:27]** requests, but now we've reversed where we run these things. And this is what it looks like. >> It's looking faster. Mhm. This is the client. This is where varnish runs. Sorry. This is uh where the pipe dream runs. The CPUs are 100% basically. And this is where the benchmark runs. We see some um Yeah, this is really So, let me just go a little bit back. There you go. >> 80 gabbits per second. Now, >> 80 gabits per second. Yes. So, we're **[64:58]** able to nearly achieve like the the performance like the saturate the network. >> Mhm. >> And we can see the CPUs. We still have plenty of CPU room on uh the new workstation, the thread ripper which has 64 threads. So plenty of CPUs there >> which is now running oh it's not running pipe >> correct not pipe dream but >> we haven't maxed the client out. Now we're actually maxing the server out. >> That's it. So in this case if this host had faster CPUs it could go faster but **[65:30]** now we're just b basically bottleneck on the CPU. >> Well you're going to have to upgrade your host to care hard. I think I will. >> We need answers. [laughter] >> We need answers. >> I think I think we will. But uh this just goes to show that the setup scales really really nicely and this is what we've been all working towards [laughter] fly and change log io. So sorry changeelock.com. So it's a good combo. Um in this case honestly fly is **[66:01]** sometimes like uh throttling us on on the bandwidth. That makes sense. I don't think they were expecting a CDN to run on fly. Honestly, I mean, some of the peaks we can be pushing up to 10 Gbits I've seen like in in in the metrics so far. There's these are like only like the peaks. I don't know what the limit is on fly, but I think talking to them about this would be a good idea. I've been mentioning this, but I think that following up on this because I mean, what happens if there's I mean, maybe throttling is what we want, but it will **[66:31]** affect other users. So, are they okay with us running a CDN? What can we expect from this setup? >> So, yeah. So, vinyl, we already know, we've been here, the vinyl is the rename. Varnish will become the open source varnish. Cache will become vinyl from March next year. So, by the time we meet next, Pipley and Pipe Dream will be vinyl. All right, we're closely approaching the wrap-up. **[67:02]** What's next? Let's talk about what's next. The first thing on my list is I want my my my my BAM. What does BAM mean? >> Oh, your big ass monitor. >> That's it. The big ass monitor. [laughter] So now when I switch my big ass monitor behind me, what I see is all of Pipe Dream. All of it. I see all the traffic going through change log. It's beautiful. Look at it. I see all the areas which are like the odds. It's just **[67:32]** so nice. Seriously, this is the best painting I could hang in my study and I have it on all the time. So I I see it. It refreshes periodically. This is the fly.io dashboard. On the left I have the edge. How does the edge behave? This is the fly proxy for our CDN. And on the right, depending on how you're looking at it, on the right, uh I see uh the fly app, which is in this case a pipe dream itself in terms of memory usage, CPU usage, all those things. And uh it's a **[68:04]** thing of beauty. Now I understand when there's a problem, I see the the memory when it drops like all the things are are just there and it works well. So the BAM is done. [laughter] That was a quick one and thank you Fly for a great dashboard. That was really good. Um, but the one thing that keeps coming up is out of memory crashes. They happen rarely, but they still happened. And even though we have to limit set up, I need to understand what exactly triggers those crashes, how to prevent them from happening. So sometimes um and I say **[68:36]** sometimes, I think the last time it happened maybe a week ago, two weeks ago, it's it's hasn't been this week. So when instance when it gets overloaded, it runs out of memory, it crashes. All it means is that the requests they get rooted to a different instance and when the instance restarts it starts like with an empty cache. It takes a while for the me for the for the cache to fill in memory. So I'd like to dig into that. Remember the logs are events are metrics Jared that we've been talking about >> where currently they're being stored in S3. **[69:07]** >> Yeah. >> And then you have the job a chron job which processes them. would really like to sort out that pipeline because part of pipe dream part of rolling this out I just realized how everything is put together basically and I think we can improve on that >> so what I'm >> yeah what I'm thinking is if we were able to ship all the logs um in a in a in the in a column-like format so like a wide format in something like click house and I'm thinking click specifically this would just make **[69:38]** everything so easier like the whole metrics analytics pipeline that we have. >> Um not to mention that in a way that would be two of everything and Honeycomb could be one of them. But Clickhouse if it stores all these requests it could be our other um event store, >> right? >> And we can visualize all those events. So that would be good. Now do we know anyone at Clickhouse at Clickhouse Cloud specifically? Do we have any Click House cloud friends? **[70:09]** >> Danny, we talked to Danny a couple years ago. Remember that, Jared? At uh Open Source Summit in Vancouver? >> I don't remember that, but that's why there's two of us. >> Uh I'm pretty sure we know some folks if not through acquisition directly. So, we'll we'll hunt our contacts and come back to you. So click house would be really interesting or anything that can store lots of events because every single request in our case would be stored there. We would batch them and do all of that. Uh but we would read qu we **[70:41]** would write quite a few events. That means basically every single request would end up in that uh data store and then would need to be able to aggregate them and read them back really really quickly which I think would do away with any of the analytics eventually but we currently doing Postgres with the background jobs all of that >> can just >> come from that >> go. Exactly. >> Yeah. kind of challenge there is that uh what stack on click house in particular because that's tying us to yet another **[71:13]** behemoth that might uh make us hold it wrong potentially and we'll be forced to build something else click house spelled h a us instead of h o us [laughter] version of it what exactly do you like about that flow >> to uh what makes that be the first class citizen for you for data transport 40. >> So I've used it for a couple of years. Um we've been using it a dagger and continue doing so. Um and it works it **[71:44]** works really well at at a large scale. So it processes not billions like trillions of events. Uh it scales really nicely and it's really fast. and the Click House cloud team specifically, they've been very supportive and um they seem to be innovating and doing things in a very thorough way. So, it's something that has always been dependable. >> Mhm. >> Now, we currently use honeycomb for like the whole UI thing, but we also store a **[72:15]** subset of the requests in S3 and then we process them and I think we store them in Postgress. So there's we we duplicate these things in a couple of places. If we had a single place where we store them and this could be like click that would be like the primary store. We could read any metrics whenever we need. We can create materialized views that like it's just so flexible in terms of how we can slice it and dice it. It would be our alternative to honeycomb and we still love them and I definitely see us continuing using them but it **[72:46]** wouldn't be the only one. Um and we would have the same view in terms for alerting or monitoring or anything like that. We would need to do something separate. So it just centralizes every single request coming from every single instance in this case. >> Mhm. >> Yeah. And obviously we still store them in S3. Click house can read from S3 which is really nice. It has um support like a special parket format which means that you can have like like data stored like in long long-term storage in an S3 **[73:18]** like um system and then if whatever was to happen with click house it's down or there's a problem with it. It it hardly ever happens. I mean I think I've seen it only happen once in like almost four years. So it's been very reliable in that way. Uh but it means that we can we can revamp [clears throat] how we do analytics. Uh I'm not sure how you think about that Jared because I mean you've been mostly using that like the analytics that you get in the app. **[73:50]** Um how does it I mean do you need to upgrade it? Do you need to work with it? Like is it always like do you just forget about it? Mostly you set it up and you forgot about it. >> Mostly forgot about it. I think that the biggest drawback is how infrequently it updates at this point and so we don't I mean maybe that's a feature because we don't like check our analytics um >> on the daily on the second obsessively like maybe we could >> that being said if we can get that information faster and learn something **[74:21]** along the way I think it's worth it and I'm certain that this this would change the way that we do things enough that we could get that information much faster >> [sighs] >> via a cron job. So, I'm interested in it. Would it bring huge value to us to be able to see the downloads faster? Probably not because we've habitually not done that. You know, we just check in on it every once in a while, but I'm up for learning and trying and and improving. So, I definitely think it's worth the **[74:52]** >> Okay. >> re the R&D budget. >> Okay. [sighs] I mean again it comes as as a suggestion. I think we've we've been doing this for such a long time in terms of discussing these things in public in that and I think this this keeps coming up. Uh we can defer it. It doesn't need to happen. Um, it's just something that I had to touch in the context of the pipe dream as I was looking at where the metrics are going, >> the feed requests, how is that like put together, what do you write in S3, all all those things like the different buckets, like all that stuff I had to go **[75:23]** through basically part part of part of this. >> I would love to remove just to chop that part off. >> It's just there as a thing that we do, you know, but yeah, doesn't have to be that way. That's just how we did it. So, do you have a thing that you would like to improve between now and the next Kaizen, either of you? Because I can go I can keep going through this list, but I'm wondering if there's something that you're thinking about or something that's bugging you that you would like to see improved. >> I've kind of just become content of late **[75:53]** with the way things are, Adam. >> Okay. Hm. I think the only thing I think about really is when I think about this system, it it's kind of sad in a way is that if eventually, and it's not the case currently, but if eventually the case is that our true traffic comes from a one of the spokes versus the hub, like how important does the hub remain? And the one thing we're not tracking really is **[76:25]** the MP4 file that we upload to YouTube in terms of the system, right? So we don't have a a store. So we have the MP3 for the plus+ version and the public version side by side in perpetuity for all of time back to episode one of all all all podcasts. What we don't have is that corresponding video file because it's not part of this pipeline. It's not part of the serving pipeline. And that's what I think about and I think about the analytics and the effort there. **[76:56]** You know, how does how does the the change to watching, viewing, listening patterns change the show over time change the system? You that's what I think about. So, I'm not really sure. >> I've certainly considered a a modification where we upload the MP4 to our system. >> Yeah. and let it redistribute to the various places that we want it to live. That's a much heavier lift and I think there's benefit at the moment at least. >> I agree. Yeah, **[77:27]** >> we still have the MP4 so it's not like we don't own that content. We just don't own it, you know, on R2 alongside our MP3s. >> And it's kind of a sad spend of money to store a file you never really you you send to YouTube one time, >> right? >> You know, psychic. You can also start to get like diverse with it and say, "Well, we're also going to be on peer tube and >> yeah, >> we're going to upload to Rumble or I don't know what all these places are now because people kind of scatter from YouTube, then they gather again and they scatter. But so far, our video, the **[77:58]** point of our video is to be on YouTube at the moment, >> right? >> And so we haven't thought beyond that. But if YouTube changes in some sort of dramatic way where I mean we've seen changes over the time where it's like let's be in more places similar to how we've we're not just on Twitterx anymore. We're on more social networks. There may be a day where that happens with YouTube and we'd be happy to have the pipeline set up to where we can just, you know, add another connector to it and say we're also on this video watching platform or we can view our **[78:28]** stuff directly, you know, watch it directly on our website and we'll let Cloudflare bear that burden. Although if it's passing through pipedream, we're going to have some serious bandwidth going through fast or through fly.io. So, those are things I've thought of, but I've never been even I at this point I'm not even close to pulling that trigger. Gard, you're on YouTube now, but you also have Make It Work.tv. So, you're actually tackling this to a certain extent. How do you do it? >> Jellyfin. >> I That's one. Yes. Jellyfin is if you **[78:59]** have a client. That's right. You connect to the server and you can download things. You can store them like offline on your device. So, it's like a media library, proper media library and a media server. Um, I find that works that works really well. I mean, I I always prefer watching it that way, but I also store it on on a CD, in this case, Bunny. So, Bunny has uh stream, I think it's called, and I think Cloud Lair has stream as well, uh where you would just basically upload um um media content, in this case, uh MP4 files. Um so, **[79:31]** the CDN part works well. I would love to replace that to be honest because I'm I'm not entirely happy with how the system works. The chapters are a bit clunky. There's quite a few things which are clunky. the the the trick play isn't great. Again, lots and lots of things. Just the way you upload things is just just just too much work. So, I'd love to um automate [clears throat] that. But YouTube really is like the main distribution mechanism. Not just because of how easy it makes it to just upload the file and then gets redistributed everywhere, but also it's almost like **[80:02]** you have something to sell. Do you go on eBay or do you go to a flea market or elsewhere? Do you build like your own shop to sell that thing? >> And eBay a lot of the time is easy because that's where the a lot of the buyers are. That's where people are looking and searching or Craig Craigslist. What if they Gum Tree, we had it in the UK. I think it's still a thing. So YouTube is a place where lots and lots of people already are. So in terms of distribution, it just makes it so easy. >> Um, [clears throat] now I like to give the option of, hey, if you if you don't want YouTube, that's **[80:33]** okay. You can also download it from the CDN that I pay for that I set up and in this case I haven't built but that's coming. So or or jellyfin. Now I don't know how many users would set up a jellyfin to be honest >> um for change log but I like the idea of basically h having that onetoone relationship between the creator and the watcher the listener the viewer that there's like nothing in between **[81:03]** >> so YouTube can't push its ads YouTube can't ban certain content in certain places if if it happens however it happens um again it's very difficult to know that because you need to be in those places, know how that works. And also like the idea of people being able to download the content and again the podcast players make podcast players makes it easy. For YouTube, you have to pay that premium. You have to go YouTube premium which I do and I have for many years. >> Um, and I think it works great. But **[81:33]** again, my experience, my YouTube experience is very different to most people because I don't think many people pay for YouTube. It's just like an extra expense. Um, so that's my take. [sighs and gasps] >> Yeah, it would be cool if there was a vibrant community of people that consumed video via open standards like they do with podcasts. I mean, the the coolest thing about podcasting is that phrase, you know, get this wherever you get your podcasts. And it's like that's **[82:04]** because it's an open standard that you can just directly subscribe to a feed and people can build apps for those. And that's amazing. That doesn't exist for video. Will it someday? Maybe. There are nerds out there and we've we get the emails about open standards for video podcasts. And actually Apple launched the iTunes uh podcast section with video podcast as like a first first class citizen. It's just that nobody the bandwidth was so expensive back then. This is like either pre-YouTube or like **[82:36]** right around the time that YouTube started and like people were just weren't watching. They didn't have the we just didn't have the technology to actually make that a thing that you just watch. You didn't have the phone. You had to like move the files around. They were large. I think Mac Break Weekly was like one of the only ones where I ever saw they were like shipping 4K video podcasts like in 2007 or something and it's crazy. >> Wow. That's hardcore >> because they come from the TV side, you know, where they're used to putting out video and where most of us are just coming from the audio side. Anyways, those things existed. Apple obviously **[83:06]** just kind of like it still actually is part of their their RSS spec, but no one uses it. Even Apple Podcasts, I'm not sure if it even uses it. Spotify has their own deal for video. It's not using the open way. It's using their own proprietary way. There's weirdness there where if your video file duration or details differ from your audio file, you may end up serving one or the other even to audio listeners, which you don't want to. And so it's just kind of murky right now. And um I think it would take some sort of a black swan event and maybe a **[83:38]** sea change in opinion and some sort of new tech that makes it feasible at which point I'm I'd be all about it. I just think right now it's like a lot of effort and it's a lot of effort to like re-encode your video into all these different formats depending on blah blah blah >> and then serve that you know a lot of cash misses if you got six versions of your video depending on the client and so it's like >> the storage too >> yeah it's like money time and effort for right now like a very minuscule **[84:09]** advantage >> let YouTube pay that price in their tech and their tech stack and their developers and their bandwidth and their servers etc. Yeah, that's that's my only concern is that really is is like where do we begin? Where do we reach diminishing returns in innovation, you know, with the CDN beyond MP3 and smaller file? It doesn't seem to naturally scale to the video because the incumbents have that solved in ways we just don't see that we need to solve **[84:41]** those problems. The only problem I guess that I see is the fact that we don't have this video file artifact alongside the MP3 artifact that is the same thing but a different flavor of it. Y >> it's elsewhere in our archiving stack and I would say largely inaccessible certainly not via an API you know potentially but it's not >> right >> I think building a standard or contributing towards a standard takes a **[85:12]** really long time yeah and lots of effort >> and that's why no one wants to do it and they're waiting for someone else to do it because they know how much of how much investment that takes. No one's in the mood for that type of investment. >> Um, I think it's going to be very interesting what happens with AI because there's a lot of money right now in AI and I think it will change, not be for long. So, where will that money go next? We'll see. Um, I'm definitely curious for for the next big thing which is coming. But um **[85:42]** one thing that may work well is if change log had an app had a native app whether it's an Android or iOS app and then you control how you display the video and the MP3. And if you had something like that then you'd be in full control how you could how you would expose your MP4 files and how you would integrate them in the player. And that's like a more holistic experience where you know you can do like transcripts really well. you can do comments really well, maybe integrated with some sort of Zulip or something **[86:14]** like that where um it feels like a it's like a system and it's more like a community of people that are interested in this type of things >> rather than you know just some content that gets distributed on different platforms. Again, the problem in that case is that most people are already on those platforms. So it's easy for them to consume things, but there will come a point where they just get they I mean they just want something different. The flow plane, I mean that's that's a new thing. Yeah, it's been around for a few years. So that's one example. There's another one. I forget what it's called. **[86:46]** I know that you can you can pay for Man, I wish I knew I remember the name. I was doing a research maybe like six months ago or like nine months ago. Actually, was more than six months ago. And I was looking at YouTube alternatives. And there's like this other um platform for media uh which stores and distributes higherend um like 4K videos, 8K videos, but for that you you end up uh paying so it's not free. And then you get creators that publish only on that platform. I forget **[87:17]** its name. Um I I can look it up. Maybe we can add in the show notes because I have it somewhere. Uh but that is another interesting thing. Vimeo. I remember when Vimeo was a thing, but I still I I know it's around, but I don't think many still use it in terms of like people going and browsing. I don't think it's even a thing. >> Yeah, they pivoted quite a bit. It's a very successful business to this day, but it's like serving enterprise and um more professionals who are using videos for various purposes, not as a **[87:48]** general consumer product at all. And video is hard because the transcoding part is really hard. And um yeah, I mean you always have like the trade-off. Do you pay storage or do you pay uh for compute as in do you transcode on the fly like Jellyfin and Plex does and then you need like GPUs, >> right? >> Or do you pre-transcode and you know you serve m you save multiple versions, you store multiple versions and then you and and then you serve those and then you have so many codecs. I mean it's just not even funny. like AV1, H.265, like **[88:21]** what do you pick? >> Um, different phones, different devices. It's it's not an easy like MP3 at this point is like almost like a universal format. >> There isn't a video equivalent. >> That's a hard problem. >> Yeah. Anyway, that is what makes video dramatically harder. And that's I think where the divide is at. And that's why I brought that up is because you got this divide of, you know, the potential of, you know, the precursor is that almost everyone that I'm aware of at least is still **[88:53]** paying attention to podcasts via not really a podcast client anymore. They're usually on some sort of platform and they're usually they're like they they ask me what show do I produce and I tell them and they immediately open up YouTube and they start searching for it. Oh, is it this one? It's like yes. Okay, so that's where folks are tending to go and here we are optimizing for this and the migration maybe that do the worlds eventually collide how do they work long term etc. **[89:24]** just don't know. And I think Yeah, exactly. Do you transcode it on the fly or do you make multi versions of it? I think you just don't do that unless you know for sure you should just transcode if you can. >> Yeah, I know the jellyfin that's exactly what it does and I really like it for that because it's very good on storage. But the CDN, the one that I use, it does transcoding so they store multiple versions. Um, now luckily I capped the maximum and I I think I only published 1080p on the CDN uh because of the **[89:54]** multiple versions which um they transcode and they make available. But on Jellyfinish is like the 4K one. So >> that's my that's my approach to it. >> What are your thoughts on number four here four five and the question mark here? >> Well, change.com new wiring. What I was thinking is there's a few few utilities that we're using that need an upgrade. For example, dagger needs an upgrade really badly. It's like such an old version on on the change log. Upgrade or replace still undecided. We'll see how that goes. Uh the deploys I wanted to **[90:26]** improve them for a while. There was always something else. So improving that time to deploy I think was like four minutes last or 3 minutes. There was 2 minutes at some point and went back to 3 minutes again. And I know that at least a minute and a half of that is fly. So what do we need to optimize there? So the deploys are a little bit quicker. Postgress I mean it's been stuck on I think 16 at this point I think 16 something. So maybe want to upgrade something there so we don't fall behind too much. Um and replacing overmind with run it. So overmind is a supervisor that **[91:00]** runs for example the log manager um the log manager it runs uh varnish vinyl that runs the proxies it runs multiple things in the context of the pipe dream and pipe really but um over mine sometimes some things can get stuck because of just how it's configured there's like some some duct taping there especially when uh the logs how the logs are streamed so that's something that I would like to improve and I know that run it. I've used it in the past. It's very reliable. It's a very old **[91:30]** supervisor, uh, very Unixy supervisor. So, I'd like to replace Overmind, which is go based with run it, uh, which is much more old school and it does everything we need. So, that's like an improvement, but that's a pipely pipe dream improvement. And the question mark was like, what else? I think we we tackled the question mark >> in in the conversation. >> Gotcha. >> All right. So, we're almost at the end. If you like this as a listener, as a viewer, you can like, subscribe, you know the drill. And comment, right? **[92:00]** That's something that um is also an option. I mean, we we touched on many things. Maybe you have a few ideas of how to do things better or maybe there's a few suggestions that uh um listeners and watchers have. Um I'll be more than happy to answer any follow-up questions as we will all. Um suggestions maybe for the next get together, for the next change get together. Um, you can do it on the YouTube video. I I think you uh do people b by the way comment on YouTube videos. >> Yeah, a little bit. **[92:31]** >> The ones that you post. Yeah. Okay. And do you reply to those comments? >> Yes. >> That's a lot of work. [laughter] I just discovered that recently last week. Oh man, like when you get like 100 comments, it takes a while to go through them. But uh it's it's a good problem to have for sure. Uh Zulip also works. I think all of us are there or even GitHub. That's also um we have the discussion for this kaizen >> right >> and just remember to tag me because otherwise I will miss your message. So I I have many things on mute and unless you CC I'll I'll miss it. **[93:02]** >> Absolutely. Same too many inbounds must get tagged or CCD. >> One more thing last thing. Last thing and then we're done. Okay. Okay. Last thing. >> So you know about make it work TV. Uh make it work. Club is something new. So the 100 g 100 gigabit home lab comes from makeitwork.cl club. Uh it's on school. It's a community of the most loyal make it work.tv members but also those that want to go beyond just watching. So the ones that want to **[93:32]** interact. We meet every two weeks. Both Adam and Jared have an invite >> while you were talking. I send you an invite so you can join. >> Uh you can see the various conversations which are happening there. And the next one is tomorrow. And it's usually every other Friday. It's usually 9:00 am Pacific time. Yesterday's tomorrow's one is going to be 7:00 a.m. Uh because some of us have kids and meetings and other commitments. So it's going to be before work for some people. We're be we'll be talking about a smart garage door **[94:04]** opener. >> Talk home labs. We talk Talos Linux that comes up quite a lot. Uh quite a few things. Kubernetes, it's all there. You can go log in and check it out. Uh Adam and Jared, [snorts] [sighs] you are part of it. I just wanted to get it I just wanted to get it into a point where there's like enough to show and enough for you to see. So, it's been going on for about a month now, a month and a half. There's plenty of threads. There's only I think 17 members, so it's not that many. It still feels like a **[94:36]** small community. Has a small vibe to it. Um but it's a bit like this but uh with with with more people so it can be a bit bit more chaotic. I think uh the 100 gigabit I think were maybe nine or 10 people so it was quite the group discuss u but still with a you know presentation and focusing on I mean you you've seen the video Jar so you know what it was like. >> Cool. >> But did you get your invites? Just double checking that in your email. I just want to make sure that that work. **[95:06]** >> I got mine right here. You got yours, Adam? Did you get yours? >> Let me see if I got mine. >> And then you can decide whether you want to accept it or not, but uh I just wanted that to be out there. [snorts] >> I do have my invite. I do see it. >> Yes, >> thank you. >> I have so invite. >> So, this is the change lock++ equivalent. >> You can think of it like the change lock plus equivalent. >> Ghart plus+. >> And yeah, you can drop any time um when we when we meet every every two weeks. And you're more than welcome to look at **[95:36]** the threads, comments, ask for like someone, for example, Misha, he was asking he wants to build his own router, like a router, how how the Americans pronounce it. So router, there you go. >> Yeah. So he he wants to wants to build that. And Nabil built, for example, he's like a smart garage door opener. He just didn't want to get out of the car to for the door to open. So now he has like the all that like program. So he's going to talk about it tomorrow. >> Interesting. And yeah, there's quite a few things. **[96:06]** So, check it out. Very cool. All right, [sighs and snorts] >> that was me. [gasps] How do we want to wrap it up? >> You put a bow on that that present there. That was cool. >> Mhm. >> Uh talking about a garage door open. Be kind of cool. >> I just talked to my phone. I just tell Siri, "Open the main garage." And she makes it happen. and it's just part of Apple HomeKit and uh the fact that my **[96:37]** >> garage door opener is on the network and it has those kinds of >> I really didn't do anything to make that happen besides just flip a switch and talk to it and that was kind of cool. >> Now sometimes she's like >> you don't have a main garage door and I'm like no no let's try this again. Hey Siri, [laughter] do this. And she's like, "Okay, gosh, don't shush. Don't be open to my garage, girl. [laughter] Gota stop her right now." >> Yeah. She was She was hearing me say her name. She's She's excited. She's always **[97:08]** like, "Can we open the garage, please?" Or maybe not. [laughter] Or maybe not. >> What are you keeping there? >> What are you like Breaking Bad sort of situation going on? >> Yeah. Yeah. Yeah. Same. >> No, no, no. It's not that. Yeah. [laughter] Um, so but but someone had to set up the smart garage door for you, right? I mean, did did you set it up because you need to have the whole it needs to be hooked up, right, to your home network. All I did was was enable uh the Wi-Fi access to my network **[97:40]** >> and the garage door opener is on the network. It has an app that runs it >> and the app allows me to install I guess it's been so long since I touched it so I don't remember how I did it but it was >> it was like shortcuts I guess essentially you can create shortcuts on your iPhone and so that's [clears throat] all I did was just leverage the shortcuts that talk to the app that has the authentication to the thing via the network and that's whether I'm at home or not at home so it's not even LAN bound it's it's WANbound so **[98:11]** it's really awesome I can literally in the mountains with very little service and I can tell my garage door to open or close and I can even tell if it's open or closed because if I say, "Hey, close it." She's like, "Uh, I'm closing your main garage." She's like, "Oops, already done." You know, it's like she reminds me that it's it's already closed. >> So, I didn't really have to do much to do that, thankfully. >> But to set it up, so for to take take a regular garage door, >> right? A normal one. **[98:41]** >> That part. A normal one. That's what Nabil did. >> A nonetwork. Yes, that would be that'd be dope. Honestly, >> that's what he's going to talk about. How he set up the whole thing, like all the devices. What did he pick? How did he connect everything? And like it was just like a regular garage door. It had nothing >> and then he made it smart. >> Well, the good thing about those garage doors, they tend to have an outlet which has two out two plugs. One being used by the garage door opener and then one that's used for nothing basically. So thankfully, if you needed an outlet for **[99:13]** your device to pair it to, I'm assuming the answer is maybe yeah, right. We'll find out. >> Make it work. >> Mine Mine is just M mine is just a regular one. Exactly. So that's what I want to do. Like if I if I wanted to set this up, what would I need to do to make it work in the as a smart one? I think the the thought pattern around how to tackle that, you know, I simplified it by just having one that was alreadyworked, but if you have one that is notworked, then there you go. You've got to create the network. **[99:44]** >> I know how we can end this show. I can just ask Ghart to review my new hat. >> Your new hat? Yes. I thought it was there. Yes. Oh, I'm so happy. >> What do you think? >> I'm so happy that was I think it looks amazing. >> I like this hat. I know you like blue. >> I love it. I love it. >> This hat's blue. It's blue underarm, man. I was looking for it everywhere. I thought I think I left it in Jared's truck. It makes me so happy to know that you have it. >> I got it here for you and we get together again. I will bring it to you. >> There you go. >> It's yours. It's yours. I bring you a gift. **[100:15]** >> It It suits you so well. I know that. I know you look like hats on. You're right. >> It does. >> Thank you. >> It's yours, Jared. Makes me so happy. That's your hat. So happy. >> Hats on to you, Jared. Hats on. >> Hats off to Gar. Hats on to me. >> I think we have I think we may have a title there. >> We might. Hats off to Garrett. All right, Kaizen. Good stuff, y'all. >> Kaizen, it was awesome. Bye, friends. **[100:45]** [music] >> [music] **[101:18]** >> Hang on.