950: Even SCARIER Web Dev Nightmares  ## Metadata - Author: [[Syntax - Tasty Web Development Treats]] - Full Title: 950: Even SCARIER Web Dev Nightmares - Category: #podcasts - URL: https://share.snipd.com/episode/068e60f2-0a9c-40b6-906b-bf20517786b6 ## Document Notes ## Highlights - **Refunds From Beyond The Grave** - The developer shipped an authorization-only payment flow and never called the capture endpoint, leaving donations uncharged for months. - They wrote a script to capture pending payments and emailed affected donors to resolve the issue. Transcript: Wes Bos By the way, I wrote all the titles for these ones. So please appreciate them. Once, I built a website for a foundation that did medical research. People could donate money for special occasions like birthdays, but mostly in memory of somebody who has passed away. That's nice. Everything worked perfectly at launch, and I even tested it myself. The payment showed up as reserved on my card. All good. You know, like it reserves the money on your card. It hasn't actually gone through. Sometimes you need to do that. That's a part of credit card processing. Six months later, I'm at my summer cabin with a beer in hand and the grill going when the phone rings. None of the donations have come through a donator uh was complaining that their donation never came through strange after some panicking and digging through all the test suites were Still passing i realized i'd done everything right except one tiny thing i never hit capture the payment provider thankfully out of business today, had a special endpoint for capturing The payments. Stripe is the exact same way, by the way. Auto capture was optional, but not default and very hurried in the documentation. So all those donations and memory for dead people have only been authorized, but never actually charged. Oh, no. Oh, so people have been donating for months to honor loved ones and the money never moved. Total panic. Imagine your entire business is taking money for people. You don't do the one thing you're supposed to do. But luckily, the payment gateway kept pending transactions for six months. It had been five months in 20 days. No. I wrote a Quip script to capture everything and sent out an email to everyone affected. In the end, it all worked out. ([Time 0:01:38](https://share.snipd.com/snip/b9baa4ce-b204-4981-8a8a-6d146c0c8253)) - **When A SQL Dump Becomes A Guillotine** - A SQL export from staging included DROP TABLE commands and wiped production content during an NHL site launch. - The team had to delay launch and reenter or restore lost content without a prior backup strategy. Transcript: Scott Tolinski I built a very nice website for the National Hockey League. A whole team of content specialists, about 10 people, start filling the site with content. The day before launch, the whole team even starts working from 6 in the morning for their last big push of content. I export a single table from our staging site, not content for a special feature, to SQL and blindly import it in the production database. Oh, blindly import it in the production database. What a sentence. The whole site goes down and my new table is left. Only my new table is left. Oh, It turns out the.SQL export didn't just contain a create table query, but also a drop or all drop table queries. That sounds like an AI-style migration, if you're asking me. Oh, gee. So they ran a SQL command that just dropped all the tables and then added their new table. The latest database table was from three days before. So all of the content was gone. Yes, the launch was delayed. Yes, I was ashamed. Yes, I always check my exported.sql files from now on. This is now more than 10 years ago, and I still get chills thinking about it. ([Time 0:04:22](https://share.snipd.com/snip/96245d14-30db-47ad-bd6e-bb6c273b8404)) - **A Single Indent Cost Hundreds Of Thousands** - A whitespace/indentation change in Python moved retry logic outside a try/catch, triggering repeated expensive video generation retries. - The bug multiplied cloud costs by 5x and cost the company hundreds of thousands of dollars. Transcript: Wes Bos I'm a full stack dev that writes mostly TypeScript, of course, like any good full stack dev. I'm a seasoned dev with lots of experience writing backend code, managing infrastructure, writing helm charts, setting up alerts, yada, yada, yada. So I'm not afraid to jump into some Python code base and make some changes. This sounds like something I would say. I've been writing a lot of Python lately. The Python service I've been working on was originally written by a messy ex-Google engineer, and I'm stuck with them. Shame. And a broom cleaning up the mess. A typical Google employee writing a bunch of messy Python. This particularly messy Python service was hitting Google's cloud video generation model. Okay, so this is relatively new. So I was cleaning up some of the messes. Every time video creation fails, multi-line logs are barfed up over the standard out in using a print statement instead of our structured logger. So I updated the main.py file in our code base to use our logger instead of print. So doing so required moving around a few things, et cetera. Makes sense. I opened up a PR. I ping my trusted front end dev to rubber stamp it and logs are pretty and people can create VO videos through the API. Everything is good. All right. Three weeks go by. Scott Tolinski That's a long time, folks. Wes Bos That's a long time for the just wait part. And I see a group of people crowded by a laptop on my way to the office and the CFO is standing by them looking moderately perturbed. They are wondering why our VO2 costs last month had shot up 5x. Certainly nothing to do with my innocu logging change right well time to find out my little cleanup pr introduced a line that unindented the message acknowledgements outside of the Try catch block resulting in failed video generations example fail due to safety checks yeah like you try to create video with something that is is not appropriate they'll they'll Fail To be retried five times. Just for reference, a 30-second video generation on Google costs about $10. It's an expensive API, so this literally had cost the company hundreds of thousands of dollars. The crazy part is it was just a white space change. It didn't even show up on our code review platform, which is graphite. So the dev I ping rubber stamp the PR didn't even notice the change. Frickin Python with his indentation. I used to be such an indentation fan. You know, I used pug and stylus and coffee script and everything was indentation based. ([Time 0:06:40](https://share.snipd.com/snip/73dab329-e4a8-46c8-8c19-7a632c377341)) - **Subscriptions That Billed Every Day** - A subscription scheduling bug placed many customers' next shipments on the current date, creating thousands of extra paid orders. - The team used a pre-launch database backup to identify affected customers and reimburse them over a week of recovery work. Transcript: Scott Tolinski Over order nightmare. I had been working at a premium pet food e-commerce store. To give you an idea, we average about 4,500 orders a day with about 300 of that amount were subscription orders. I subscribe to some dog food. Maybe I'm a customer. The customer had an option to start a subscription of their pet food orders. This would create an order on the interval the customer set him or herself however the person that created the whole subscription system went into a burnout and eventually left the Company i joined when that person uh was already burnt out let me tell you working in reoccurring billing without like to do it for you can absolutely drive you to burnout. Absolutely. Billing. What's the worst? Wes Bos Can we ask this? What's the worst parts to work in? I'm going to say single sign-on. Reoccurring billing, single sign-on, maybe auth in general. Access control. Scott Tolinski I don't mind access control either i think those are the worst anything that involves kind of like ping-ponging back and forth but like with reoccurring billing like testing it you Have to be it has to be done through webhook so you can send a test webhook and then like when an actual subscription like webhook comes in yeah you could have tested it, whatever. But it's still like, it's it's still always very scary for for like, what actually is is doing that there was a big demand of the customers that they could choose which day of the week they Would receive their food as hybrid remote work was very popular in our country. They wanted to receive their food when they worked from home. This would be for regular orders and subscription orders. For example, for regular orders, you could choose that you would receive it the first Thursday after placing the order. This got really complex as we sold it to about 20 countries. Oh, yeah. Oh, man. We worked together with different shipping companies depending on the warehouse that order came in from. These shipping companies all had different lead times for each country. Eventually, we worked this out and thinking about the implementation of the existing subscriptions. For running subscriptions, we had to calculate what was the last day of the week they received their last order. Then we calculated the next shipping date so that it would receive on the same day. But with the given subscription interval calculated into it, we had to keep in mind the subscriptions were running so that we could need to calculate the time from their last order until Now and subtract that from the interval they had specified. We then took the closest day calculated from the next shipment date and would then change the shipment date for the existing subscriptions. Oh my gosh. I i hate subscriptions with digital products with shipping and then shipping dates that things have to arrive in take me out back i'm done man yeah logistics i don't like logistics i'll Tell you about that I'm not good at planning. Jeez, that's crazy. At the time, I had been there for about a year. The subscription system was a bit of a gray box for everyone, where we could change some things, but nobody on the team had full context on how it all worked exactly. And there was no documentation. As I said earlier, the original creator had left the company. Had been fixing some bugs in the system at my time there and felt confident to implement this new feature. I took some days to get familiar with the system and wrote out exactly how I would change it in the areas that would be touched. From running subscriptions, I wrote a migration strategy that discussed everything with the team and my manager. Everybody gave the green light. We tested the implementation. We put this into production. Everything ran smoothly and we were getting positive feedback from our customers. Oh man, the hammer's about to drop here. But after 10 days, we were getting complaints. People who had subscriptions and had their next order deliver in these days had been paying every day since the feature launched. That's the hang on. I knew I was waiting for it. They'd been paying every day. Turns out we had been calculating the next order date wrong, and once the new date was calculated, the next order date was placed on the current date. This is a running system. It had been going for 10 days before we noticed. Thousands of extra orders created, packed, and shipped. Oh, no. Wes Bos It's not even like you're getting like a hard drive or something fun. It's like dog food. It's so huge. Scott Tolinski I'd take 10 extra orders of dog food because you know they're going to refund it at this point. Luckily, I created a backup of the database when I launched the feature into production so we could see who had upcoming orders. ([Time 0:10:55](https://share.snipd.com/snip/ea1d5609-e595-4f3c-838a-e65431e42774)) - **LocalStorage Turned Into A DDoS Hammer** - An analytics client moved from localStorage to in-memory when storage filled, causing duplicated events and huge scale issues. - The pipeline ballooned to billions of events per day until a dedicated team fixed the overload and moved to IndexedDB. Transcript: Wes Bos The year was 2019. I was on the experimentation platform team at the time, but was given responsibility over everything analytics after being told it should be in the keep the lights on mode, which is Minimal development, just keep it running. The service API was based on Segment.io. In order to leverage Alaskian specific wrappers. I changed the name. Wrappers around the clients. Alaskian developers had been given a halt order for feature development after being plagued with reliability issues in the front end that devs were flying blind to. It was decided that the best way to give front end quick visibility into performance and reliability was to leverage the analytics platform. The front end JS client used a library called segment local storage retry in order to capture as many events as possible. This is pretty common when you're sending events. You'll throw them in like a variable or you'll throw them in local storage so that if there is if they're blocked by like a thing, if the person goes offline for whatever reason, at least They're in local storage and you can retry over and over again. The queues for processing and to be able to recover queues from tabs that have been closed. That's another good reason for it. We started getting reports from internal users that local storage was filling up and it was due to the analytics client. After digging around, I found that the client would detect that local storage was full and switch the entire storage solution to memory, which prevented any further cleanup inside Of local storage. Once we got that fixed out, we pushed it to all products only to start causing more instability to our backend analytics service. So it took us a while to realize that we would occasionally be seeing single analytics event be able to be duplicated tens of thousands of times, which was also causing additional slowdowns On the other clients. It was a hard case for us to get budget to add more fixes for a service that should have been in maintenance mode. But the volume it was needed for processing grew from $1 billion today to $3 billion with the influx of additional monitoring front-end teams that were trying to be shoehorned in. Yeah, this is monstrous. Like, we've had folks on the past from Sentry when it's just like, you're essentially getting DDoS. Like, being able to just receive the events from everybody is a massive, massive thing. Soon, it became apparent that the fix was causing more slowdowns on client browsers and also sometimes prevented garbage collection from running due to the frequency of local storage Checks. This is really interesting because local storage is a synchronous API. So every time you're touching it, you're actually running a synchronous function, which if you do it often enough can start blocking the main event loop. In testing I even managed to crash my entire laptop. I was thinking we should do a video of who can crash the browser tab first. Who can write the worst code to crash a browser tab. I think that would be good. Scott Tolinski I can crash a browser tab in seconds, milliseconds. I can crash it faster. Oh. You know how to do it? You just do it. I mean, just use effect. Just use effect. Use effect. Wes Bos We tried a few hacks to get on top of this, but at the end of the day, we were trying to treat local storage like a transactional database, and it became very clear that only one thing we could Do is move the functionality to index db which removed a lot of complexity and overhead and almost all of the issues went away after that except for the fact that many devs kept throwing More and more load at it because the analytics pipeline turned into a hammer for many problems but that's a different story moral of the story, don't treat local storage like a transactional Database. Yes. Yeah. As much as, especially like you can, you can run out, you can run out of space in local storage. And every time you write to it, it only holds string. So you gotta, there's like a JSON parse, you gotta pay for there. And a second story here is the analytics is not designed for platform monitoring. I think there's another tool for that, like dot, dot, dot, century.io. Why, thank you for the ad there, dear friend. I don't know what your name is, but yes, you guys should be using century at century.io forward slash syntax and get two months for free using the coupon code TC treat. Scott Tolinski Yeah. Take a bite out of your errors. That's good. Wes Bos P S the main problem, uh, was overload Alliance on the analytics problem kept growing despite being in maintenance mode. And it hit a peak of 13 billion events a day before dedicated team was assigned to reign in the cost. ([Time 0:16:51](https://share.snipd.com/snip/e18c50b7-e639-4e16-8ef2-5306a3626461)) - **When rm -rf Meets No Backups** - Running rm -rf on the wrong path destroyed a production server with no working backups at a real estate company. - Rackspace support rebuilt the server and recovered most data, but the engineer lost days of work and learned to check backups immediately. Transcript: Scott Tolinski Right the next one rack spaced out i'm gonna little arnold there i've run the dreaded rm forward slash star when I meant to run the RMRF dot forward slash star. Wes Bos Never. Never do this. Scott Tolinski On the production server of a large commercial real estate company. I didn't use Git or any other version control. Backups had been offline for years. The server was completely trashed. But the Rackspace tech was entertained. Can you imagine getting this email? They'd be like, hey, brother, I just ran RMRF on my entire server. 23-year me was thoroughly impressed that the dude stayed on the phone with me for hours. Ended up spinning up a whole new dedicated server and moving over what pieces he could to the new box i lost a few days of data in the end but got most of it back up well that's that's a lucky Uh end result one time i ran get clean on my rude directory uh i did that one time i think i talked about it one of these spooky episodes and then uh i did i deleted a bunch of um video production Files i had going and kind of borked my system. But I noticed what I was doing. So I was really quick on the command C. But it's still chomped through some stuff either way. Very scary. Wes Bos Everybody go right now and check that your backups are working. Because, man, they often are not. And when you need them, it's not the time to find out they are not working. And also, like, shout out to, like, support engineers or, like, the technical support staff that actually know what they're doing. ([Time 0:22:51](https://share.snipd.com/snip/0cd93abd-69b6-4df7-8362-bbfd06ab9852)) - **Formatting The CEO's Laptop** - An IT technician accidentally formatted the CEO's laptop during routine imaging and was fired the next day. - The incident underscores the need for multiple safeguards and verified backups for high-profile machines. Transcript: Wes Bos The process involved wiping the ORM images and loading custom images on my colleague and I maintained depending on the laptop or in rare case desktops. My job was very routine and robotic. I loved it. One day, however, the CEO's laptop was brought in for servicing and definitely not a wipe. Sadly, in the routine of my job, I formatted a CEO's hard drive. So true. You do something every single day. We have a backup of his laptop before, but it was not one of the CEO's laptop that was backed up. I was looking for work the next day. Oh, that sucks. ([Time 0:25:19](https://share.snipd.com/snip/e133149c-f3d6-44cd-96fa-ca02a8ae32ce)) - **Staging Sync That Erased Orders** - Syncing a staging WordPress site to production overwrote recent e-commerce purchases and lost days of order data. - The freelancer kept a three-star review but the client had to reconstruct orders from logs or customer messages. Transcript: Scott Tolinski I was doing some freelance work for WordPress when I was on a hot streak with quite a few jobs done at all five-star reviews until this job. It was actually pretty simple. It was just a WooCommerce site that needed some minor changes. So I did what I always did and created a backup, a staging site, etc., and got to work. I completed everything, uploaded onto staging, and then got the client to overview it with a green checkmark. Awesome. They were using SiteGround, so I was able to just hit the sync button, merge the staging sites to production, forgetting that this was an e-commerce site. I just overrode the database entries for all of the last few days of purchases. There was no way to tell who had placed the orders or when or for what. Suffice to say, the client was very upset and I still got lucky. I still got away with a three-star review, even though it should have been a lot worse. ([Time 0:26:53](https://share.snipd.com/snip/9e895932-1595-463b-9669-dbb91d8cbd20)) - **WP-CLI That Bombarded Customers** - Running WP-CLI during a shop migration triggered WooCommerce emails and sent 18,000 notifications to customers. - The developer salvaged the relationship and kept five-star reviews after urgent damage control. Transcript: Wes Bos Whoa. One time I was working on converting an old shop site. This guy submitted like 10, 10 WordPress horror stories like he never learned from his thing. The guy wanted his own site. I was doing everything in WPCLI to speed things up because there was tons of data. Well, I didn't realize that WPCLI was going to trigger the WooCommerce email system as I was migrating the data over. It sent out emails to people their purchases. There weren't actual charges made to thanks goodness, just re-triggering of email notifications. And the guy I was doing the job woke up to over 18,000 emails from angry and confused customers. Luckily, I was able to salvage our relationship and get the job done with a five-star review. ([Time 0:29:06](https://share.snipd.com/snip/4fbc2a49-548d-4bf8-9926-5f9dc84ed6c1)) - **Local Dev That Charged Everyone Twice** - A local dev environment triggered production Stripe webhooks and double-charged about 5,000 customers, costing many customers and losing clients. - The agency lost the retainer and learned to isolate live payment keys from local testing. Transcript: Scott Tolinski I was working for a small WordPress based company when we had a client who was thinking of hiring us on retainer. So we said we would spend a lot of time to look at their site and their project to get up and up to speed and discuss what we could offer them. One of the big things they needed was help with when they approached us was dealing with stripe. The way we set up was not correctly doing something. I don't remember why exactly. So we asked if they had any custom code within the site dealing with stripe and they told us no. When I pull up the site to my local machine and spun up in my dev environment at the time, and I start to look through some things. Well, it turns out that they were wrong about custom stripe stuff. And there was some code that automatically ran a bunch of webhook stuff when loaded locally all of their customers uh around 5 000 got a second charge to their credit card for the month No matter how recently it was paid all in all they lost about 500 customers by the time we talked to them the next afternoon we were not hired on a retainer and we learned some very valuable Lessons that day you didn't even get hired and they lost 500 customers for that's a bad day office you know what they probably had the stripe api keys in the database and he pulled the production Wes Bos Database locally yeah holy don't put api keys in your i guess in some cases you have to but man that sucks a 10 loss of all their customers and on top of that how many of those 5 000 people that Got the second charge like filed a like a charge back because that's 15 bucks out of your pocket every like regardless if if you the way that stripe works is if you can refund it before they Do a charge back charge back then you're in good shape but if you don't if they file a charge back and then you immediately refund it you still pay that $15 even if you win or regardless. So like, I don't know, 5,000 people, maybe 1% of them charged back. That's still 750 bucks out of your pocket. Scott Tolinski Yeah, if you want to get piss off people, your customers, start messing with their money. ([Time 0:30:53](https://share.snipd.com/snip/f9010e60-05cd-4a99-aa86-3cd72df387cc)) - **Forgot To Update DNS, Site Turned Adult** - After switching VPS providers, a portfolio's DNS still pointed to the old IP and became an adult site when the IP was reused. - The developer discovered the issue only after a friend notified them weeks later. Transcript: Wes Bos This is a really quick one. I switched my VPS provider and forgot to point my portfolio to the new IP address. How do you do that? A month later, a friend texted me saying my portfolio was turned into an adult website. How can you not check it? Dude. Scott Tolinski I guess that ip address was reused and you just never changed where it was pointing to that's great yeah one of the first agencies i worked for we were doing wordpress sites and the only Site i'd ever worked on it wasn't even my my client but um it had gotten hacked somehow i don't know how and that what had happened to it. ([Time 0:33:25](https://share.snipd.com/snip/5cfc9d3a-a606-467e-b79c-3de9768f945e)) - **Push Notifications About Deceased Artists** - A push notification system sent alerts tagging deceased artists as live, causing notifications about dead artists to fans. - The team added guardrails to check for deceased status before dispatching real alerts. Transcript: Scott Tolinski The only issue being that that artist had died a few months prior. Oh, no. That's a much bigger oh, no, than I had initially given that. Oh, my God. Luckily, I work for a nice company. No blame was placed on me, and the new guardrails were put in place before tagging deceased artists and test events, dispatching real alerts. Bro, that's two now that have dealt with people who have passed away that is very spooky uh that i give that one 10 ghosts out of 10 people talk about like vibe coding and building an app Wes Bos In an afternoon this is the stuff that you're not told about how these companies have unfortunately push notifications about a dead artist to those people. And now there's some code in their code base that checks if an artist is deceased or not before it sends events. ([Time 0:34:39](https://share.snipd.com/snip/26ed0971-c149-44e4-91da-cd6360c576ec)) - **Hotfix That Broke CI Across Repos** - A library release lacked failure states and caused downstream Cypress tests to break, blocking CI and prompting rushed hotfixes. - The team accidentally released a hotfix on the wrong branch and spent days restoring stability. Transcript: Wes Bos I once worked on new features that introduce API calls to a somewhat complex web component. From everything my team tested, it worked great. We thought we had accounted for everything, so we push it up to our repo, get it approved, publish the version, and get it installed in our front-end applications. Fast forward a smidge, the reports start to roll in that tests in the front-end application are failing and holding up CI and CD. The common thread amongst the reports was the test suite can't find any elements. I don't pay much mind because we ran a colossal Cypress test suite for a monolithic repro. So that could be anything. And that kind of thing has happened before. It's so true. Your Cypress tests are failing. Almost nobody goes, oh, must be actual error. Your initial thoughts are just like, probably Cypress is acting up. Then I see the error. Crap. This is our API call failing in their local dev environment because we didn't add a proxy or dummy data. We forgot to add a failure state. And because we didn't catch the error, it's throwing at Webpack's red screen of death, which blocks Cypress from querying anything on the page. We rushed out a failure state, which solves a problem in our main development branches. Nope. The hotfix branch is running into the same issue. Let's get that fix out. Oh, crap. They weren't supposed to get the absolute so they did a hot fix on uh not on their production but on their i guess their main branch which is not yet released so they pushed out a bunch of Code that was not meant to be pushed out yet suffice to say it was a comedy of errors that took days to fix and if we had century in our build system we likely wouldn't have run into this issue. Thank you, Sentry.io forward slash syntax. Use coupon code TastyTreat. ([Time 0:35:40](https://share.snipd.com/snip/95f3b9a7-fd79-41cc-a459-117c42216f0d)) - **Prank Redirect Nearly Lost A Contract** - A coworker redirected suspected hackers to an adult site as a prank and accidentally sent the boss to it during a demo. - The company nearly lost a contract and the prankster narrowly kept their job. Transcript: Scott Tolinski In a company I worked where everything was basically in shambles and barely making ends meet, a co-worker thought it was funny to somehow magically redirect hackers to an adult site When he thought something was of malicious intent the boss made a demo for the company that was playing during the demo he was redirected to the adult site several times it was a miracle We didn't lose the contract and that he didn't get fired you sent your you sent your boss to an adult site? Don't put adult sites in your work. Wes Bos Don't do that. Never, ever. Never. Ever put a, even put a swear in. I was having some frustrations with my recording the other day. I was sent like a screen recording to someone. And I sent it to Randy, our producer as well. And the name of the screen recording was F this. And guess what? They saw never, never put swears or any adult stuff or or even like think that you understand. Because also a lot of this, like trying to catch people doing bad stuff, you're going to eventually catch somebody who's not doing bad stuff for whatever reason don't do that yeah that's Scott Tolinski Not a don't do that you can you can just send them to a blank page that's just as easy and they're like no harm no foul you can trap the you can do honey pots you can do all that stuff and you can Still annoy them by i had a band system on level up that would like act as if they weren't logged into the site so then they would go to try to log in again but then the login would fail because They were already logged into the site oh yeah like and that way like if if for some reason someone ever got accidentally stuck into the that ban system it would just be like hey i'm trying To log in the login says this. Oh, let me fix it for you. You must have tried five different credit cards 40 different times and the system failed. You know, like you must have hit these specific criteria, but like, don't send them to an adult site. That's bad, bad idea. ([Time 0:37:55](https://share.snipd.com/snip/ed338e30-406a-4da7-9f35-8636bbbbed90)) - **Dev Mode That Cost Everyone Bonuses** - Leaving a development SMS mode enabled caused the company to be billed for free SMS numbers, wiping out the bonus pool. - The CEO forgave the engineer but warned them to prevent recurrence. Transcript: Wes Bos One, instead of making money, you spend money. So back when I started web development i was working for one of those sketchy cell phone game ringtone wallpaper companies and we're billing the victim i mean customer via premium sms Messages that's how it used to be you want a ringtone you you send an sms for development and testing purposes we had regular sms numbers that we use that didn't bill our cell phone. Okay. So you have like a, you have a premium one, which you make money. You have a regular SMS, which you actually, you pay to send. One morning, our CEO called us into a meeting to give us the bad news that there wouldn't be a bonus that month as the SMS provider had screwed us over. Instead of paying out the bumper sale, they were billing us for excessive use of the free SMS. I'm sure you had already guessed what happened. So at a moment, I had to sheepishly confess. I expected to be written up for having left the dev mode enabled in production, but I had an amazing CEO that just said, make sure it doesn't happen again and left it at that. Like you cost not just the company, but your coworkers real money by leaving a dev flag in production. ([Time 0:40:03](https://share.snipd.com/snip/f899d89a-d471-4268-a2cf-509b6cd1b78c)) - **Silent Certbot Failure Lost Recordings** - Certbot silently failed and Let’s Encrypt certificates expired, breaking RTMP recording streams and losing meeting recordings. - The team renewed the cert and purchased longer-validity SSL after losing hours of recordings. Transcript: Scott Tolinski How to lose hundreds of hours of meeting recordings in one easy step. Just let Certbot fail silently and wait for the SSL to expire. Hmm. We record our Zoom meetings through their RTMP streams. All of these streams land on our loyal ninja server which is tied to a domain name and secured with a let's encrypt ssl certificate like responsible grown-up adults we even let certbot To handle renewals automatically yeah that's that's fine then one day during the mysterious deployment probably a friday certbot stopped working. No alerts, no warnings, just quiet little failure sitting there, waiting. Days later, when the SSL certificate finally expired, boom! Every single meeting stream started failing. And how did we find out? Not through fancy monitoring or alerts. An employee casually said, hey, that little streaming icon isn't showing up anymore. Translation, congratulations, you've lost recordings. A round of chaotic debugging later, we discovered the issue. The server was alive, but the SSL was not. We renewed the cert, the streams returned, and we immediately begged the leadership to approve buying proper SSL certificate with a longer validity because apparently relying on Cert bot alone is a bit like relying on goldfish to guard your house. Moral of the story, if cert bot fails quietly, it's not the failure that hurts. It's the moment SSL expires that breaks your soul. ([Time 0:41:29](https://share.snipd.com/snip/1a03b2ee-f87c-4744-bb3d-ef43a7b6b470)) - **Name Server Change Killed Email** - Migrating DNS to Netlify removed Google Workspace MX records and caused email to bounce for two weeks. - The designer restored MX records in Netlify DNS and crafted apology emails while avoiding further backend work. Transcript: Wes Bos Oh, classic. It's always about DNS. I had been debating about staying with Bluehost or moving over to Netlify for my small business website. I was tired of paying for hosting and figured a free Netlify plan was sufficient for what I needed. I mean, my site was pretty simple, just for a portfolio, some case studies. And why was I paying $120 a year to Bluehost when Netlify could host it for free? WordPress was over guilt for what I needed. Now, I should mention, I'm a designer who dabbles in front-end development. I can build it, style it, make it look gorgeous. But deployments and server-side stuff? That's not really my wheelhouse, and I'll usually work with other developers that would handle those things. Still, I watched a few YouTube tutorials on Netlify, and everybody made it look so easy. How hard could it be, right? So one Saturday morning, in the air of confidence in an iced coffee, I decided to cancel my hosting and begin migrating to Netlify. I signed up for Netlify, connected my GitHub repo, and watched my site deploy in minutes. It felt like magic. Then came the DNS settings. Now, I know DNS was important, as we all do. But when I logged into my domain registrar and I saw that wall of cryptic records, A records, MX records, C name, TXT, SPF, my brain just went, yeah, I got this. Shouldn't be too hard. Updated my name servers to point to Netlify as their documentation walked through and watched the little propagation may take 24 to 48 hours pop up. And I felt like a tech genius. Is this really what backend devs deal with? Super easy. As a cell certificate check working. I gave myself a little victory fist pump. What I didn't realize is I had removed my email, which was running through Google Workspace and those MX records I casually deleted. Those were the only things telling the internet where my email lived. For two weeks, it felt like months. I'm sitting there thinking, wow, business is slow today. Nobody's reached out. Guess it's a quiet week. Meanwhile, my email service is completely non-functional and every email sent to me is bouncing into the void. Clients trying to send me project files, potential leads, and time-sensitive requests all gone, never received. I get that way sometimes where turn on do not disturb for the entire day. And then I go, wow, nice, nice, quiet day today. Oh, shit. Do not disturb on all day. Hey, I've been trying to email you all week. Are you getting my messages? They keep bouncing back. The panic like that set in. Oh, no. What? But that's where it gets worse. I had no idea to fix it. Remember, I didn't know backend network DNS stuff. So I did what any panic designer does. I started Googling. I spent the next two days. Brother, you could have DM'd me and I would have fixed it for you in like 10 seconds. Oh, going down the most frustrating documentation rabbit hole of my life. I've read Netlify's DNS documentation, which would tell me to check my email provider settings. So I go to Google Workspace documentation, which would tell me to configure my DNS records with my hosting provider. So I go back to Netlify docs. I was stuck in an infinite loop. I even went to ChatGPT and gave it to me back and forth in the same flow. I watched YouTube tutorials, read hundreds of Reddit's thread, all with different variations of how to fix DNS records Netlify Google workspace. I finally pieced it together enough information to understand the need to manually add Google's MX records back into the Netlify DNS settings. I copied those magic strings of text from Google's support page. I pasted them into netlify i waited for what seemed like another agonizing 24 hours for propagation and then i finally emailed myself started working again you don't even need to wait 24 hours there you don't know there's like two things with dns is when you're changing the name servers you're literally changing who manages your DNS. If you can change it from provider to provider or you can just use your registrar to change the DNS. You know, that's what I do. I put them all in Cloudflare and then I just point them at whichever servers you actually need because that way you don't have downtime. But also like when you import a domain name into cloudflare it scans the domain name for as many records as it can because you could accidentally be bringing down so much stuff more than Email if you don't know about it that's the fun part i craft the most embarrassing sorry for the technical difficulties email i've ever written and sending it to every client i could Think of i had to do damage control apologize prof and basically admit that yes, the UX guy broke his own website user experience. ([Time 0:44:05](https://share.snipd.com/snip/f478fb91-8094-4814-8e48-272932e96e11)) - **Cache Mismatch That Whited Out Users** - A minor dependency mismatch slipped past local cache and caused a white screen in production on Vercel for many users. - The engineer rolled back the deploy and added checks to CI to catch version mismatches. Transcript: Scott Tolinski Ruins everything around me. Cream, get the money. Dollar dollar bill, y'all. First day on the job was for a small startup serving hundreds of thousands of users daily. I picked up an easy ticket, updated some dependencies, ran fine locally, ran the test. CI was fine. PR was approved and merged. I headed out for lunch with the CEO and founders who were in the office that day. Well, a nice little day at the office. Mid-conversation, his phone starts buzzing. He checks it and goes pale. Sights down. White screen. My stomach drops. That two-line PR? No way. Turns out our monorepo had the same package at two slightly different minor versions, locally and in CI. Yarn's cache handled it fine, but Vercel's build didn't use the cache. Whitescreen for a thousand users. The CEO stepped away to make some calls. I pulled out my phone, jumped into the dashboard, and rolled back the deployment myself 12 minutes of downtime my second pr implementing sync pack lint to the ci to catch version mismatches Oh yeah build cash bro people always get annoyed ([Time 0:50:03](https://share.snipd.com/snip/309a5bc3-8f0d-4eb3-9945-3961329907df)) - **Restart-On-Fail Ripped Out Mall Ventilation** - An IoT sunlight-guiding system restarted crashed apps repeatedly, causing motors to torque and rip fiber through mall ventilation. - Restart-on-fail plus a faulty encoder caused physical damage and six-figure equipment loss. Transcript: Wes Bos I ripped out and destroyed the ventilation system at a US mall, along with $100,000 in equipment from a single retry. A number of years ago, I was working as an embedded engineer for Internet of Things products. This specific product is designed to guide sunlight into building. It consists of a large array of approximately eight inch lenses and a big stack of fiber wire led into the building as special fixtures, which is lamps scattered the sunlight in the room. That's crazy. So it's sunlight coming from outside being piped through fiber and then into lamps. So like they're lighting the mall without any electricity. It's that easy. Sun hits a lens, travels through the fiber wire and lights up the room. The entire lens. Okay, he just explained how it works. The entire lens. Sorry, I didn't read this whole thing because I was like, this is going to be good. The entire lens array is programmed to follow the sun with fractions of degrees of accuracy. Everything in these projects is expensive because of the insane tolerances required for good light, but the fiber wire especially needs to be custom made. Normal fiber for internet audio, et cetera, do not work for visible light. They are made to transmit signal frequency data, not broad spectrum light. Interesting. One of the first of these devices was installed in a mall in the US. The fiber wire was simply fed down into the mall through the air ducts. The code had validation checks for everything, including valid encoder entries. That's the position of the motor, right? For the motor that spun the entire array of lenses. If any value was wrong, the application would just stop. Last minute, we made, these are always the best. Half of these stories are, last minute, we made a separate decision. The operating system should just restart any application that crashed. Well, unfortunately, one of the encoders turned out to be faulty. The application would start up, validate everything, start to spin the device, realize the motor wasn't turning, and crash. Then it would start up, spin for a bit, and crash. For an additional context, due to the insane precision needed, the motor was mounted on a 1,000 to 65,000 gear. It took 30 minutes to rotate 180 degrees in full speed. It had crazy torque. It took a few days before the thick fiber wires finally ripped out large chunks of the ventilation system in multiple rooms and then finally snapping in half. Somehow I made those two decisions all by myself and I did not make the connection. This guy caused physical damage in the magnitude of hundreds of thousands of dollars. ([Time 0:51:54](https://share.snipd.com/snip/557d2803-8466-4cc6-8dd6-3046089322de))