949: Web Dev HORROR Stories + Spooky Trivia!  ## Metadata - Author: [[Syntax - Tasty Web Development Treats]] - Full Title: 949: Web Dev HORROR Stories + Spooky Trivia! - Category: #podcasts - URL: https://share.snipd.com/episode/db60f2a7-4bf1-4a65-a82d-d4946df35d3c ## Document Notes ## Highlights - **Keep Recent Tested Backups** - Always keep recent backups and test restores before doing destructive actions in production. - Automate backups so a human mistake in a tool like phpMyAdmin doesn't cause data loss. Transcript: Scott Tolinski Let's grab the first one. PHP My Headache. On my very first job, we had a state-of tech stack. PHP backend, MySQL database, and brace yourself, PHP my admin in production. We used it for everything. Browsing tables, running queries, breaking things accidentally, you name it. Oh, the rest of the days of PHP my admin. Wes Bos PHP my admin is like the OG SaaS, you know? Or not the OG SaaS, the OG CMS. Scott Tolinski CMS back end, yeah. Yeah. One fine day, I decided to refresh my local database with a copy from production. Yeah, I know, solid DevOps. I opened PHP MyAdmin and I clicked the big red, dropped that database button, and got the classic warning, are you sure you want to do this? Hell yeah, I'm sure I said confidently, click. Two seconds later and our slack and sentry exploded like a fireworks show of despair. Table does not exist. Query fail. Everything is on fire. My soul left my body. I had been logged into production, not local. I had just deleted the entire production database. Hey, the site's down. Yeah, I said with a forced smile. Working on it. Meanwhile, one of my vacationing co-workers messaged me, probably from a beach somewhere. Hey, what the hell is happening? My phone is blowing up. Nothing to worry about. I lied through my teeth. Miraculously, I had a five-year My dump. Five minutes, not year. Five-year yeah. Major difference. Major difference there. A five-minute old MySQL dump on my laptop. I imported it back into production like my life depended on it, which in sense it did. Within minutes, the site was back online. ([Time 0:03:32](https://share.snipd.com/snip/5889db74-7460-434d-a0d9-cdaed71d5f21)) - **Add Guardrails For Production Tools** - Add visual guardrails and environment separation for production admin tools to avoid accidental operations. - Make production interfaces visibly distinct and restrict dangerous actions behind stricter checks. Transcript: Wes Bos Wild that so many of these stories are people that don't have automated backups in place. They don't have any sort of guardrails where you can just YOLO PHPMyAdmin in production. At the very least, change the background of that production PHPMyAdmin to be like red or something. If you're going to have scary places, have a few guardrails in place. ([Time 0:05:39](https://share.snipd.com/snip/83de053f-3238-429e-93c9-b844f0d9310a)) - **A Semicolon Broke Hundreds Of Sites** - A single stray semicolon saved browsers except older Internet Explorer, which broke for many clients. - The error came from a shared PHP config file that prefixed every site's HTML with a stray character. Transcript: Wes Bos One's called a browser hack. I worked for a small, literal mom and pop shop. Wife was the programmer. Husband was the designer and head of company as my first job out of college and it had its own proprietary CMS e-commerce platform, the class of LAMP stack. I was a jack of all trades doing all the technical stuff from programming the front end design work to database and server administration. We hosted hundreds of our clients' websites on a server we rented. This was before the cloud was big and it was obvious from my comment about the LAMP stack. Yeah, so many of these horror stories come from just shoddy PHP LAMP stack. Not that it's a bad programming language, but just like the around being safe with production was nothing. Everyone's just FTPing and working in production. You're just YOLO winning all. One day, our phone started ringing off the hook with a bunch of customers saying their websites weren't coming up. For us in the office, though, the sites were coming up fine. There was an utter dread for maybe half an hour, which felt like an eternity as we were struggling to figure out what was going on. Eventually, we determined that it was affecting our users below a certain version of Internet Explorer. Remember those days. After investigating further, I eventually realized I had accidentally saved a semicolon at the very beginning of a PHP file that was a configuration shared by every site on the server. As a result, a semicolon at the very beginning of the HTML for every website before the HTML tag, most browsers completely ignored it. Yeah, browsers are super forgiving about mailform markup. But of course, IE threw a fit about it. Fortunately, it ended up being an easy fix. We figured out what was going on. I am forever grateful. I don't get nightmares from having to support Interform anymore. ([Time 0:06:23](https://share.snipd.com/snip/56a03d8e-daf3-4a2f-abc5-00b5502cf8f1)) - **Document Legacy Browser Workarounds** - Keep an archive of legacy browser hacks and conditional workarounds if you must support old browsers during migrations. - Use conditional comments and targeted CSS hacks sparingly, and document why they exist for future maintainers. Transcript: Scott Tolinski Our ass. So I've prepared for you a soul-sucking game where we have eight browser hacks. You all have to tell me, is this a real browser hack, did i make it up the first browser hack the backslash nine trick this targets ie 5.5 and ie 6 specifically by doing a backslash nine at Wes Bos The end of a css property uh do we both get to guess you both have to guess that i'm gonna say that's a legit one there was backslash i don't know if nine was the one for whatever reason riverside Is messed up but can you hear me scott i can cj cannot yeah it's all right i tell me when i should talk yeah i can hear you scott i will tell you when you should talk west says this is a real hack Scott Tolinski Cj what do you say i don't think it's a real hack i say no cj says fake it is fake yes oh but there was one wasn't there there was like a weird ie 5 and 6 one maybe maybe uh js and css you can write Javascript inside of css to extend c capabilities using the expression function. That's fake. Wes says fake. CJ, what say you? Wes Bos I mean, this seems really useful. Scott Tolinski Like, maybe it existed somewhere and then didn't become a standard. I'm going to say real. Real. This is real. No way. What? This is in all versions of internet explorer 5.5 and 4 apparently uh you can this blog post is really interesting apparently it worked really well it was just slow as hell so uh people Didn't use it um real fascinating all right unbelievable cj two for two wes oh for two conditional comments used to deliver html or c to different versions of IE, if LTIE7. Wes Bos This is real. This was the move in HTML5 boilerplate. You had it and it added those conditional classes to the HTML element. Yes. CJ, what say you? I believe it's real. I saw Wes saying a whole bunch of stuff, but yeah, I've used this in the past. It's real. Scott Tolinski It is, of course, real. For those of you who never worked in IE, this magic actually worked. And we used it all the time for various stuff. Even like banners that said, get the hell off this browser. All right. Number four, the ping fix for IE6. Shim a png alpha and ie6 by replacing images using microsoft's filter pro good dx image transform microsoft alpha image loader wes this is very much real um i actually went down a rabbit Wes Bos Hole several years ago to figure out like how did that actually work to me it was magic at the time that it figured out how to just punch out all the alpha pixels of a png um or all the i guess White pixels but it's now it's a now i know how it works it's very real cj what about you what say you okay i didn't hear what west said he said a lot i'm gonna say fake just because the colon In the syntax seems like that'd be hard to parse. It's real. It's real. Whoa. Whoa. Okay. Scott Tolinski Yep. Here's a CSS tricks article for it. PNG hacks from IE6. I think this article is from like 2009 or something. This is a blast from the past for sure. ([Time 0:10:16](https://share.snipd.com/snip/e96695e7-1ef1-4771-888d-3a58cf693c54)) - **Locked Out Of 1Password And NPM** - A maintainer lost access to 1Password and then his NPM account because he lacked recovery codes. - The package ownership was lost permanently due to NPM's recovery policy requiring 2FA or recovery codes. Transcript: Scott Tolinski Ghost password. So I'm a maintainer of a fairly popular NPM package. Nothing huge, but it recently crossed 2 million monthly downloads mark. Nothing huge. 2 million monthly. That's huge. Give yourself a round of applause. Of which I was super proud of. For a long time, I was the only dev behind it until a few months ago. I even wrote in with a potluck question about how to start sharing ownership. I do remember that. Now, vampires never forget. Then, about a month ago, the spooky part started. I'm on my phone, and 1Password suddenly stopped authenticating me. You know, Face ID fails, and it asks for the master password. I type in my master password. Denied. Oh, man. That's the worst. Okay, weird typo. Try it again, denied. Little not forms in my stomach. The next day I try it on my Mac, same thing. Now I'm officially locked out. I'm thinking, okay, don't panic. I'll just contact support, spend a few hours going back and forth with one password support team, but they're firm. There is nothing they can do on their end to help me recover the account. Is that serious, 1Password? Oh, man. And here's the kicker. The part that felt like a line from a horror movie, they told me, sir, we can confirm from our logs that your master password has not been changed in five years. It was like being locked out of your own house with a butler telling me the key that I've used every day for five years is the wrong one. I eventually had to give up. I started the soul-crushing process of resetting every single password for every account I own. Oh, my God. In the end, I managed to reset every single account except for NPM. Due to their security policy, an account cannot be recovered without the 2FA codes or the recovery codes. They tell you to save those recovery codes for a reason. ([Time 0:17:04](https://share.snipd.com/snip/f55acf78-e4e8-4da1-8c30-3c782406b149)) - **Save And Test Recovery Codes** - Store password-manager recovery codes in a secure, accessible place and test account recovery approaches. - Assume stringent account policies; keep multiple recovery options to avoid permanent lockouts. Transcript: Scott Tolinski Wes? Yeah. Do you save your recovery codes? Yes. Like, there's... Man, you should... Wes Bos Especially your 1Password recovery codes or, like, your password manager recovery codes, you Oh my God. Need to store those for in the first place and you need to store them somewhere safe because you are screwed if you lose those so with extreme regret i had to say goodbye to my npm account Scott Tolinski And package i built from the ground up no no open source project of billions of downloads and now opened because of a password i thought i knew i feel like i feel like you're just typing In that password a little bit wrong don't you yeah like try see if caps lock was on maybe yeah or did you try from another phone or computer or something because like i know me if i'm on let's Say a new version of mac os something might not always work the right way i was on like helium browser and my reddit login didn't work for two days but i didn't change the password i just Wes Bos Tried it again in two days and it worked so it makes me wonder if this guy's if something mal spooky is going on you know like the guy's computer has been compromised. Because like everybody right now wants to get a hold of NPM packages. And one with two million. You got an NPM package with two million monthly installs. You are certainly being targeted. So I would probably wipe your computer because something funky is being put in between you and the sign in there. I think that's what's happening. Yes. Yikes. This was very spooky. Send us the name of it because, yeah, that's odd. Yeah. But also, like, props to 1Password for not giving anybody, random people, access to your account. ([Time 0:19:03](https://share.snipd.com/snip/13445854-5d80-4b3e-924c-9e16af830525)) - **Hard-Coded Staging Flag Crashed Payments** - Two junior devs refactored a messy payment-eligibility tree, hard-coded validation for staging, and accidentally deployed it to production. - The release temporarily disabled payments until they flipped a hard-coded Boolean and watched transactions recover. Transcript: Wes Bos When I started web development, I did not know everything about how SQL patches were made. There were a few requests that I worked on, bugs and new features. Worked fine for me. And then I was asked to patch it up to prod. I had new tables and some extra columns on existing tables. So I thought it would patch it all up at once. So I generated a script for every object in the database. Not to miss anything. Yes. Drop create scripts. The boss ran it on the prod DB. And then he asked me why every table was empty. That is when I realized what I did. Fortunately, there was a fresh DB backup and it was fixed immediately. Man, awful. There's so many of these stories where people just YOLO some script, like at least run it, pull your data. If your database is not so big, pull it locally and just run it just to check. Or a lot of like ORMs will a like a test mode where it will tell you how many things it will update yeah what are the dry run or something like that yeah uh very uh very common and and i are you Scott Tolinski One of those people sometimes i see the dry run option and i'm like it's all good i got this yeah Well, I've been there as well. Wes Bos Even just my home assistant the other day, I needed to update it. And I was like, should I back up my config? I was like, that'd be fine. It wasn't fine. I had everything go missing. Luckily, I had a backup from, I don't know, a couple months ago. But always back up before you do anything where you say, it'll be fine. Scott Tolinski Yeah, always do that. Hey, Wes, this next one is a little bit long. I don't know if you've scrolled, but it goes on and on. Wes Bos So you think we should just read this in its entirety? Did you read this one already? Yes, it is a spooky. For the next show, I read them all to make sure they were all good. Is it worth it? Is it worth it? Scott Tolinski Sure. Let's do it. Yep. All right. This is a true story. Wes Bos If it's too long, we'll cut it out. Scott Tolinski Folks, you're getting the live feed here. Yeah. The live, unedited version of this spooky stories. Wes, I didn't hit you with any spooky voice today. Wes Bos Does that make happy? Scott Tolinski Yeah. Wes Bos I can tell you I'm getting really hot in this hammerhead shark costume. My wife's always talking about natural fibers. They're like, don't wear plastic, folks. This is so hot. Scott Tolinski Oh, yeah. For JS Nation, we wore space costumes and they were like, it was like literally wearing plastic bags. I was just absolutely dying in it the whole time okay this is a true story and it happened in 2019 about five months before the pandemic hit it is a catastrophic production failure that I'm 100 guilty keep in mind i was still under six months probation inflicted upon new hire. So this was a new hire, then they're six months, and they have committed a crime. The company I was working for at the time was releasing to a new market. With a new market comes new payment options. Of course, we wanted to allow every single payment method in every single county, right? So we had multiple conditions to build our final list of available payment methods for each particular customer. That check was done in FE. In the FE. What is the FE? Why am I blanking? Wes Bos Finance? Front end? No, front end. No. No. Yeah, of course. Scott Tolinski That check was done in the front end. My domain. While the implementation of the new payment method was done by a colleague in the payments team. I'll call him Steve. He was a newer hire than I was. During the whole thing, no one else could really help us since every single senior there was too busy with some other part of the new market release. It was a whole company effort of the miracle releasing to a new market in a couple of weeks instead of months. It was me and Steve figuring out our way through microservices. We only had each other to rely on, so we pair-coded most of it. Just to add some spice to my life, the requirements kept changing. We were having issues with PayPal and Visa, so for around a little over a month, I'd get different requirements and conditions for which payment method should be available every other Day. Wes Bos Oh, this is a nightmare. It's a nightmare. I hate working with this stuff. Like, people always complain about Stripe and PayPal, and the reason people hate them is because they grow so large that it is impossible to figure all of this stuff out and keep it all Straight as to what's available in which country. Scott Tolinski The conditions were also fairly complicated, depending not only on the customer country, but also how many purchases the customer had done in the past. Whether it was a high-ranking customer, whether or not they had submitted documents before, whether or not the customer was considered validated, since every time a new payment method Was added, the norm had been touch as little as possible. The conditional tree was an absolute mess, with many functions scattered through the utils file, some of which add one line in them to call yet another function. Bro, we have all been there. At some point, me and my partner in crime, Steve, couldn't tell if the new payment method was showing or not under the correct conditions it was held just to figure out what the conditions For the requirement sheets were already added and correct or not. Steve and I decided to refactor the whole thing. Beautiful. Wes Bos Can we stop there for a second? The legacy code that had been tweaked over and over and over to every little edge case possible, it is almost never a good idea to just scrap it and start from scratch unless you have an Extremely well-thought test suite covering all of these, which it doesn't sound like they did. Scott Tolinski Even for like, markup, I had a back in back in the day, I had an old HTML site that had like, you know, 25 files in each directory that was like, index forward slash hyphen, hyphen, new 20, You know, whatever. And you'd update something, you'd be like, why isn't this where I was not using the right one or whatever. And I decided to refactor that and that was a nightmare too. And that was just straight up HTML and CSS. Okay. We looked at each other and say at the same time, screw it. We yanked out the whole component, gutted the so-called conditional tree. We burned the utils file to the ground, and we added tests when there were none. Days of pair programming, blood, sweat, and tears, and all that's left is to confirm in the staging server. I have cobwebs on my mouse pad, and it keeps messing up my scroll. I'm like, going like this every single time. Then I lose my spot because this one's so long. Okay. We hit a couple of simple problems. First, we couldn't properly validate test customers, so we set it to manual. We use a forced response that had to be configured in certain test API. Everyone else was also testing stuff ([Time 0:21:34](https://share.snipd.com/snip/5db2ce50-5ad4-48b4-bcc1-952621392aa4)) - **Use Safe Database Migrations** - Always run schema-change scripts in a safe, repeatable way and verify operations with dry runs on copies. - Use database migrations and backups rather than generative drop-create scripts executed directly in production. Transcript: Wes Bos There were a few requests that I worked on, bugs and new features. Worked fine for me. And then I was asked to patch it up to prod. I had new tables and some extra columns on existing tables. So I thought it would patch it all up at once. So I generated a script for every object in the database. Not to miss anything. Yes. Drop create scripts. The boss ran it on the prod DB. And then he asked me why every table was empty. That is when I realized what I did. Fortunately, there was a fresh DB backup and it was fixed immediately. ([Time 0:21:39](https://share.snipd.com/snip/908ba125-ec8e-47be-aa91-86c4991c6787)) - **Refactor Carefully Without Tests** - Refactoring legacy, fragile conditional logic without exhaustive tests risks introducing critical regressions. - Tests and gradual changes beat wholesale rewrites when business rules are spread across many hidden places. Transcript: Wes Bos Can we stop there for a second? The legacy code that had been tweaked over and over and over to every little edge case possible, it is almost never a good idea to just scrap it and start from scratch unless you have an Extremely well-thought test suite covering all of these, which it doesn't sound like they did. Scott Tolinski Even for like, markup, I had a back in back in the day, I had an old HTML site that had like, you know, 25 files in each directory that was like, index forward slash hyphen, hyphen, new 20, You know, whatever. And you'd update something, you'd be like, why isn't this where I was not using the right one or whatever. And I decided to refactor that and that was a nightmare too. And that was just straight up HTML and CSS. Okay. We looked at each other and say at the same time, screw it. We yanked out the whole component, gutted the so-called conditional tree. We burned the utils file to the ground, and we added tests when there were none. ([Time 0:27:03](https://share.snipd.com/snip/b8c11229-8333-4932-b95d-156a7b3418db)) - **Empty Object Made Discounts Apply Globally** - A discount feature bug caused all products to inherit the first discount because the code checked object existence, not content. - The team deployed the change and discovered many products gained unintended discounts, causing financial loss. Transcript: Wes Bos And then coming up on Wednesday, we have a full hour of them as well. Plenty of them. It's called Discount. You get a discount and you get a discount and everybody gets a discount. I work for a large company that offered multiple subscription-based products. Man, these ones. I like the ones where the database goes away, but I really like the ones where real money is involved. And I really, really like the ones where products go out before they realize it's... That's a problem. My task was to implement a discount feature in the CMS so that a content editor could add discounts to a product. Everything was tested and employed to production without a hitch. Until 12 hours later, someone discovered that each and every product now had a discount. Products that had no discount set up used the first available discount in the list. It had happened to be conveniently sorted from high to low. This was a cause of an API in our CMS that returned a field such as object.discount but my code only checked if object.discount exists, which always was the case. ([Time 0:34:51](https://share.snipd.com/snip/fd058191-5de5-46e5-b09c-504bbaaad70f)) - **Pricing Bugs Scale Fast** - Pricing and coupon errors spread rapidly and expose businesses to huge, immediate losses due to customer behavior. - Public deal trackers and social sharing amplify the impact of accidental discounts. Transcript: Wes Bos It had happened to be conveniently sorted from high to low. This was a cause of an API in our CMS that returned a field such as object.discount but my code only checked if object.discount exists, which always was the case. An empty object is truthy, folks. Although the financial loss was very extensive, my boss was relaxed about it and understood that mistakes happen when i eventually left the company years later i got a postcard with The actual oprah you get a car meme as a goodbye gift ah oh it's sometimes you see these like like pricing errors and people like freak out like um abercrombie had a had a major pricing error Maybe about six months ago and i'm still not sure if it was a pricing error or if it was just like a intentional like marketing campaign but when when those pricing errors go out it's nuts Like once when i when i was in, we found a coupon in Canada. We had this website called Red Flag Deals. And anytime there was a pricing error or a coupon or a good deal, it would go up on that website and people would just destroy it. ([Time 0:35:45](https://share.snipd.com/snip/7cb23b37-e4fb-4e5a-8ada-5dbd0e737902))