You Call It Tech Debt I Call It Malpractice ![rw-book-cover](https://wsrv.nl/?url=https%3A%2F%2Fcdn.changelog.com%2Fstatic%2Fimages%2Fpodcasts%2Fpodcast-original-f16d0363067166f241d080ee2e2d4a28.png&w=100&h=100) ## Metadata - Author: [[The Changelog: Software Development, Open Source]] - Full Title: You Call It Tech Debt I Call It Malpractice - Category: #podcasts - URL: https://share.snipd.com/episode/b3111f69-fdeb-4cfb-b193-0950c827bc96 ## Document Notes ## Highlights - **Family Background in CS** - Kris Brandow's parents both studied computer science, and his mom was a programmer. - She quit after spending an absurd amount of time debugging a missing semicolon. Transcript: Jerod Santo Like your parents or your dad, at least. You have some of that in your background, but you didn't want to do that. You want to be a writer. Kris Brandow Is that right? Yeah, exactly. So both my parents, well, they both studied computer science in college. That's actually where they met. And my mom was actually the first one to start professionally working within the CS field. She was a programmer for a decade, and then my dad got his master's, and then he's been a software engineer ever since. He does a very different type of software engineering from me. He does very low-level stuff, so he's writing a lot of C, a lot of Ada, that sort of stuff. I remember my mom told me she quit programming because she spent some absurd number of hours. I think it was like 16 or 30 hours or something trying to find a problem with the compilation of a program. And it turned out to be a missing semicolon somewhere. And she was like, I'm done. Like, absolutely not. The last straw. Linters or something. Even with that, she wound up going back and doing like not CS stuff, but where she worked for quite a while was a psychiatric hospital and she managed their entire phone system that was Running on DOS for a very long time. So even as she left programming, she was still within the wheelhouse. But as a result of that and growing up around all of that, I was like, I don't want to do this as a career because like no semicolons in my life please every family gathering would be like Oh you're gonna go into computers like your dad and i'm like no i'm gonna be an author that's what i'm gonna do like building computers is fun video games is fun but i want to go be an author Writing is my passion writing is what i love and my dad gave me this wonderful advice when i was getting ready to go to college. He was like, I'm paying for college. So you go to college and you choose a major for something that you love and you will figure it out afterward. My dad has lots of great advice, but that's one of the best piece of advice he ever gave me. So I was like, okay, I'll go. I went, I got my creative writing degree. After a bit of shuffling, wound up with a second major of broadcasting and mass communication and really fell in love with the world of not just writing, but also audio and video. The original plan for me going into college was actually to become an audio engineering major, but that major wasn't really ready at the time. So I kind of did this shuffle and wound up getting what was effectively an audio minor. But yeah, so I got my writing degree and I was kind of building websites and things along the way for a while. And then it really ramped up my end of my junior year going into my senior year because I actually got elected as the basically IT person for the television station that I worked at because We had this student run television station that was like completely student run, like no adults in the room, really. So it was all of us maintaining this multimillion dollar enterprise. It's kind of fun. Yeah. And I was like, we have this like $4 million studio that like, you know, Al Broker was an alumnus of my college. So he had donated a bunch of money with NBC to build us a studio. We have all this nice, shiny digital equipment, but we still managed all of our field equipment checkout with paper forms. And I was like, this is stupid. Like, I don't like this. So I sat down over the summer and I built an equipment reservation system using Drupal. Good old software. There was like a module available that basically did exactly what I wanted. So I went in, I configured it, set the whole thing up. I was like, OK, great. Now we can do equipment reservations online. And my college caught wind of this. And they're like, we've been looking for a system like this for years and we cannot find one that's affordable. And you clearly just built one. Can you build one for us? And I was like, yes. And my advisor was like, for money. I was like, yes, money uh so that was my first paid software engineering gig nice building and configuring the system for is it still in use uh no it is not they used it for a little bit there Was actually a snafu where i wrote a whole user manual but they never gave the user manual to the person that was operating it so he had no idea how the system worked it was just a very miserable Experience and he was like, I like went back and saw him. He's like, this thing is not working. And I'm like, I'm pretty sure I wrote down how to do this in the manual. I was like, manual? What manual? I'm like, oh, God. Jerod Santo And that's when you learn the acronym, RTFM. Kris Brandow Yeah. Yeah. First time. What manual? That's too bad. I built that. I built a whole bunch of other websites. And then I was like, I kind of like this website thing. I like this Drupal thing. I think my options are be a writer, which was fun in college, but professionally is a lot more challenging to do. Or go be a broadcast engineer, which would have been a lot of fun. A bunch of my friends were broadcast engineers. But I was like, this website thing, this website thing fun so i'm going to keep doing this and then that's when i also realized the irony of me spending so much time being like i'm not going To go into computers like my dad and then winding up in computers like my dad ([Time 0:02:42](https://share.snipd.com/snip/1c05404f-615a-489a-a5ff-190d10d1266b)) - **First Software Gig** - Kris Brandow built a Drupal equipment reservation system in college, which led to his first paid software gig. - Ironically, the client never used the user manual, leading to a poor user experience. Transcript: Kris Brandow But yeah, so I got my writing degree and I was kind of building websites and things along the way for a while. And then it really ramped up my end of my junior year going into my senior year because I actually got elected as the basically IT person for the television station that I worked at because We had this student run television station that was like completely student run, like no adults in the room, really. So it was all of us maintaining this multimillion dollar enterprise. It's kind of fun. Yeah. And I was like, we have this like $4 million studio that like, you know, Al Broker was an alumnus of my college. So he had donated a bunch of money with NBC to build us a studio. We have all this nice, shiny digital equipment, but we still managed all of our field equipment checkout with paper forms. And I was like, this is stupid. Like, I don't like this. So I sat down over the summer and I built an equipment reservation system using Drupal. Good old software. There was like a module available that basically did exactly what I wanted. So I went in, I configured it, set the whole thing up. I was like, OK, great. Now we can do equipment reservations online. And my college caught wind of this. And they're like, we've been looking for a system like this for years and we cannot find one that's affordable. And you clearly just built one. Can you build one for us? And I was like, yes. And my advisor was like, for money. I was like, yes, money uh so that was my first paid software engineering gig nice building and configuring the system for is it still in use uh no it is not they used it for a little bit there Was actually a snafu where i wrote a whole user manual but they never gave the user manual to the person that was operating it so he had no idea how the system worked it was just a very miserable Experience and he was like, I like went back and saw him. He's like, this thing is not working. And I'm like, I'm pretty sure I wrote down how to do this in the manual. I was like, manual? What manual? I'm like, oh, God. Jerod Santo And that's when you learn the acronym, RTFM. ([Time 0:04:58](https://share.snipd.com/snip/94d2b7ea-d34d-4fe8-970d-ba08cc30e39b)) - **Waterfall is Iterative** - The waterfall methodology, contrary to popular belief, is actually iterative, as stated on the first page of Royce's paper. - Misinterpretations arose from later papers and a shift in software development culture. Transcript: Jerod Santo I'm kind of pragmatic in that way. But because of that, I don't do things like read papers. I was kind of razzing you because you have this blog post that came out of our Unpopular Opinion segment about the waterfall paper. And I was like, wait, people read these papers. I thought you just write papers and you publish papers and then we reference papers, but do we read them as well? Because Chris does, but I'm looking for the, the CliffsNotes always. Is that, have you always been just like willing to put in the time to just sit there and read like long sessions of reading as well? Or how do you go about acquiring the deep dives? Kris Brandow Yeah, I have this deep desire to understand how things work. And like my brain will not rest until I have like a satisfactory understanding of whatever it is I need to like learn. And I also have this deep appreciation for nuance. So I can't take like simplistic explanation of things and be like, Oh, okay, yeah, that's fine. So like with the waterfall paper, I was specifically one of my friends actually pushed me to go read the whole paper because I for years I've known like waterfall is not how we describe It. It's something that's very different. The paper doesn't really say what it said. But that was actually when I first read the paper it was really just to satisfy your curiosity around like where did this diagram come from what is this diagram and I saw and I saw the other One in the paper I'm like okay that's enough this is an iterative development model and I like went on so it was deep in some other rabbit hole and this happened to be a tangent but I was talking To my friends a couple months ago and I was talking about you know software development and software engineering process and he was like oh yeah yeah you know this thing is like the program Design person in the waterfall paper and I was like program design person I'm like what are you talking about I was like so I like went home after that and I sat down and read through the Paper and my mind was just like oh no no this is one of those things where like everybody has gotten it wrong about what this paper says like yeah all of the summaries all of the stuff like All of the kind of industry knowledge i'm like this paper does not say what people claim it says and that's where it kind of just went from there yeah ([Time 0:10:45](https://share.snipd.com/snip/4c465341-7126-4691-af19-5998d4ca76b3)) - **Rigor in Software Engineering** - Rigor in software engineering is not just about formal documentation but about understanding the problem. - It also includes assessing if the right problem is being solved at all. Transcript: Kris Brandow I mean, there is like this element of rigor, I think being a little bit different than what people think it is. And I think that actually is what has led to a lot of the problems we have is that I think people think that rigor is like, we have to write all of these documents and there have to be in this Very, I guess, in a way, academic-like writing. And we have to be very formal in how we're describing things and how we're designing our systems. And there is a place for that level of very precise documentation, which is like what Royce was getting at. Royce was building spaceships and stuff, rockets to get us a space. For sure. You got to be very... And those places I need. Yeah, you need to document things very, very well because there's very low margins for error. Right. And I think a lot of people are not in that space. So the type of documentation you need isn't as much a – it has to be strictly formal so we make the fewest mistakes possible. It needs to be more of do you actually understand the problem that you're trying to solve? And should you be solving the problem you're trying to solve? And answering those base level questions. And that's kind of fundamentally what the engineering process is about. It's like do we have enough information to actually proceed forward? Which is one of the things that Bennington in his paper highlights is the top-down methodology that they're all used to as engineers isn't about, you know, you write this doc and then You write the next doc and then you write the next doc and then you write the next doc and then you go build the software. It's, okay, we're trying to write this doc. Do we have enough information to actually write it or do we need to go do a prototype and experiment and acquire more information? That's the thing. I think that when we say we need rigor, that's what we need. And I think that's something that you, Jared, do quite often. I think you do find like, you know, information and you use the information to decide, should I do something? Should I not do something? That's like your whole motto of like, keep the main thing, the main thing. Yeah. And I think a lot of software people wind up trying to build stuff without enough knowledge of what it is they're trying to build this leads to the estimation problem that we have where It's like people cannot estimate how long something will take and we say well this is because estimation is so hard it's like estimation is not that hard if you have the proper information But if you're sitting in a room and one person's like this is going to take two days the other person's like this is going to take two weeks you need some way of knowing which person is right Or if they're both right, why they're both right. Or if they're both wrong, why they're both wrong. But no matter what, if you have that type of discrepancy, that's definitely a sign that you need to go do some more exploration, some prototyping, some investigation, something to Acquire more information to pull it out of that realm of just optimistic engineering judgment, because they're probably both wrong. And it's probably gonna take you two months. So yeah, you probably miss a lot of things. And you're thinking, take ([Time 0:19:27](https://share.snipd.com/snip/360f870d-baa5-4b06-856a-66935cee53f2)) - **Tech Debt vs. Malpractice** - What's often called "tech debt" is actually "malpractice" due to irresponsible practices like skipping tests or documentation. - This is not true debt but rather a failure to do one's job properly. Transcript: Jerod Santo What you're saying chris is that they acquired a whole bunch of technical debt i would say i'm pushing his buttons here technical debt is our next topic this is another one so another Thing that chris likes to do as a writer as a graph person, he likes to think deeply about words and their meanings and analogies. Him and I have that in common. We like to bike shed the meaning of words. Is this a good analogy? Is this a good name? A bad name? You took issue with the tech debt analogy, which is why I was pushing your button there. Kris Brandow I think that the tech debt analogy, we should get rid of it, because I don't think the thing that we're talking about when we're talking about tech debt is debt. I think it's more akin to malpractice, and people are being irresponsible. Because I think most of the time when tech debt gets brought up, it's like, oh, we're just going to skip writing the test or skip writing documentation so that we can get this thing out The door faster, or we're just going to code this in a really messy way so it gets out the door faster. And I'm like, that's not debt. That's you not doing your job properly. Please just write the comments and the docs and the tests. It's part of the job. You can't cut out vital things. Or if you do, then you're committing malpractice. And we should call it that. And that's why I think it's going to be unpopular. So it's not tech debt, it's malpractice. Jerod Santo Which I think is a great analogy. And so one of the things I like to do is disagree with people. Adam Stacoviak The tech debt analogy is a good analogy and the non-tech debt, the malpractice analogy is bad. Jerod Santo Well, there's some nuance here. Chris, explain maybe your side of this stance, because I think it's shifted slightly because we've had some slack conversations. Kris Brandow Yeah, I agree with you on one aspect, Jared, where I think tech debt is a good analogy. I think where I draw issue is that it is just an analogy. It's a thing you use to help people understand something when they're not in your contextual domain. The problem is we're using that analogy not with people who don't understand our contextual domain, but with people who do, right? So it's like we use it with each other to try and explain our code bases and their state to each other. And that's where I think it really starts to fall apart because if you're not careful with analogy i think that's something you learn as a writer is you can only stretch an analogy so far Before it just becomes ridiculous and it starts falling apart and i think that's basically where this tech debt analogy is where it's like everybody like people that use it want to use It to explain so much of software so much of like you should make decisions about what choices you make and abstractions you use or how you build things. I was listening back to the tech debt episode that you all did. Jerod Santo Yeah, with John Thornton. Yeah. Kris Brandow And one of the things that stood out to me was this whole like, oh, well, it's tech debt to build this abstraction and only build the front end of the email service and then not build the Back end for it. And I'm like, is that tech debt or is that just planning stages? It's like, oh, well, the first thing we want to build is the front end and we want to get that right. And then the next thing we're going to build is the thing that actually can deliver all of the emails. And that just seems like bog standard planning and approach of a problem that gets you, you know, the biggest bang for your buck along the way. And it's kind of weird to, at least to me, to phrase that as debt, because I think that is a misunderstanding of what debt in the financial sense is, and tech debt derives rather directly From the idea of financial debt. But once again, an analogy of financial debt, not like a debt itself. ([Time 0:30:03](https://share.snipd.com/snip/084f39f2-f198-47be-b9cc-a6e194140327)) - **Document Everything** - Documenting decisions, timelines, and rationale is crucial for future reference and accountability. - This helps avoid repeating past mistakes and builds a knowledge base. Transcript: Kris Brandow It does all go back to like kind of the original point that I had or the main point I have in the blog post of like documentation. Like we don't know how long things are going to take because like, how do you actually acquire an understanding of how long things take? Like, do we sit down and keep all of the documentation for the projects that we've done so we know why we made the decisions we made, what the timelines actually look like, recorded things So that we know, okay, well, in the beginning we said that this was going to take two months. One month in we figured out it's actually going to take four extra months. And then we made these decisions to try and get the timeline down. We did all of this. If you don't write any of that information down, you have nothing to go back to to refer to, which means that the next time you're in that position, all you have is your intuition of what Happened, which will almost always be incorrect. This is why we write things down, because it gives you a tangible thing that you can go back and look at and be like, oh, this is the timeline of events because everybody's going to have A different perspective on why something happened and there's a whole bunch of people that have more knowledge than other people and they kind of need to blend that all together and Get everybody in a room and be like like if a product manager comes to you says well last time you told me that this feature that seems like it's going to be similarly sized would take two Months and it wound up taking six months like i don't like that and if you can sit down and say well that's because we started that project in september we had the lull of december and january And if we look at our attending data we get less work done in the winter months because of vacations and all of that and then we have like the beginning of new year and we have to do all this Extra budgeting stuff and all this planning and that's why this project wound up taking six months but now we're in April and we're planning this project and we're about to hit the boost Of summer. So we know that we'll be able to do this work in a focused way and deliver it within this timeline. And if you can say that to people, they'll be like, oh, OK, no, that makes sense. Yeah, OK, let's go through it that way. But if all we're saying to them is like, well, you know, things happened, then like that's not going to be enough. They're going to be like, OK, it's going to take six months then right because you don't have that information to be like this is why it went this way you're saying it's called malpractice Adam Stacoviak Though right are you advocating that it should not be called tech debt it should be called tech malpractice like what can you label your specific perspective can you document this for Us ch Put this in writing. Well, because I wanted to hear all of you. You had to say before I brought my argument in, and it seems like you – I couldn't quite tell if you had an issue with the analogy of tech debt being akin to financial tech debt or if it was Simply tech debt should be renamed to tech malpractice. You feel like the majority of what we call tech debt is a is a result of a version of malpractice in some shape or form not just simply pragmatism or you know quick analysis cutting corners Because we do that in order to get through something in every whether it's dishes or not like did you rinse them before you put them in dishwasher okay is that malpractice no not really Kris Brandow It's just more like it didn't need rinsing because my dishwasher is powerful enough i think there was a period of time probably a couple decades ago when it was like nah this is just like We haven't learned but by this point like enough of us i'm specifically talking about people that are running projects people that have been in the industry for quite a while and they're Planning projects out at this point you should know better, right? Like it's things like the number of code bases I've walked into that lack not just commenting, but any type of coherent documentation about what this thing does. And the people that are like, this thing is, I don't want to touch it because I feel like I'm going to break it or whatever. And it's like, we built this thing 12 months ago. Like there is no excuse for why something we just built from scratch should be in this state. Adam Stacoviak Yes, I agree with that. Kris Brandow We should have hygiene around our code bases, like not washing our hands as a doctor. And if you get someone sick, and if you cause lots of problems, that's malpractice, right? If you as a lawyer, like don't keep good notes, you don't keep track of things, and you lose your case, like that could be malpractice. And for us, like, if we greenfield a project and in less than a year or less than two years, we wind up back where we were with saying, oh, we need to greenfield it again. That is wholly on us. That's wholly our fault and our problem. And when we talk about it as debt, we kind of get to escape that accountability for ourselves. It's like, oh, well, we just acquired this debt and now we have to pay it off and we don't have to be, we don't have to internalize as we're the ones that acquire this debt irresponsibly And it is our fault and we're not using this tool responsibly. So it's to shift the focus from the tool, which is a perfectly good tool. Like, should you write comments in your code base? I think that there is no concise answer to that. Sometimes you need comments, sometimes you don't, sometimes comments can be harmful. There's a whole bunch of nuance that goes into there. But should we wind up with code bases that we don't understand because there's not a single line of commenting in 15,000 lines of code? No, that's clearly wrong. You've done something wrong. You didn't have a plan of how you were going to document it, of how you were going to make sure that you could maintain it in the long run, continue to add new features to it, onboard new people. And we're at a point in the industry where it's like, this shouldn't be happening anymore, right? It shouldn't be nearly impossible to onboard an engineer onto a project that is less than a year old, or really any project at all. ([Time 0:40:26](https://share.snipd.com/snip/aab9e4a2-534d-4fe7-befe-b06e6e155553)) - **Worthless Code** - Code becomes valuable when it is considered worthless because it allows for easier disposal and refactoring. - Valuing code too highly leads to clinging to it, hindering progress. Transcript: Kris Brandow Compounding of knowledge. Yeah. I love that. Actually, I think there's this interesting thing about, you know, I was thinking about what debt is in the financial sense. And debt is kind of this weird thing that it's like at the core of how money actually works, right? Like how do you create money in an economic system? It's like with debt, right? You give your bank some of your money, they give it out to someone else, and they get more of it back. And that's what bootstraps the entire system and keeps the whole thing running. So it's like debt is foundational to how money works. And I think in that same sense, like knowledge is foundational to how software works. And we should build things around that. And I think there probably is some sort of equivalent to debt for software. I don't know if we should call it debt directly, because I feel like it probably operates in a slightly different way. But I think there is something there that we got to get ourselves toward. And I think that's what I'm trying to do here is like, what is the thing? What are the tools that we have to work with this knowledge? Because I also have another hot take, unpopular opinion. Jerod Santo We got time. Let's hear it. Kris Brandow So I guess I'll preface this with in writing, one of the most annoying things to deal with is like prose. So if you're writing a novel, the words in the novel are very, very expensive because it's very difficult to keep a story coherent and all of that when you've already written a bunch of Words on the page. And the thing that I've realized, pondering this for a long time now, is that the most valuable words, the most valuable prose, is prose that you consider to be worthless. Because if you consider it to be worthless, you have little problem with throwing it away. But as soon as you put value in those words, it becomes quite expensive. Because now you don't want to get rid of them. Now you're falling into something cost fallacy. And there's a whole bunch of sayings around this. But I think the most popular one is kill your darlings, which is basically saying, I know you love that sentence you wrote, but it don't fit anymore. Just get rid of it. I know you like this character, doesn't fit in the story, get rid of it. And I think the same applies to software and specifically to code. I think code is valuable when we consider it worthless. Because when we consider it worthless, we are okay with throwing it away. But as soon as we assess it with value, now we want to try our best to not get rid of it. And where we are now, code is the thought stuff. It's the way we prototype with code. We do designs with code. We We do everything with code. So if every line of code we write, we see as having value, we will try and hang on to every line of code that we write. And it will be painful and difficult for us to delete that code. And I think this causes a whole host of problems. This is why our operating systems still have code in them from the 70s for some reason. And it's why we have all these like, you know, multi-million line code bases that we're like, well, we can't really get rid of this thing because, you know, we have all of this code invested. And so my unpopular opinion there is we should consider code to be worthless and that will derive value from it in the long run, which is really weird. I don't think that's the best phrasing of it. I think I need a better phrasing. But yeah, it's like I think we should believe that code is worthless and we should not be so tightly integrated with it. ([Time 1:07:40](https://share.snipd.com/snip/3d52d24f-6cb6-4dc5-bfbb-028c32839a6b)) - **Resource Explosion** - Modern hardware's power has drastically increased, offering resources equivalent to 100 times the world's total money. - This shift necessitates rethinking basic software development principles. Transcript: Kris Brandow Because I think that we don't understand the scale of how much things have changed in the last 40 or so years. So I've been working on this. It started with me trying to do it based on like square footage, but I figured out that doing it with money probably resonates with people a lot better. So if you take the Apple II that was released in 1977, and it costs about $1,295, which is about $6,500 in today's money. And you kind of do a nice little conversion with transistors and clock cycles. So you're like, okay, one transistor running at 50 kilohertz equals $1 per year, let's say. The equivalent resources of an Apple II were $60,000 in this conversion of transistor's clock speed. And so you could be like, okay, $60,000 a year, that's your budget for what you want to get done, what you can get done in a single clock cycle with the Apple II. If you take a Mac Studio that was released this year, and it cost roughly the same, right? If you spec it out with the Mac processor, Mac RAM, you wind up with a roughly $6,500 machine. The equivalent amount of money in that same transistor slash clock speed is $9 quadrillion, which is 100 times the amount of money that exists in the world. So we went from having the resources of decent middle class income to all of the money in the world 100 times over. And I don't think we as an industry understand that that's how far hardware has moved in the last 46 years. And I think the problem is that we basically just have unlimited money now. And you can't use the same things that you use when you were budget constrained, when you have unlimited money, right? You can use an Excel spreadsheet to manage $60,000 a year. It might be a little tough, but you can do it. You cannot use an Excel spreadsheet to manage $9 quadrillion. You'd have to actually build something very, very customized and very advanced. I mean, you're talking about managing world level economies at that point or even large government at that point. So the fact that so much of what we have today is rooted in the time when the resources were that constrained, I think shows that we need to really start giving up things that we've had in The past and really start rethinking the very basics of how we do things like i think i very much i deeply appreciate unix but i think it's a model that is kind of too stuck in the past because We're doing a lot of the hardware that we are using is spent trying to emulate that old world trying to pretend as if we're still a single processor running at maybe a megahertz or two megahertz So that, you know, going and fetching RAM isn't that problematic. But we truly live in a ridiculously different world than even 20 years ago, right? There's another analogy, I guess, comparison I've been working on of roughly 20 years ago, we got our first commercial multi-core processor uh it was until xeon well i think there was A consumer version but the the server version was until xeon had two cores four threads two of them in a machine you get a total of what that's like 16 threads oh no it would be total eight Threads because it's two cores four threads each so you have eight threads and about a megabyte of cache or two megabytes of cache amd released a processor this year that has 96 cores, 192 threads per socket. So if you put two of them in, you want to put 384 cores. And each of those processors has a total of a gigabyte of cache. So you have two gigs of cache. That is an extremely different computing device than the one we had back in 2005. And while we could stretch a lot of the parallelism uh you know we could just utilize four cores or eight cores or whatever you can't use the same thing go routines baby yeah you can't utilize The same thing to go you know you can't use those same resources to go uh use 384 cores and multiple gigs of ram or multiple gigs of cash because it's also cash too right before it's like You had a little tiny bit of cache so it couldn't really keep a lot in there yeah now you have a lot of cache and you know if you write your software wrong you're going to be doing less of cache Expiration and your program's going to crawl to a halt but you want to be able to use all those extra cores because the over that 20 year span clock speeds have not increased at all in fact That AMD processor we're talking about has a slower clock speed than that original Intel Xeon. ([Time 1:12:00](https://share.snipd.com/snip/6aad1c5d-a8fc-46c1-b435-6a3aa08b0118)) - **Everything is a Distributed System** - All computing devices, including individual processors and disk operations, are distributed systems. - This realization changes how we approach software development and debugging. Transcript: Kris Brandow Not almost. Not almost, Jared. Not almost. It is a distributed system. Jerod Santo So are you here to announce your brand new operating system? I don't understand. What are you going to work on in this space, Chris? New OS? Throw Unix out and just start from Greenfield? Kris Brandow So I don't think, obviously, saying we should throw Unix and all the Unix operating systems out is extremely impractical. Yes. But the main thing I've been frustrated with is that I've been a distributed systems engineer for most of my career. I love distributed systems problems. But even I was puzzled when I first started looking into the literature around distributed systems and really reading it. And there were some things that I found very confusing. Like I have this book that I bought called Fault Tolerant Message Passing Distributed Systems. And I'm like, okay, this is an interesting title. Why is message passing in there? Because aren't all distributed systems message passing ones? And I learned, no, there's shared memory distributed systems. I'm like, oh, okay. I'm like, okay, well, what's fault there? Don't all distributed systems need to be fault tolerant? And it's like, no, there's distributed systems that aren't fault tolerant. So I went on this journey over the past couple of years of learning a lot about distributed systems. And what I found is that all of our computing devices are distributed systems. And not just like in the sense of like, you have multiple cores, but like, you know, you have PCIe. PCIe is a network is a packet switched network system. And that's at the core of literally all of our computers. So yeah, you have a little distributed system that's, you know, running your computer right now, running the computer. We're all having this, you know, recording this episode on, which has some profound implications. Even your processor, once again, is its own distributed system because you have all this cache coherency that you need to deal with, right? In C, if I wrote five to the variable X and then I read it the next clock cycle, I expect that to be five because I just wrote five. But if I got scheduled to a different core that has a different value in this cache, then I might not get five. And that's going to break my code. So there's a lot of hardware logic that's done to actually synchronize all of that cache coherency that we have. And we call it atomicity when we're talking about, you know, the processor. We're talking about atomic memory and all of that and transactional memory, perhaps. But in distributed systems, we call it linearizability, which is also the kind of the gold standard for a lot of databases that they want to reach. And it's like, that's at the core of every processor that we make that has multiple cores, right? So it's like, it's not just that these are technically distributed systems when you look at them. It's like, it's using the same basis of research that we use to go build Kubernetes to go make your processor work, Which means that there isn't a difference between when you're programming Locally and when you're programming this kind of Kubernetes or what have you. And the reason I have frustration about this is because I've been to too many places where I say, okay, well, we've got to go solve this problem. And I hear, well, we want to avoid having distributed systems problems. Or we don't want to take on distributed systems problems right now. And that annoys me because I'm just like, you can't not. It is not just like the basic types of distributed systems either. One of the things I've learned about is Byzantine fault tolerance systems, which Leslie Lamport, I guess you had previously wrote. He coined that term of like the Byzantine generals problem and, you know, Byzantine consensus. I don't know if we coined Byzantine consensus, but the whole Byzantine thing is his creation, his and some other people. And I was like, when am I ever going to need this? And like, for one thing, blockchain is actually a Byzantine fault-tolerant distributed system, which is actually pretty cool. But also I realized through thinking about this, that writing to disk is a Byzantine faulty distributed system because the disk will lie to you about what it's done. You might say, please write this to disk and then do an F-sync and be like, no, no, no, really write this to disk. And it'd be like, yep, totally. I wrote it to disk. And then you read it. It's like, I don't know what you're talking about. I didn't. You didn't ask me to write that to disk, it lies to you. So even something as simple as writing to disk is a distributed systems problem. Buy your hard drives, man. Yeah, you got to be like, these lying hard drives, we need to get better ones. Jerod Santo Right? Ask him better. Ask him again. So everything's a distributed system is what you're saying. Kris Brandow And the thing here is that I think a lot of it is not difficult to understand if we just sat down and taught people. But we're still very focused on maintaining the illusion of the old world that we refuse to move into this new world. And so that's what I'm trying to do is be like, how can I teach people about distributed systems so that it's actually approachable? And you're not going and reading academic textbooks and academic papers trying to understand what even for someone like me that's been doing distributed systems for 10 years was just Like, this is difficult to parse. ([Time 1:16:40](https://share.snipd.com/snip/bde2ebfa-967c-4e08-8484-d8a3c07648b8))