Code Is a Liability  ## Document Notes ## Highlights - **Code Is A Liability Not An Asset** - Code's capabilities are assets, but code itself is a liability that accrues costs over time. - The goal is code whose capabilities generate more revenue than maintenance and operational costs. Transcript: Cory Doctorow Code is a liability, not an asset. Tech bosses don't understand this. They think AI is great because it produces 10,000 times more code than a programmer. But that just means it's producing 10,000 times more liabilities. AI is the asbestos we are shoveling into the walls of our high-tech society. Code is a liability. Code's capabilities are assets. The goal of a tech shop is to have code whose capabilities generate more revenue than the costs associated with keeping that code running. For a long time, firms have nurtured a false belief that code costs less to run over time. After an initial shakedown period in which the bugs in the code are found and addressed, code ceases to need meaningful maintenance. After all, code is a machine without moving parts. It does not wear out. It doesn't even wear down. This is the thesis of ([Time 0:05:54](https://share.snipd.com/snip/453e3a50-4310-4a0f-8709-ac58a9f6808b)) - **Software Engineering Versus Coding** - Writing code is distinct from software engineering, which focuses on long-term operation and failure modes. - Software engineering designs for legibility, adaptability, and graceful failure across changing contexts. Transcript: Cory Doctorow To understand why code is a liability, you have to understand the difference between writing code and software engineering. Writing code is an incredibly useful, fun, and engrossing pastime. It involves breaking down complex tasks into discrete steps that are so precisely described that a computer can reliably perform them, and optimizing that performance by finding Clever ways of minimizing the demands the code puts on the computer's resources, such as RAM and processor cycles. Meanwhile, software engineering is a discipline that subsumes writing code, but with a focus on the long-term operations of the system the code is part of. Software engineering concerns itself with the upstream processes that generate the data the system receives. It concerns itself with the downstream processes that the system emits processed information to. It concerns itself with the adjacent systems that are receiving data from the same upstream processes and or emitting data to the same downstream processes the system is emitting To. Writing code is about making code that runs well. Software engineering is about making code that fails well. ([Time 0:07:22](https://share.snipd.com/snip/f42ab7c3-6313-4d44-b37b-b3f1375c0d54)) - **Location Defaults Caused Real Harm** - Location-services defaults caused dangerous misreports, like placing unknown IPs at the center of the continental USA. - These defaults once sent armed strangers to innocents and caused hours of distress for Doctorow's family. Transcript: Cory Doctorow Take the code that devices use to report on their physical location. Originally, this was used for things like billing, determining which carrier or provider's network you were using and whether you were roaming. Then, our mobile devices used this code to help determine your location in order to give you turn-by directions in navigation apps. Then, this code was repurposed again to help us find lost devices. This, in turn, became a way to locate stolen devices, a use case that sharply diverges from finding lost devices in important ways. For example, when locating a lost device, you don't have to contend with the possibility that a malicious actor has disabled the find my lost device facility. These additional uses, upstream, downstream, and adjacent, expose bugs in the original code that never surfaced in the earlier applications. For example, all location services must have some kind of default behavior in the very common event that they're not really sure where they are. Maybe they have a general fix. For example, they know which cellular mass they're connected to, or they know where they were the last time they got an accurate location fix. Or maybe they're totally lost. It turns out that in many cases, location apps draw a circle around all the places they could be, and then set their location to the middle of that circle. That's fine if the circle is only a few feet in diameter, or if the app quickly replaces this approximation with a precise location. But what if the location is miles and miles across, and the location fix never improves? What if the location for any IP address without a defined location is given as the center of the continental USA, and any app that doesn't know where it is reports that it is in a house in Kansas, sending dozens of furious, occasionally armed strangers to that house, insisting that the owners are in possession of their stolen phones and tablets. You don't have to just fix this bug once. You have to fix it over and over again in Georgia, in Texas, and in my town of Burbank, where Google's location-sharing service once told us that our then 11-year daughter, whose phone We couldn't reach, was 12 miles away on a freeway ramp in an unincorporated area of LA County. She was in a nearby park, but out of range, and the app estimated her location as the center of the region it had last fixed her in. ([Time 0:10:26](https://share.snipd.com/snip/7517c5f3-7a52-4652-83ea-a302e9ae0072)) - **Legacy Systems Breed Fragility** - The longer software runs, the more technical debt and cruft accumulate, making replacement costly and risky. - Large legacy systems become fragile, patched over with incompatible layers that invite failures and attacks. Transcript: Cory Doctorow The longer that code sits there, the more superannuated its original behaviors become, and the more baroque, crufty, and obfuscated the patches layered atop it become. Code is not an asset, it's a liability. The longer a computer system has been running, the more technical debt it represents. The more important the system is, the harder it is to bring down and completely redo. Instead, new layers of code are slathered atop it, and wherever these layers of codes meet, there are fissures in which these systems behave in ways that don't exactly match up. Worse still, when two companies are merged, their seamed, fissured IT systems are smashed together so that now there are adjacent sources of tech debt as well as upstream and downstream Cracks. That's why giant companies are so susceptible to ransomware attacks. They're full of incompatible systems that have been coaxed into a facsimile of compatibility with various forms of digital silly putty, string, and bailing wire. They are not watertight and they cannot be made watertight. Even if they're not taken down by hackers, they sometimes just fall over and can't be stood ([Time 0:13:17](https://share.snipd.com/snip/0043a3dd-c3e1-4cc2-bb93-3ca1e26e937f)) - **Airlines Locked Into Ancient Code** - Airlines computerized early and remain locked to ancient systems, which makes their apps and operations brittle. - British Airways and Southwest suffered outages tied to legacy compatibility and unsustainable tech debt. Transcript: Cory Doctorow Airlines are especially bad because they computerized early and can't ever shut down the old computers to replace them with new ones. This is why their apps are such dog shit and why it's so awful that they fired their customer service personnel and require flyers to use the apps for everything, even though the apps Do not work. These apps won't ever work. The reason the British Airways app displays an unknown error has occurred 40 to 80 percent of the time isn't just that they fired all their IT staff and outsourced to low bidders overseas. It's that, sure, but also that BA's first computers ran on electromechanical valves, and everything since has to be backwards compatible with a system that one of Alan Turing's protégés Gnawed out of a whole log with his very own front teeth. ([Time 0:14:37](https://share.snipd.com/snip/71d2d9ea-ce30-4c71-8dcc-6a3c41a6bd43)) - **AI Produces Code, Not Engineering** - AI can write code but cannot perform software engineering because it lacks wide, anticipatory context. - Mass-producing code with AI multiplies liabilities when maintainability and failure modes aren't engineered in. Transcript: Cory Doctorow AI can write code, but AI can't do software engineering. Software engineering is all about thinking through context. What will come before this system? What will come after it? What will sit alongside of it? How will the world change? Software engineering requires a very wide context window, the thing that AI does not and cannot have. AI has a very narrow and shallow context window, and linear expansions to AI's context window require geometric expansions in the amount of computational resources the AI consumes. Writing code that works without consideration of how it will fail is a recipe for catastrophe. It is a way to create technical debt at scale. It is shoveling asbestos into the walls of our technological society. Bosses do not know that code is a liability, not an asset. That's why they won't shut the fuck up about chatbots that shit out 10,000 times more code than any human programmer. They think they found a machine that produces assets at 10,000 times the rate of a human programmer. They haven't. They found a machine that produces liability at 10,000 times the rate of any human programmer. ([Time 0:16:06](https://share.snipd.com/snip/38e5724c-df8a-4198-b24f-98714f9a9844)) - **Always Validate AI-Generated Code** - Avoid treating AI-generated code as finished product; validate it with human software-engineering judgment. - Cultivate fingerspitzengefühl and process knowledge to anticipate novel failure modes. Transcript: Cory Doctorow Writing code that works without consideration of how it will fail is a recipe for catastrophe. It is a way to create technical debt at scale. It is shoveling asbestos into the walls of our technological society. Bosses do not know that code is a liability, not an asset. That's why they won't shut the fuck up about chatbots that shit out 10,000 times more code than any human programmer. They think they found a machine that produces assets at 10,000 times the rate of a human programmer. They haven't. They found a machine that produces liability at 10,000 times the rate of any human programmer. Maintainability isn't just a matter of hard-won experience teaching you where the pitfalls lie. It also requires the cultivation of fingerspitzengefühl, that fingertip feeling that lets you make reasonable guesses about where never-before pitfalls might emerge. It's a form of process knowledge. It is ineluctable. It is not latent, even in the largest corpus of code that you could use as training data. ([Time 0:16:44](https://share.snipd.com/snip/dce58cd4-d470-4a3d-a6de-60c762efc3fe)) - **Use AI For Isolated Utility Tasks** - AI helps engineers in isolated, one-off utility projects where upstream/downstream interactions are minimal. - But scaling AI to produce enormous volumes of code creates unreviewable technical debt at scale. Transcript: Cory Doctorow Now, some of you hearing this have heard software engineers extolling the incredible value of using a chatbot to write code for them. Some of you are software engineers who have found chatbots incredibly useful in writing code for you. This is a common AI paradox. Why do some people who use AI find it really helpful while others loathe it? Is it that the people who don't like AI are, quote, bad at AI? Is it that the AI fans are lazy and don't care about the quality of their work? There's doubtless some of both going on, but even if you teach everyone to be an AI expert and call everyone who doesn't take pride in their work out of the sample, the paradox would still Remain. The true solution to the AI paradox lays in automation theory and the concept of centaurs and reverse centaurs. In automation theory, a centaur is a person who is assisted by a machine. A reverse centaur is someone who has been conscripted into assisting a machine. If you're a software engineer who uses AI to write routine code that you have the time and experience to validate, deploying your fingerspitzengefühl and process knowledge to ensure That it's fit for purpose, it's easy to see why you might find using AI when you choose to, in ways you choose to, at a pace you choose to go at, to be useful. ([Time 0:20:14](https://share.snipd.com/snip/a9d1133a-3cc2-4e85-afe7-286bfb1aada5)) - **Centaurs Versus Reverse Centaurs** - Automation theory distinguishes centaurs (people assisted by machines) from reverse centaurs (people assisting machines). - Forcing engineers to validate massive AI output turns them into reverse centaurs and worsens work quality and risk. Transcript: Cory Doctorow The true solution to the AI paradox lays in automation theory and the concept of centaurs and reverse centaurs. In automation theory, a centaur is a person who is assisted by a machine. A reverse centaur is someone who has been conscripted into assisting a machine. If you're a software engineer who uses AI to write routine code that you have the time and experience to validate, deploying your fingerspitzengefühl and process knowledge to ensure That it's fit for purpose, it's easy to see why you might find using AI when you choose to, in ways you choose to, at a pace you choose to go at, to be useful. But if you're a software engineer who's been ordered to produce code at 10x or 100x or 10,000x your previous rate, and the only way to do that is via AI, and there is no human way that you Could possibly review that code and ensure that it will not break on first contact with the world, you'll hate it. You'll hate it even more if you've been turned into the AI's accountability sync, personally on the hook for the AI's mistakes. ([Time 0:20:55](https://share.snipd.com/snip/d45b6f50-3d45-4637-a9e0-3d1033c81b66)) - **Plan For Digital Asbestos Removal** - Expect large-scale work cleaning up AI-produced code; plan for 'digital asbestos removal' jobs and costs. - Invest in long-term refactoring and cautious deployment rather than mass-producing quick code with AI. Transcript: Cory Doctorow For every blacksmith put out of work by the automobile, there's a job waiting as an auto mechanic. In the years since the AI bubble began inflating, we've heard lots of versions of this. AI would create jobs for prompt engineers, or even create jobs that we can't imagine, because they won't exist until AI has changed the world beyond recognition. I wouldn't bank on getting work in a fanciful trade that literally can't be imagined because our consciousnesses haven't been so altered by AI that they've acquired the capacity to Conceptualize of these new modes of work. But if you are looking for a job that AI will definitely create by the millions, I have a suggestion. Digital asbestos removal. For if AI code, written at 10,000 times the speed of any human coder, designed to work well but not to fail gracefully, is the digital asbestos we're filling our walls with, then our descendants Will spend generations digging that asbestos out of the walls. There will be plenty of work fixing the things that we broke thanks to the most dangerous AI psychosis of all, the hallucinatory belief that writing code is the same thing as software Engineering. ([Time 0:23:15](https://share.snipd.com/snip/be18dea9-0a92-40e0-9211-e3f0b81a3b36))