Things Fall Apart - Architecture to Avoid Progressive Collapse  🔗 **Source:** [Things Fall Apart - Architecture to Avoid Progressive Collapse](https://www.youtube.com/watch?v=nkNvVNfp1F0) > [!note]+ Document Notes > tags: [progressive-collapse, sam-newman, leaddev, resilience-engineering, distributed-systems, cascading-failure, multi-region, bulkheads, redundancy-paradox, race-conditions, incident-reports, civil-engineering-analogies, complexity-cost, root-cause-myth] > author: Sam Newman > source: youtube.com > reference: https://www.youtube.com/watch?v=nkNvVNfp1F0 > date created: 2026-09-03 09:05 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the speaker. > > Do not quote or attribute any other line to the named author. > > > [!warning] Four things to know before trusting this note > > **1. Speaker identity comes from outside the capture.** The transcript never names the speaker and Reader recorded the author as "YouTube." The LeadDev talk page for this title lists Sam Newman (author of *Building Microservices*) as the speaker at LDX3 London 2026; the channel is LeadDev. Uploaded 2026-09-02, 25:25, 415 views at capture. The talk itself was given earlier (it references "October of last year" for the AWS outage and London tube strikes on the day). > > **2. Auto-captions, not a transcript.** 51 caption paragraphs, no slides captured. Known mishearings: "Ronan Point" appears as "Roman Point," "Rhoden Point," "road point," "Ronin Point" · "Miss Ivy Hodgej" = Ivy Hodge · "Constantina effect" = concertina effect · "DynamoB" = DynamoDB · "enacctors" = enactors · "multi-reion" = multi-region · "multicow" = multi-cloud · "tool chest talks" = toolbox talks · "standin" = stand-in · "US East1" = us-east-1 · "eight sleep" = Eight Sleep · "the void" = The VOID (Verica Open Incident Database). Quotes are left as captured. > > **3. Numbers are as spoken.** Four deaths at Ronan Point, "close to a thousand" large-panel-system buildings still standing in the UK, Monzo's "about 3,000" versus "about 13" services, the 216-page NIST report, and the AWS outage mechanism are all stated from the stage without on-screen citation. The speaker says he fed the NIST PDF to NotebookLM rather than reading it. None of these were independently checked for this note. > > **4. Commercial context.** The speaker opens with a "blatant plug" for his forthcoming book (this research came from writing it) and jokes about not offending sponsors in the "solutions track upstairs" before saying anyone who calls two-way data sync easy "is trying to sell you something." No product is endorsed. > > # 🏷️ Title: Things Fall Apart - Architecture to avoid progressive collapse > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - Source: https://www.youtube.com/watch?v=nkNvVNfp1F0 (LeadDev channel, uploaded 2026-09-02, 25:25) - Talk page: https://leaddev.com/technical-direction/things-fall-apart-architecture-to-avoid-progressive-collapse (names Sam Newman as speaker) - Named in the talk, not linked in the capture: NIST report on progressive collapse (216 pages) · AWS's own write-up of the October 2025 us-east-1 outage · AWS disaster-recovery options document (source of the four-tier diagram) · The VOID incident database · a "fantastic video" on Ronan Point · slides on the speaker's website. The capture contains zero outbound links. ## 📝 Summary **What this is.** A 25-minute LeadDev keynote that takes one concept from structural engineering, progressive collapse, and uses it to read the October 2025 AWS us-east-1 outage. The speaker pairs two incidents throughout: the 1968 Ronan Point tower collapse in Canning Town, London (a small gas explosion on the 18th floor took out load-bearing walls and a corner of the building fell in a concertina; four people died), and the AWS outage, in which a race condition between DNS "enactors" inside DynamoDB deleted Route 53 entries and cascaded through every AWS service built on DynamoDB and every customer built on those. **The definition.** A progressive collapse is "something where an initial small failure, a small inciting incident, results in a wider collapse in the overall system" (3:02). The domino video is the intuitive version; the point of the talk is that real collapses aren't linear and aren't easy to trace. **The three mitigations, lifted from the NIST report and mapped to software.** - *Reduce the hazards.* Ronan Point: consider banning gas in high-rises; at minimum, improve ventilation. AWS: turn off the DNS automation until the fix is understood; add tests that would have caught it. "These are quite simple things" (10:30). - *Strengthen the components.* Ronan Point: the large panel system's joints were supposed to be packed with concrete and were packed with newspaper and cigarette packets; the fix was quality control, building regulations, toolbox talks, and training the workforce. Software: redundancy, multiple instances behind a load balancer. Then the twist: AWS already had redundant DNS enactors across availability zones, and that redundancy is what produced the race condition. "Increasing a system's complexity to handle more failure scenarios can actually introduce more sources of failure" (13:31). - *Reduce the interconnection.* Ronan Point: alternative load paths, now standard in tall buildings. Software: bulkheads (with the Titanic's too-short bulkheads as the cautionary case), local fallbacks for smart devices, and the multi-region ladder from backup-and-restore through pilot light and warm standby to active-active. Each rung costs more and adds complexity; "the decision about this stuff ends up being a balance between cash and complexity" (17:30). Companies that stayed single-region "did the sums," they weren't idiots. **The two sharpest points.** Multi-cloud hedges the failure of a whole vendor, not a region, and it doubles your skills bill. And if your own code is the inciting element, a second region or a second vendor running the same code fails the same way. Monzo's stand-in is the rare counterexample: roughly 13 services on GCP with no shared code, versus about 3,000 on AWS. **The close.** "That idea of a root cause itself, of course, is a massive lie because there never is one root cause" (23:31). Mitigation has to be multifaceted. And the reason he could give the talk at all is that AWS wrote up what happened, so write up yours. ## 📖 Main Content **Structure.** The talk is a two-strand braid. Strand one is a physical disaster with a well-documented investigation: Ronan Point, 1968, opened two months earlier, an 18th-floor tenant lights the hob for tea at about 5:45 in the morning, a faulty nut on a new cooker connection has leaked gas, the explosion blows out load-bearing side walls, the floors above fall through. Strand two is the AWS outage that many in the room lived through. Each of the three mitigation categories is applied to both strands in turn, which gives the talk a rhythm a listener can predict by the third repetition. That predictability is a feature for a keynote; it's also why the talk feels shorter than 25 minutes. **The AWS mechanism as told.** DynamoDB, "ostensibly a database service," also owned updating DNS routes for internal AWS services (4:30). Inside it, a DNS planner generates plans; multiple DNS enactors pick them up and apply them in Route 53. One enactor ran slow. Newer plans queued; another enactor picked one up and finished fast. Then the slow enactor completed and, in its cleanup phase, "overwrote and deleted all the other DNS entries that were already executed" (7:31). Internal services lost their routes, services built on those services failed, customers built on those failed. The speaker calls it "basically a race condition" and stops there; he doesn't discuss why the cleanup didn't check plan freshness, which AWS's own post-incident summary covers and which is the design question an engineer would ask next. **Why the consumer-device list matters.** The list (alarm clocks, smart locks stranding Airbnb guests, Eight Sleep mattresses stuck hot or cold, self-cleaning litter trays) is played for laughs, and it earns them. It's also the talk's only demonstration of *distance* in a cascade: none of those product teams were AWS customers in any direct sense. They were customers of customers. The domino analogy fails exactly here, because the litter tray couldn't see the domino behind it. **The redundancy paradox is the load-bearing idea.** Every other point in the talk is standard resilience material. This one isn't, or at least isn't standard in keynotes. The speaker's own recommendation for strengthening a service (run several copies, distribute load) is the thing that produced the race condition at AWS. He generalizes: sometimes "the answer is to not make the system more complicated because it might open you up to other types of issues" (14:01), with the "all I wanted was three copies and now I've got a Kubernetes cluster" line as the applause moment. He does not offer a rule for when redundancy is net-negative, which is the honest position; nobody has one. **The multi-region ladder, with costs attached.** Backup and restore: cheap, up to a day of data loss, slow recovery. Pilot light: incremental replication, no provisioned compute, minutes-to-hours recovery if reserve capacity exists. Warm standby: some services already running, critical workloads served immediately. Active-active: "the holy grail," and the place where two-way data synchronization lives, which "if anybody tells you is simple or easy, they are trying to sell you something" (17:00). The diagram is borrowed from AWS's own DR document, credited on stage. What the speaker adds is the refusal to shame single-region companies: Zoom knew. They did the sums. **Multi-cloud gets the same treatment.** It doubles the skills requirement, it hedges a whole-vendor failure rather than a regional one, and it does nothing about the case where your code is the inciting element. The Monzo stand-in system is offered as the rare design that takes the same-code problem seriously: a stripped-down core-banking replacement, no shared code, different vendor, an order of magnitude fewer services. The speaker flags his numbers as approximate and invites correction. **Ronan Point's aftermath as a governance case.** The building system wasn't necessarily the flaw; the implementation was. Prefabricated panels were meant to be joined with steel ties and packed with concrete; investigators found newspaper and cigarette packets. The remedies were institutional: regulation, quality control, morning toolbox talks ("effectively standups," 12:30), and training a workforce the system had been designed to let stay unskilled. The speaker doesn't draw the software parallel explicitly, but it's sitting there: a system designed so that low-skill operators can assemble it fast is a system that will be assembled with whatever's to hand. **What the talk skips.** No discussion of blast radius design at AWS (cell-based architecture, static stability), which is AWS's own preferred framing for exactly this. No mention of chaos engineering or game days as a way to find hidden interconnections before an incident does. No treatment of organizational cascade: the talk stays at the architecture layer. For a 25-minute keynote these are reasonable omissions; for a reader using the note they're the obvious follow-ups. ## 🗒️ Notes - **Two case studies, one frame, three categories: the talk's structure is its argument.** The NIST taxonomy (reduce hazards, strengthen components, reduce interconnection) does the analytical work; the speaker's contribution is picking two incidents legible to a software audience and refusing to let either become a morality tale. - **The redundancy paradox is stated as a paradox, not resolved.** "Almost an unsolvable conundrum in resilience engineering" (13:31). That's the correct claim. The talk would be weaker if it pretended to a rule. - **"They did the sums" is a defense of engineers against internet hindsight.** The speaker names the companies that stayed single-region and says the people behind them "are not dumb" (18:02). That's a rare stance in a genre that usually shames. - **Same code, same bug, any region.** The most transferable sentence for WordPress: replicating a system with the inciting defect in it replicates the collapse. Monzo's no-shared-code stand-in is the only design in the talk that addresses this. - **The root-cause line is the thesis.** "There never is one root cause. There's always another software bug" (23:31). This is why the mitigations have to be layered rather than sequential. - **The build-quality story is a labor story.** A system designed for low-skilled assembly, joints packed with rubbish, fixed by training and regulation. The speaker says "take the low-skilled people and give them some skills" (12:30). The framing is slightly condescending toward the workers and squarely accurate about the incentives. - **Sourcing is thin on stage but pointed off stage.** The speaker names his sources (NIST, AWS's write-up, The VOID, AWS's DR doc, a Ronan Point video, his own slides) but the capture has no links. He also volunteers that NotebookLM read the NIST report for him, which is candor about method that most speakers would omit. - **The closing ask is the DevRel point.** "Next time something goes wrong, create the opportunity to learn from that failure and share what you've learned" (25:01). The whole talk was only possible because AWS published a post-incident summary. ## 📚 References - Sam Newman, *Things Fall Apart - Architecture to avoid progressive collapse*, LDX3 London 2026 keynote, LeadDev channel, YouTube, uploaded 2026-09-02, 25:25. - Talk listing: leaddev.com, "Things fall apart – Architecture to avoid progressive collapse" (technical direction track). - Named on stage: NIST progressive-collapse report (216 pp.) · AWS post-incident write-up of the October 2025 us-east-1 outage · AWS disaster-recovery options document (four-tier diagram) · The VOID incident database (graphical map of related incident reports) · Monzo stand-in system · Ronan Point investigation history. - Cross-references in Reader: the Core Security Initiative note (2026-09-02) for the same "interconnection is the attack surface" logic applied to plugins; the Melapress malware note (2026-09-02) for an incident where the fix addressed the hazard and not the interconnection, and the files came back. ## 🗣️ Paraphrase In 1968 a small gas explosion in a London tower block took out a corner of the building because the load had only one path down. In October 2025 a race condition between two DNS workers inside AWS deleted the routes every other AWS service depended on, and the failure travelled outward until people's alarm clocks didn't ring and their cat litter trays stopped cleaning themselves. Both are progressive collapses: a small inciting incident, a wide failure. Civil engineers have a taxonomy for stopping them and it maps cleanly onto software. Reduce the hazards, which is necessary and never enough. Strengthen the components, and notice that your favorite strengthening move, redundancy, is what caused the AWS race in the first place, because complexity you add to handle failures is also complexity that can fail. Reduce the interconnection with bulkheads, local fallbacks, and multi-region, keeping in mind that every rung of the multi-region ladder costs more money and more complexity, that the companies who stayed single-region did the arithmetic, and that a second region or second cloud running the same code fails the same way. Monzo's no-shared-code stand-in is the exception that shows how rare that thinking is. There's no single root cause, so there's no single fix. And the only reason anyone can learn from any of this is that someone wrote it up. ## 📰 Newsletter **Sam Newman's progressive-collapse keynote is the best 25 minutes on the AWS outage you'll get without reading the NIST report.** He pairs the 1968 Ronan Point tower collapse (gas explosion, load-bearing walls gone, one load path, four dead) with October 2025's us-east-1 outage (DNS enactor race inside DynamoDB, Route 53 entries deleted, everything downstream dark, including the smart litter trays) and runs both through NIST's three mitigation categories. Reduce hazards: AWS turned off the DNS automation and added tests; London considered banning gas in high-rises. Strengthen components: Ronan Point's panel joints were packed with newspaper instead of concrete, fixed by regulation and training; in software the move is redundancy, except AWS's redundant enactors *were* the race condition, so "increasing a system's complexity to handle more failure scenarios can actually introduce more sources of failure." Reduce interconnection: bulkheads, local fallbacks, the multi-region ladder from backup-and-restore to active-active, each rung trading cash for complexity. Two lines to keep. "If anybody tells you two-way data synchronization is simple or easy, they are trying to sell you something." And: same code in a second region or second cloud fails the same way, which is why Monzo runs a 13-service stand-in on GCP with no shared code. Root cause "is a massive lie." Write up your incidents; this talk exists because AWS wrote up theirs. ## ✅ Action Items **DevRel** - The talk is a model for the "borrow a taxonomy from another field, apply it to two incidents your audience lived through" keynote shape. Steal the structure for any resilience or security talk: physical case, digital case, three categories, repeat. `#action/h3` - The closing ask (publish your incident write-ups) is the community-facing version of the whole talk. Any post that cites this should end the same way. `#action/h4` **OSPO** - Not applicable to this source. **Community** - The VOID incident database is named as the place to study cascades. Worth a look for WordPress-ecosystem incident reports, and worth asking whether WordPress hosting incidents ever land there. `#action/h4` **WordPress teams** (Core, Core Performance, Core AI, Core Program Team, Mobile, Playground, Community, Training, Marketing, Support, Design, Documentation, Photos, TV, Polyglots, Accessibility, Meta, CLI, Hosting, Tide, Openverse, Themes, Plugins) - **Hosting:** the multi-region ladder with honest cost and complexity per rung is the conversation hosts have with agencies after every regional outage. A hosting-team guidance page that names the four tiers and says "single-region is a defensible choice when you've done the sums" would be more useful than another uptime promise. `#action/h3` - **Core:** the redundancy paradox applies to any core mechanism that runs redundantly across instances (scheduled events, background updates, object-cache invalidation). Where two workers can act on the same plan, the AWS cleanup race is the failure to test for. `#action/h3` - **Core Security:** "same code, same bug, any region" is the plugin-ecosystem argument in one line: a vulnerable plugin on ten thousand sites is one inciting element with ten thousand load paths. Pairs with the Core Security Initiative's framing of AI-driven report volume. `#action/h3` - **Plugins:** auto-update is a DNS-enactor pattern (a planner decides, enactors apply). A slow enactor overwriting a fast one is worth a test case in the update pipeline. `#action/h4` - **Meta:** WordPress.org's own infrastructure dependency map is the "which dominoes are in the middle" question for the project. Not asserted here to be single-region; asked. `#action/h4` - **Documentation:** the Ronan Point aftermath (regulation, quality control, toolbox talks, training) is the case for documentation as a strengthening measure, not an afterthought. `#action/h4` - All other teams: not applicable to this source. **Personal (Courtney)** - You've got a wp-release-followup skill and a site-health audit loop. Neither asks "what's the interconnection?" Add one question to the audit: which single dependency, if it vanished, takes the most down? `#action/h3` - The talk structure is directly reusable for a DevRel talk in your next role: one physical failure everyone knows, one digital failure the audience lived, one borrowed taxonomy. `#action/h4` - The Melapress note from 2026-09-02 (hack that looked fixed until the files came back) is a reduce-the-hazard-only remediation. Read it against this taxonomy; it's the same shape at a smaller scale. `#action/h4` ## 🧪 Hypotheses - **H-A. Redundancy that introduces coordination introduces races.** Prediction: any redundancy design where N workers can act on the same plan without a freshness check will eventually produce a stale-overwrite incident. Test: audit one WordPress mechanism (cron, updates) for the pattern. - **H-B. Single-region companies mostly made an explicit decision.** The speaker asserts Zoom and peers "did the sums." Test: look for public post-incident statements from October 2025 that describe the cost-benefit; absence of any would weaken the claim. - **H-C. Same-code replication does not reduce collapse risk when code is the inciting element.** Prediction: multi-region incidents caused by software bugs will show simultaneous regional failure. Test: search The VOID for multi-region incidents tagged as code defects. - **H-D. Published incident reports increase cross-industry learning more than internal ones.** The talk itself is evidence, but one talk isn't a measure. Test: count citations of AWS's October 2025 write-up in conference talks within a year. ### 📊 Methodology A case-comparison keynote: two incidents, one borrowed taxonomy from a 216-page NIST report the speaker says he queried through NotebookLM rather than read. Evidence is narrative and authoritative (NIST, AWS's own write-up, Monzo's public architecture) rather than measured. No original data. The speaker flags approximate numbers and invites correction. Sources are named aloud but not linked in the capture. Appropriate rigor for a keynote; a reader should verify the AWS mechanism against AWS's summary before repeating it. ### ❓ Questions - Why didn't the slow enactor's cleanup check whether its plan was still the newest? Is the answer "it did, and the check also raced"? - What's the speaker's rule, if any, for when added redundancy is net-negative? He says "sometimes"; a working engineer needs "when." - Do any WordPress hosts run a Monzo-style stand-in (different code, different vendor) for the control plane, or does everyone run the same stack twice? - Where does the WordPress ecosystem's own incident learning get published, and is any of it in The VOID? - Is the "close to a thousand" large-panel buildings figure current, and how many have been remediated since the 2017 Grenfell-era reviews? (Not in the talk; a natural follow-up.) - The book: title, publisher, and whether the progressive-collapse chapter is an expansion of this talk. ### 🔮 Future Research - AWS's cell-based architecture and static-stability posts as the vendor's own answer to interconnection, compared with the speaker's three categories. - A WordPress dependency map: which shared services (w.org API, update servers, CDN, translation delivery) are the middle dominoes for the majority of sites. - Case studies of stand-in systems beyond Monzo: who else runs a no-shared-code fallback, and what it costs. - The labor angle of Ronan Point: the parallel between "low-skilled assembly systems" and low-code/AI-generated site ecosystems assembled by operators who can't inspect the joints. - The October 2025 outage's second-order consumer-device failures as a dataset for "distance from the incident" in cascades. ### 💡 Implications - **For resilience talks and docs:** the three-category taxonomy is a better outline than "best practices" lists because it forces the paradox (strengthening adds complexity) into view. - **For hosting decisions:** the honest framing is cash versus complexity per rung. Anyone selling active-active as simple is selling. - **For WordPress governance:** same-code-same-bug means a widely installed plugin is one inciting element with many load paths; the mitigation is fewer shared paths or bulkheads at the site level, not more copies. - **For incident culture:** publishing the write-up is what makes the industry stronger; the talk is a direct beneficiary of AWS doing so. ## 🧠 Learning Styles Assessment **Visual** — moderately served on the day (building photos, floor plan, domino video, Titanic bulkhead diagram, the AWS DR ladder), poorly served in this capture: audio-only reconstruction, no slides. The speaker says slides are on his website. **Auditory** — well served; a practiced keynote voice with jokes placed to release tension after the deaths are mentioned. **Read/write** — the NIST report and the AWS write-up are the read/write path; this note is the summary layer. **Kinesthetic** — not served; no exercise. The implied one is "map your own system's middle dominoes." **Sequential** — strongly served; the two-strand, three-category structure is announced up front and followed exactly. **Global** — served by the opening definition and the closing "root cause is a lie." **Overall:** a story-driven talk that works without slides, which is why it survives as audio. Weakest for anyone who needs the mechanism in detail. ## 📊 Accessibility Assessment Assessment of the video as an artifact and of what the talk says. Not a WCAG audit. #### Alternative text and non-text content YouTube auto-captions only; the speaker's British delivery and proper nouns produce a dozen consistent mishearings (listed in the warning). No transcript or slides in the capture; slides exist on the speaker's site. #### Keyboard navigation Standard YouTube player; not evaluated further. #### Contrast and readability Not assessable from audio. The floor plan and building photos described on stage would need alt text for any written version. #### Text resizing and reflow Not applicable to the video. Applicable to a transcript, which doesn't exist here. #### Media and captions Auto-captions present; accuracy on names is poor. Clicker failures on stage (twice) mean slide changes and speech drift apart for a listener following along. #### Forms and interactions Not applicable. #### Standards reference No accessibility standard is discussed. The talk's own subject touches disability indirectly: the smart-lock and smart-mattress failures land hardest on people who depend on those devices, a point the talk plays for laughs rather than making. ## 🌍 DEIB Assessment #### Cultural breadth Anglo-centric by design: a London incident, a London venue, tea-making jokes, tube strikes. The AWS material is global. The Monzo example is UK fintech. Nothing outside the UK-US axis. #### Inclusive language "Low-skilled people" is used four times for the Ronan Point workforce, and "these people are idiots" is quoted as the internet's verdict on single-region companies so the speaker can reject it. The first is period-accurate industry language and slightly condescending; the second is handled well. #### Neurodiversity support Clear announced structure, repetition, and humor placed predictably. Good for listeners who need signposting. Auto-captions undercut it. #### Representation One speaker, male, well known in the field. Ivy Hodge, the tenant whose kitchen exploded, is named and given agency ("she put a pan of water on"). Workers who packed the joints with newspaper are unnamed and unheard, which is how the historical record left them. #### Psychological safety The talk defends engineers against hindsight ("not dumb," "did the sums") and frames failure as material for shared learning. That's the psychologically safe stance for a room of engineering leaders, and it's earned by the content rather than asserted. ## ✅ Logical Validity **Claims and their support.** - *Progressive collapse is a useful frame for distributed-systems failure* — argued by two worked cases. Persuasive; a frame, not a falsifiable claim. - *The AWS outage began as a DNS enactor race in DynamoDB* — consistent with AWS's published summary as generally reported; stated from memory on stage without citation. Verify before repeating. - *Redundancy caused the race* — follows from the mechanism as told. Valid. - *Complexity added for resilience can add failure modes* — supported by the case and by the field's literature. Valid, and correctly left as a tension rather than a rule. - *Single-region companies made a considered trade-off* — asserted ("do you think Zoom didn't know?"). Plausible, unsupported by any company's statement in the talk. - *Multi-cloud hedges vendor failure, not regional failure, and doubles skills cost* — the first half is definitional; the "double" is a rhetorical estimate. - *Same code fails the same way in any region* — valid for code-caused incidents, which the speaker scopes correctly. - *Monzo runs ~13 services on GCP with no shared code* — public in Monzo's own engineering writing; numbers flagged as approximate. - *"There never is one root cause"* — a stance in the resilience-engineering literature (Cook, Dekker), presented as obvious. Defensible; not argued here. **Fallacies.** None structural. The consumer-device list is comic relief that also does argumentative work (distance in cascades) and the speaker doesn't overclaim from it. The Ronan Point deaths are handled with acknowledged awkwardness rather than exploited. **Internal consistency.** Consistent. The one apparent tension (recommend redundancy, then show it causing the incident) is the talk's point, and the speaker names it as such. **Verdict.** Valid as a framing talk. The mechanism claims need one check against AWS's write-up; the numbers are flagged approximate by the speaker himself. ## 🛡️ DARVO Analysis Not applicable to this source. There is no accusation to reverse. The nearest structural moment is the speaker rebutting the internet's "these people are idiots" charge against single-region companies (15:00, 18:02); that's a defense of a third party against hindsight, with reasons given, not a reversal of victim and offender. AWS is neither attacked nor excused; the talk credits them for publishing the write-up and describes the fix. ## 🎯 Conclusion The talk's value is a taxonomy and a paradox. The taxonomy (reduce hazards, strengthen components, reduce interconnection) comes from structural engineering by way of a NIST report and maps onto distributed systems without strain. The paradox is that the second category's favorite move, redundancy, is what produced the AWS race condition, so added resilience is also added failure surface and there's no rule for when the trade flips. Around those two ideas the speaker arranges a defense of engineers who made single-region decisions with their eyes open, a warning that a second region or second cloud running the same code fails identically, and the Monzo stand-in as proof that taking that warning seriously is rare. For WordPress the transferable sentence is "same code, same bug, any region": a plugin on ten thousand sites is one inciting element with ten thousand load paths, and the mitigation is bulkheads and fewer shared paths, not more copies. The closing ask is the one worth acting on today: write up the incident, because this talk only exists because AWS wrote up theirs. Speaker identity, numbers, and the AWS mechanism are all as spoken; check them before citing. ## 🔄 Refactor Appendix — Things Fall Apart - Architecture to avoid progressive collapse ### 📚 Literature Atoms (H3) #### [H1] A progressive collapse is a small inciting failure that produces a system-wide one > [!quote] ID:rw_01m1kmx0ezzgnc08tzhkqjszm1 > "a progressive collapse is something where an initial small failure a small inciting incident results in a wider collapse in the overall system" (3:02) The definition the whole talk hangs on; borrowed from the built environment. Tags: `#atom/literature/h1` #### [H2] The AWS outage was a cleanup race between redundant DNS enactors > [!quote] ID:rw_01m1kmx0ezzgnc08tzhkqjszm1 > "the earlier plan completed and in its cleanup phase overwrote and deleted all the other DNS entries that were already executed. This was this is kind of a race condition." (7:31) The mechanism as told from the stage; verify against AWS's write-up before repeating. Tags: `#atom/literature/h2` #### [H3] Complexity added for resilience adds failure modes > [!quote] ID:rw_01m1kmx0ezzgnc08tzhkqjszm1 > "increasing a systems complexity to handle more failure scenarios can actually introduce more sources of failure" (13:31) The redundancy paradox, stated as "almost an unsolvable conundrum" rather than resolved. Tags: `#atom/literature/h3` #### [H4] Two-way data sync is never simple; whoever says so is selling > [!quote] ID:rw_01m1kmx0ezzgnc08tzhkqjszm1 > "if anybody tells you that two-way data synchronization is simple or easy, they are trying to sell you something." (17:00) The active-active rung of the multi-region ladder, priced honestly. Tags: `#atom/literature/h4` #### [H5] Single-region companies made a trade-off, not a mistake > [!quote] ID:rw_01m1kmx0ezzgnc08tzhkqjszm1 > "The people behind these companies are not dumb. They're not idiots, right?" (17:30) A defense of engineers against hindsight; cash versus complexity is a decision. Tags: `#atom/literature/h5` #### [H6] There is never one root cause > [!quote] ID:rw_01m1kmx0ezzgnc08tzhkqjszm1 > "that idea of a root cause itself, of course, is a massive lie because there never is one root cause. There's always another software bug." (23:31) Why mitigation has to be layered across all three categories. Tags: `#atom/literature/h6` ### 🧠 Permanent Candidates (H4) #### [P1] Redundancy that requires coordination is a new failure surface, not just a safeguard Multiple workers acting on shared plans need freshness checks or they will eventually race; AWS's redundant DNS enactors produced the very outage redundancy was meant to prevent. The trade has no general rule for when it flips, which is why "add more copies" needs a coordination audit attached. *Candidate; one strong case plus the field's literature.* Links: [[Resilience Engineering MOC]] · [[Distributed Systems]] · [[WordPress Hosting Security Stack MOC]] Tags: `#atom/zettel/z1` #### [P2] Replicating a system replicates its inciting defect A second region or second vendor running the same code fails the same way when code is the cause; only a no-shared-code stand-in (Monzo) breaks the path. For WordPress, a plugin on ten thousand sites is one inciting element with ten thousand load paths. *Candidate; the mechanism is clear and the Core Security Initiative note carries the ecosystem version.* Links: [[Vendor AI Agents Need Bounded Scope Before Launch]] · [[WordPress Plugin Governance MOC]] · [[Resilience Engineering MOC]] Tags: `#atom/zettel/z2` #### [P3] Fixing the hazard without changing the interconnection leaves the collapse path intact Stopping gas explosions is necessary and insufficient when the building has one load path; removing malware is necessary and insufficient when the upload vector remains. The Melapress "files came back" case is the small-scale instance. *Candidate; two sources from two days, same shape.* Links: [[WordPress Hosting Security Stack MOC]] · [[Incident Response]] Tags: `#atom/zettel/z3` ### ✏️ Fleeting Queue - "Cash versus complexity" as the honest axis for the multi-region ladder. `#atom/fleeting/f1` - Toolbox talks as standups: the 1968 building trade invented the daily sync. `#atom/fleeting/f1` - NotebookLM reading the 216-page NIST report: a speaker admitting AI-assisted source reading from a keynote stage. `#atom/fleeting/f1` - The smart litter tray as the unit of cascade distance. `#atom/fleeting/f1` - Which WordPress shared services are the middle dominoes? `#atom/fleeting/f1` ### → Things 3 (max 2 actions) 1. **Add one question to the weekly site-health audit:** which single dependency, if it vanished, takes the most down? `#action/h1` 2. **Check the AWS October 2025 summary** for the enactor-cleanup mechanism before citing H2 anywhere. `#action/h1` <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Things Fall Apart - Architecture to Avoid Progressive Collapse.md|Things Fall Apart - Architecture to Avoid Progressive Collapse by Sam Newman]] ## Highlights - collapse is something where an initial small failure a small inciting incident results in a wider collapse in the overall system and kind of we've seen ([View Highlight](https://read.readwise.io/read/01m1kna3vfaybbwfhfz16pm778)) - Note: H1 — The definition the whole talk hangs on: a progressive collapse is a small inciting failure that produces a system-wide one. Borrowed from structural engineering (Ronan Point, 1968). #atom/literature/h1 - Tags: atom/literature/h1 - earlier plan completed and in its cleanup phase overwrote and deleted all the other DNS entries that were already executed. This was this is kind of a race condition. This is a example of a ([View Highlight](https://read.readwise.io/read/01m1kna7ycktf7v3sc8sgyjykq)) - Note: H2 — The AWS us-east-1 mechanism as told from the stage: a slow DNS enactor's cleanup deleted entries a fast one had already applied. Verify against AWS's own write-up before repeating. #atom/literature/h2 - Tags: atom/literature/h2 - increasing a systems complexity to handle more failure scenarios can actually introduce more sources of failure. ([View Highlight](https://read.readwise.io/read/01m1knabh4k8vhwbw9zr6kr9xt)) - Note: H3 — The redundancy paradox: the redundant DNS enactors that were meant to survive an AZ failure are what produced the race. Stated as "almost an unsolvable conundrum," not resolved. #atom/literature/h3 - Tags: atom/literature/h3 - anybody tells you that two-way data synchronization is simple or easy, they are trying to sell you something. ([View Highlight](https://read.readwise.io/read/01m1knaemqzmz6p1y2smnv2mr7)) - Note: H4 — Active-active is the top rung of the multi-region ladder and the one where two-way sync lives. Every rung trades cash for complexity; single-region companies "did the sums." #atom/literature/h4 - Tags: atom/literature/h4 - fact, that idea of a root cause itself, of course, is a massive lie because there never is one root cause. There's always another software bug. So, our ([View Highlight](https://read.readwise.io/read/01m1knahxzsz081dg4ra25hhem)) - Note: H6 — The thesis: no single root cause, so mitigation must be layered across all three categories (reduce hazards, strengthen components, reduce interconnection). #atom/literature/h6 - Tags: atom/literature/h6