Building for Reliability  🔗 **Source:** [Building for Reliability](https://www.youtube.com/watch?v=ugZjcHoLQQo) > [!note]+ Document Notes > --- > tags: [monzo, leaddev, site-reliability, emergent-failure, anomaly-detection, deploy-frequency, cost-of-quality, technical-debt, postmortem-culture, kubernetes, behavioral-metrics, auto-remediation, non-deterministic-systems, fintech-engineering] > author: LeadDev > source: youtube.com > reference: https://www.youtube.com/watch?v=ugZjcHoLQQo > date created: 2026-09-03 08:20 > --- > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the speaker. > > Do not quote or attribute any other line to the named speaker. > > [!warning] Read this before trusting the note > > **Source shape.** A 22-minute conference talk (22:25) on the LeadDev YouTube channel, published 2026-09-02, 311 views at capture. Speaker introduces himself as "Mate," a software engineer at Monzo; the captions never give a surname, a job title beyond "software engineer," or the conference edition, so none is asserted here. 2,668 caption words across 39 paragraphs. > > **ASR artifacts.** Auto-captions mishear "LeadDev" as "like Dev," "Monzo" once as "Monsa," "org" as "orc," "incident response runbooks" as "instant response runbooks," "out-of-order delivery" as "out of delivery." Quotes below are left as captured. > > **Evidence quality.** Every number (15 million customers, 3,000 microservices, 300 deploys a day, 99.996% availability, 100 ms → 1% completion drop, 0.4% acceptance-rate drop) is the speaker's own account of internal Monzo data. No slides, dashboards, or postmortems are visible in the transcript; nothing is independently checkable. Treat the figures as practitioner testimony from a regulated bank, not as published measurements. > > **No sponsor.** No product pitch; the speaker's employer is the case study. > # 🏷️ Title: Building for reliability > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ### 🔗 References > - Source: https://www.youtube.com/watch?v=ugZjcHoLQQo (LeadDev channel, https://www.youtube.com/@LeadDev) > - No outbound links in the transcript. Named entities: Monzo, Kubernetes, the UK and Ireland markets. > ## 📝 Summary > **What this is.** A Monzo engineer's talk arguing that reliability is an emergent property of a whole system plus the organization running it, not a feature you add to components. The talk is built on three incidents of increasing size and then draws six operating rules from them. > **The three incidents.** > - *100 milliseconds.* An A/B-tested change to currency conversion passed every metric. An adjacent screen, "move money," lost 1% of completions. The cause was an unintended 100 ms added to that screen's load time. Nothing errored; customers "voted with their attention." > - *Unhealthy nodes.* A few Kubernetes nodes going unhealthy under load slowed the app for a small subset. Monzo added a monitor-and-replace loop: mark the node unhealthy, later swap it for a fresh one. > - *The big one.* A routine rolling upgrade restarted nodes one by one. Each was marked unhealthy during restart and came back fine, but the unhealthy marks stayed. The auto-recovery loop then replaced large parts of the cluster at once, took out core services, and forced a failover to a stand-in system. The speaker's summary: "an upgrade process performed flawlessly, interacted with automated monitoring and recovery mechanisms," and did something "locally reasonable" that was "globally harmful." > **The six rules.** > 1. You can't test your way out of production failure; assume partial failure, timeouts, retry storms, and out-of-order delivery as the default and prefer idempotent workflows and compensating actions. > 2. The first design choice is how fast you notice, so measure everything, including behavioral metrics like customer intent. The 100 ms failure surfaced as a behavior change, not an error. > 3. High-impact incidents are usually slow movers living in the tails of distributions; anomaly detection on subgroups caught a 0.4% drop in card acceptance that population-level dashboards would never show. > 4. Ship fast because it's safer: about 300 deploys a day, each small enough to reason about and roll back. "The goal is to break in a detectable, reversible manner." > 5. Technical debt is a tool with three rules: know you're taking it, cost it (isolated and reversible vs. trunk and hard to reverse), and give every item an owner and a payback plan. Adversarial domains like fraud justify speed over polish. > 6. Balance three buckets (proactive reliability, reacting to production, new features) using a "cost of quality" metric whose proactive and reactive components have a minimum; over-alerting pushes reactive cost back up. > **Culture clauses.** Zero risk appetite collapses learning; treating proactive work as a tax leaves response unpracticed; treating rollbacks as failures stops people rolling back; blameless, mandatory, tooled, broadcast postmortems are "one of the most important things you have to get right." > **The closing turn.** More code shipped than ever, more of it by people who don't read or write code, and "a new class of non-deterministic component" (AI) inside the system. The speaker says none of the lessons change; they intensify. > ## 📖 Main Content > **Why a bank talks about reliability as trust.** The speaker frames trust three ways: competence, integrity, reliability. Reliability is the one where the product shows up when needed, and for a phone-only bank every failure is felt in the app. That framing makes the 100 ms story land: a "functioning" system that feels slower is a reliability failure by his definition, and the customer response (abandoning a transfer) is the measurement. > **The composition argument.** The middle incident is the talk's real content. Every component involved was well-tested: the upgrade tooling, the health monitor, the node replacer. The failure is in the interaction: a health mark that should have cleared on restart didn't, and an automation designed to protect reliability read that stale state as an instruction to replace most of the fleet. His conclusion is stated as a principle, "reliability is not something that you add to individual components. It is the composition of systems and tools and operators and third parties that must be reliable together." The word "operators" is doing work there: he includes humans and third parties in the system boundary. > **Measurement before prevention.** "Our first choice is not whether we allow a failure to happen or not. Our first choice is actually how fast we notice." That ordering is the operational thesis. He gives two supporting mechanisms: behavioral metrics (customer intent) catch what error rates miss, and anomaly detection on subgroups catches slow drift in the tails. The 0.4% card-acceptance example is the size of signal he says matters: invisible in aggregate, real for the affected subgroup. > **Speed as a safety property.** Three reasons, in his order: faster feedback, failure is inevitable so delay only delays learning, and small changes are easier to reason about and undo. He contrasts 300 daily deploys with bundling "hundreds and thousands" of changes into a monthly release that is "much harder to undo." The prerequisites he names are culture, an articulated risk appetite, and heavy investment in automation and low-friction tooling. > **Technical debt with rules.** The rules are crisp enough to be adopted verbatim: if you don't know you're taking it, "it's not technical debt, it's just bad engineering"; cost it by blast radius and reversibility; every item has an ow Hello everyone. I hope you're having, uh,fun and enjoying the conference. It's agreat conference, like Dev. Uh, one ofthe reasons is that you get to hear fromvery many smart people talking about thecool things that they've built. Not thistalk, though. This is just going to be metalking about things going wrong, for themost part. My name is Mate. I'm asoftware engineer, and I have the honorand the privilege of serving a greatteam at Monzo. Monzo is a bank that lives on your phone,and our mission is to make money workfor everyone. We're extremely proud tobe serving over 15 million customers, inthe UK today, and a growing number inIreland as well. You could say thatwe're in the business of safeguardingour customers' money. And you know,people have to trust you before they letyou safeguard their money. And there's afew different ways that we can thinkabout trust. One is through the lens ofcompetence. I trust my bank because theyProvide the best possible products andservices for my needs. There's integrity. I trust my bank because theywill have my interests at heart and willhelp me achieve my financial goals. Andthere's reliability. I trust my bankbecause they consistently show up when Ineed them, whether that's to check mybalance, make a payment, or save moneyfor the future. And reliable userexperiences are predictable. The sameaction in the app leads to the samepredictable outcome. They're fast,smooth, and instant. Feels safe. Clunkyand slow feels riskier. And when a reliable experience breaks, Iwant my bank to be able to communicateclearly and transparently and recoverfast. And that's reliability.And for us at Monzo, reliability isinseparablefrom customer trust. We are primarily amobile banking experience. So everythinghappens right there in the app. It'sinstant. So even small failures inReliability can have a disproportionateimpact on trust. And to illustrate that,let's look at a small failure to beginwith. So we have this feature in the appthat you can use to convert your moneyinto different currencies, and a point, uh,it was last year we were shipping fairlymajor change to how this part of thesystem works, and as with most changes, weran an A/B test, which is we made thischange available to a subset of ourcustomer base, and the rest received nochange. So we're able to detect andobserve differences between the twopopulations. And sure enough, this particular featuretested really well. All metrics areeither neutral or they’re positive.Ready to roll.Well, turns out there’s this adjacentscreen that wasn’t supposed to bereceiving any change. It’s called themove money screen, and you can use it tomove money between your Monsa accounts,or, for example, to send pocket money toYour children. And when we looked atthat, we realized that we had a 1% dropin completion rates on that screen. Now,this is a big change, and it wascompletely unexpected. So, what we did was we went broad. Wecanvased all the metrics that we collectand compared them. And what we realizedwas that we actually inadvertentlyintroduced an additional 100milliseconds into the loading time ofthat screen. And that's what caused theproblem. Now, this was a very subtlechange, and nothing bad really happenedhere, right? The system was functioning,but that does not mean that it wasreliable. For a small proportion of ourcustomers, the system felt a little bitslower, and therefore a little bit lessreliable, and they voted with theirattention, and they chose not to completetheir transactions with Monzo on thatday. So, reliability really matters.At Monzo, we choose to build and operateour own stack. It's a modern distributed. Systems. Um, we run over 3,000microservices on top of Kubernetes. Andwe deploy about 300 times on any givenday. And in the past financial year, ouravailability was 99.996%.This means that, roughly across the year,we had about 20 minutes of downtimeacross our important, uh, businessservices. So, this is a relativelycomplex system. It's made out of manyvery well-engineered and very well-tested components. But when you putthose well-tested components intotogether, into production, sometimesunexpected behavior emerges.And to illustrate that, let’s look at aslightly larger failure. And it allstarts with us at some point observingthat we had a small number of nodes onour cluster entering an unhealthy state. It’s not a big deal. This is actuallyexpected. You still got lots of healthynodes able to serve customer traffic.But when the system, in that state, getsput under more load, then for most ofThe customer base, there's nodifference. But for a small proportionof our customer base, the system wasstill functioning, but it was slowerthan it should have been; it was a failure inreliability. So we decided to go and fixthat. And we introduced essentially amonitoring and auto-recovery loop. So wehave, uh, a special agent, a special component monitoring the health of ournodes. And when a node gets intotrouble, it is marked as unhealthy. Andat some point later, another componentswoops in and just replaces that nodewith a fresh, new healthy one, bringingsystem health back up to perfect. And weproceed. And it was this investment thatwas made with reliability in mind thatwas actually a very big contributingfactor to a major platform incident thatwe had earlier this year. Let's talkthrough what happened. It started with a routine upgrade tosome of the software components on ourKubernetes cluster. And we did as we do always, we did arolling restart of the upgradedcomponents. So, one by one, the nodes wentdown for restart.One by one, the nodesgot marked as unhealthy during restart,and one by one, the nodes came backhappily, and we had a perfectlyfunctioning upgraded cluster. Unfortunately, the nodes were stillmarked as unhealthy.So, sure enough, ashort time later, auto recovery kickedin and started replacing large parts ofour Kubernetes estate.And that caused aunavailability of core system services. And to mitigate customer impact, weactually had to fail over to ourstand-in system while we recovered theprimary and switched back. So, in thiscase, what happened wasan upgrade process performed flawlessly,interacted with automated monitoring andrecovery mechanisms, and while the systemwas doing something that was locallyreasonable, it was globally harmful. Point that reliability is not somethingthat you add to individual components.It is the composition of systems andtools and operators and third partiesthat must be reliable together. Soreliability emerges, or it doesn'tthrough how all of these componentsinteract. And this emergent property hasvery important implications forengineering organizations. One, you can't test your way out ofeverything. No matter what you do, stuffwill happen in production.Two, you have to make measurement a corepart of your system, or you will beflying blind where you have the mostneed not to. And your organization hasto be set up with a capability andreadiness to respond, and to learn fromthose failures. Because you see,reliability is not actually a componentlevel feature. It is rather anorganizational ability that shows up andshines in production service to users. And it’s not just a technical concept. Either. It has an important socialcomponent because ownership andincentives will shape the outcomes asmuch as your system architecture.So, if we are to build reliable systems,we must then assume and build for thedefault case of partial failure,timeouts, retry storms, out of delivery,all of the above, right? We prefer itempotent workflows and compensatingactions over focusing on happy pathcorrectness. If that's our assumption, then our firstchoice is not whether we allow a failureto happen or not. Our first choice isactually how fast we notice. So, we haveto measure everything. Importantly, wehave to identify and measure behavioralmetrics, customer intent. Remember the100-millisecond example from before?That failure was not signaled by anerror state. It was signaled through anunexpected change in customer behavior.So, we have to measure that. And look, inour experience, a lot of high impactIncidents are not sudden large spikes. They're slow movers.Latency creeps up. Model correctnessdrifts.There's a slow decrease in successfulcustomer outcomes that doesn't have anobvious explanation. And the detectionand the diagnosis of those errors liesin the tails of the distributions. Andthat is squarely where our attentionneeds to be, not in the middle. And itturns out anomaly detection is a superuseful tool to detect slow burningincidents. Our motivating example hereis a condition that we had that wascausing a very small decrease in ourcard transaction acceptance rates, likeup to 0.4%.This is not a change that you notice ifyou look across the entire population ofcustomers. But with anomaly detection,we can now detect meaningful movementsin metrics that affect much smallersubgroups of our customers. We’re ableto fix those things and thereforeImprove the reliability of our systems. I strongly believe that if you want tobuild reliable systems, you have to begeared up for moving fast, not slow.There’s a few reasons for this. One, ifyou get your products to your customerssooner,they will give you feedback sooner, youwill learn sooner, and you will makethose products better sooner. Two, as we’ve established, regardless ofwhat we do, some things will break inproduction.So, delaying deployment isn’t reallypreventing the problem. It’s justdelaying the discovery and the learningand the fix. And very importantly,shipping frequently is actually safer. At Monzo, we ship about 300 changes,um on any given day. And on balance, thosechanges are smaller and simpler, andthat makes them easier to reason about.Was it a good change or does it need tobe rolled back? And if it needs to berolled back, the rollback is usuallyFairly straightforward. Now, contrastthat with the alternative where webundle these changes, hundreds andthousands of them into, say, a monthlyrelease. If that goes wrong, it'll be alot harder to reason about. And manytimes, it'll be much harder to undo. Sothe goal is not to never break anything. The goal is to break in a detectable,reversible manner. That's the main goal. So, changeshould be as cheap as possible, andundoing that change needs to be evencheaper. If you do that, then you willbuild better products because you willhave a faster pace of learning, and youwill actually increase the reliabilityof both your systems and theorganization. But you know, you can'tjust will that into existence overnight.You have to set the right culture. Youhave to very carefully articulate yourrisk appetite, and you have to investheavily in automation and low frictiontooling. And that through a lot ofPractice gives the organization theconfidence to move fast. And arelated concept to speed is technicaldebt. So accepting high degree oftechnical risk in order to speed updelivery of certain outcomes. It's not just a useful tool. Sometimesit's necessary. A good example,adversarial domains. An example, uh,fraudsters keep coming up with new waysof how to scam people out of theirmoney. When we respond to those newtactics, we routinely prioritize speedover everything else that allows us toprotect our customers quickly, and we canalways follow up later with atechnically more robust and morepolished solution. And at Monzo, teams own code, whichmeans they're fully accountable for themanagement of the technical debt, and theoverall quality of the codebase. And they are fully empowered andencouraged to use technical debt whenthat leads to better outcomes for ourCustomers. And our engineeringprinciples give us some ground rulesthat make this work. One isyou have to know you're takingon technical debt. If you don't, it'snot technical debt, it's just badengineering. Uh, two, you have to reasonabout what it will cost. Taking on thatin an isolated component or on an easyto reverse change is one thing. Doing soon a trunk system or on a hard toreverse change is something completelydifferent, and you have to know thedifference. And every item of technicaldebt needs to have an owner and a planto pay it back. And we actually findthat this combination of power andaccountability with teams demonstrablyleads to better outcomes. The anomalydetection capability I mentionedearlier, that was a team choosing tomake a proactive investment because theyhad full power and freedom to do so. But look,you can't spend 100% of your time just. Building for reliability, right? If youdo that, you're spending zero timebuilding the new products and featuresthat your customers want from you. So, and because things will happen inproduction, every team is essentiallytrading off between three buckets ofwork. There's proactive investments intoreliability.There’s reacting to stuff that happensin prod, and then there's building thenew products and features. And theanswer, of course, is never in theextremes. There’s a trade-off somewherein the middle. And ideally, thattrade-off is backed up by data. And tohelp our teams reason through thistrade-off, we introduced a metric calledthe cost of quality. And it’sessentiallythe total cost that you pay as anorganization to maintain your systems ata certain level of quality, you know,and you pick the level of quality thatyou want. And it has two components, theProactive one and the reactive one. Proactive component includes things likewriting tests, instrumenting alerts,improving instant response runbooks. Thereactive, of course, is any cost, direct orindirect, that you spend reacting tothings that inevitably happen inproduction. And the more you investproactively, the less you have reactivecost, up to a point. Remember, thingswill happen regardless. So, your reactivecost will never go down to zero. Andimportantly, if you overdo the proactiveinvestment, your reactive cost will goup as well. A good example of this isinstrumenting a bunch of oversensitivealerts. If you do that, your teams are going to spend time responding toincidents that aren't actually a failurein production. And our metric takes thatdynamic into account as well. Inprinciple, there is a global optimumhere. It is an appropriate level ofproactive investment that minimizes thesum of both components. If you're inThat zone, then I think you can haveconfidence that your long-termreliability is in good hands and you canspend the rest of your time buildingthose new products and features thatcustomer wants from you. And I’ve said earlier that reliabilityis not a technical thing only; it also has animportant social component. Culturematters. Incentives matter. And look, ifyour organization is not allowed to takeany risk, your pace of learning willcollapse. That's detrimental. Ifproactive investment is seen as anundesired tax, then your prevention willbe unfunded, and your response will be unpracticed,and that's not good. If rolling back change is seen asfailures, then teams will not roll backchange, and your response to incidents isgoing to suffer, and you’re going to havemore cost and cause more harm as yourespond. And finally, if you don’tencourage a culture of transparency,Then your orc is essentially doomed torepeat the same mistakes over and overagain. So a very healthy postmortemculture, both in principle and inpractice, I think is actually one of themost important things you have to getright in an engineering work. And youknow these postmortems they have to besuper easy to do, mandatory sometimes. Ideally supported by good tooling, andthey have to be focused on learning andunderstanding, and not on blame.And all learnings need to be broadcastfar and wide through the organization. This achieves two things. One, more ofthe organization learnsfrom more of these incidents. And veryimportantly, you normalize the cultureof transparency, and you normalize thefact that failures happen, and that thegood thing is to fail and learn from itand move on. And that’s moreimportant than to not fail in the firstplace. And you have to keep doing thatso that it becomes an integral part ofYour team's culture and of how you work. I think that's the most important part.You know, this is a talk aboutreliability. So it would be remiss of meto not reflect on the fact that we livein a very fast changing world right now.We're shipping more code than we've evershipped.An increasing proportion of changes thatwe deployare made by people who don't write orread code.We have this interesting new class ofnon-deterministic component in oursystem. They're interacting witheverything else, the rest of our system,our teams and our customers. So youcould say that in some ways the notionof how you build reliable systems iskind of being tested with all thesechanges and are these lessons stillrelevant? Actually our experience tellsus that these lessons are onlyreinforced by this changethings moving quickly. Yeah, you bet. Suddenly, you need a lot of automationand low friction tooling. Yeah, exactly.Things fail in new unpredictable ways.Yeah, that's the default. You shouldassume it. You should build for it. Youshould measure for it. And you shouldhave a practice response to it. And youknow,some of the ways in which thesecomponents fail is actually quiteinteresting. So it's good to learn fromthose things as well. So I think as Ireflect on, you know, Monzo's experience, Iwould say that all these lessons thatI’ve spoken about, that we’ve learnedover the years, they are also a big partof what gives us the confidence to moveboldly in what we do next. I'd love to hear what you think. Comeand talk to me, and thank you very muchfor listening., Building for Reliability, LeadDev, 01m1kmx2n320maxy3nv13tyaw2.