Try, try again Sam Newman LeadDev Berlin 2025

Try, try again | Sam Newman | LeadDev Berlin 2025 🔗 **Source:** [Try, try again | Sam Newman | LeadDev Berlin 2025](https://www.youtube.com/watch?v=29NNiZhXe2Q) > [!note]+ Document Notes > tags: [distributed-systems, resilience, retries] > author: Sam Newman > source: LeadDev Berlin 2025 > reference: https://www.youtube.com/watch?v=29NNiZhXe2Q > date created: 2025-01-11 14:30 > > # 🏷️ Title: Try, try again >  ## 🧰 Action Inbox - [H1] Implement idempotency keys in client-server protocols — 05:23 #architecture - [H2] Add server-side request fingerprinting as fallback — 12:45 #resilience - [H3] Combine both approaches for better error handling — 18:30 #protocols ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Try, try again - LeadDev Berlin 2025](https://www.youtube.com/watch?v=29NNiZhXe2Q) ## 📝 Summary - Sam Newman explores retry strategies and idempotency in distributed systems - Covers three fundamental rules that explain why distributed systems are complex - Demonstrates how retries can both solve problems and create cascading failures - Provides practical solutions using idempotency keys and server-side fingerprinting - Emphasizes making operations safe to retry multiple times without adverse effects ## 📖 Main Content Newman argues that distributed systems complexity emerges from three simple rules: information cannot be transmitted instantaneously, network connections can fail, and resources are finite. These constraints make retries both necessary and dangerous. The talk focuses on making retry operations safe through idempotency - ensuring operations can be applied multiple times without changing the result. Two main approaches are presented: idempotency keys (client-generated unique IDs) and server-side request fingerprinting (hash-based deduplication). The combination of both approaches provides the most robust solution for handling client misbehavior and protocol violations. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/29NNiZhXe2Q?t=0) — [H1] Introduction and context - Key idea: Newman introduces himself as author of microservices books trying to "clean up the mess" - Evidence: "Despite what I put in these books, people still thought that microservices were a good idea" #### ⏱️ [02:15](https://youtu.be/29NNiZhXe2Q?t=135) — [H2] Defining distributed systems - Key idea: Simple technical definition versus lived experience reality - Evidence: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable" - Leslie Lamport #### ⏱️ [04:30](https://youtu.be/29NNiZhXe2Q?t=270) — [H3] Three fundamental rules of distributed systems - Key idea: Complexity emerges from three simple constraints - Evidence: "Cannot beam information instantaneously, sometimes cannot reach the thing you want to talk to, resources are not infinite" #### ⏱️ [06:45](https://youtu.be/29NNiZhXe2Q?t=405) — [H2] Why retrying makes sense - Key idea: Transient failures justify retry logic despite the "definition of insanity" quote - Evidence: "Sometimes you can't reach the thing you want to talk to and that might be a transient issue" #### ⏱️ [08:20](https://youtu.be/29NNiZhXe2Q?t=500) — [H3] Resource consumption limits - Key idea: Unlimited retries consume finite resources and tie up system capacity - Evidence: "The activity of retrying consumes resources and resources are valuable things" #### ⏱️ [09:40](https://youtu.be/29NNiZhXe2Q?t=580) — [H2] Retry storms and cascading failures - Key idea: Aggressive retry logic can create positive feedback loops that worsen outages - Evidence: "Square 2017 outage: multipass tried to pull from Redis, timed out, retried 500 times in a loop without delay" #### ⏱️ [12:15](https://youtu.be/29NNiZhXe2Q?t=735) — [H1] The safety problem with retries - Key idea: Network failures create ambiguity about whether operations completed - Evidence: "Two things could have happened: request never got through, or payment was made but response never made it back" #### ⏱️ [15:30](https://youtu.be/29NNiZhXe2Q?t=930) — [H1] Idempotency as solution - Key idea: Operations must be designed to be safely repeatable without changing results - Evidence: "An idempotent operation is one that you can apply multiple times without changing the result" #### ⏱️ [17:45](https://youtu.be/29NNiZhXe2Q?t=1065) — [H2] Idempotency keys implementation - Key idea: Client-generated unique IDs allow servers to detect duplicate operations - Evidence: "Generate a UUID, send to server, store with result - subsequent requests with same ID return same response" #### ⏱️ [20:30](https://youtu.be/29NNiZhXe2Q?t=1230) — [H3] Response consistency principle - Key idea: Servers must return semantically identical responses for duplicate operations - Evidence: "The response should always be the same semantic response - if payment was accepted, always send back 200" #### ⏱️ [22:10](https://youtu.be/29NNiZhXe2Q?t=1330) — [H2] Server-side request fingerprinting - Key idea: Hash-based deduplication when client protocol changes aren't feasible - Evidence: "Rather than client giving unique ID, server calculates hash based on request to determine uniqueness" #### ⏱️ [24:25](https://youtu.be/29NNiZhXe2Q?t=1465) — [H3] Fingerprinting problems: varying fields - Key idea: Timestamps and other changing fields break hash-based deduplication - Evidence: "Fields that vary between attempts like timestamps create different hashes for same operation" #### ⏱️ [26:40](https://youtu.be/29NNiZhXe2Q?t=1600) — [H3] Fingerprinting problems: legitimate duplicates - Key idea: Hash collisions prevent legitimate repeat operations - Evidence: "Pay £100 again a week later - identical parameters create identical hash, second payment rejected" #### ⏱️ [28:55](https://youtu.be/29NNiZhXe2Q?t=1735) — [H2] Combining both approaches - Key idea: Using idempotency keys and fingerprinting together provides better error detection - Evidence: "If client changes amount but keeps same idempotency key, server can detect protocol violation and return 409" ## 📚 References - [Mark Brook's blog post on retry storms](URL) - [Square 2017 outage postmortem](URL) - [AWS API idempotency documentation](URL) ## 🗣️ Paraphrase - Distributed systems fail in complex ways because of three simple constraints: network latency, connection failures, and resource limits - Retrying failed operations is often the right approach, but must be implemented carefully to avoid creating worse problems through retry storms - Operations must be made idempotent through unique identifiers or request hashing to safely handle duplicate requests from retry logic ## 📰 Newsletter Sam Newman's latest talk reveals why your retry logic might be making outages worse, not better. While retries are essential for resilient distributed systems, they can create cascading failures if not implemented safely. The key insight: make operations idempotent using client-generated unique IDs or server-side request fingerprinting. This ensures duplicate requests don't cause double-processing while providing clear error messages for protocol violations. Key takeaway: Before adding retry logic, ensure your operations are safe to repeat multiple times. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create retry pattern demos, conference talks on distributed systems resilience - Education: Workshop curricula on idempotency patterns, office hours for architecture questions - Developer experience: SDK retry configuration documentation, default timeout recommendations - Developer marketing: Case studies of retry failures, best practices content series - Community management: Moderate discussions on system design patterns, collect retry horror stories #### OSPO - Policy & Governance: Contribution guidelines for distributed systems libraries - License Compliance: Review retry/backoff library licenses across projects - Security Management: Analyze retry amplification attack vectors - Software Inventory: Catalog retry implementations across organization - Developer Support: Training on distributed systems patterns, tooling for testing retry logic - Community Engagement: Fund upstream retry/resilience library development - Strategic Alignment: KPIs for system reliability, OKRs for reducing retry-related outages #### Community - Maintainers: Roadmap support for adding idempotency to APIs - Contributors: First-issue programs for implementing retry patterns - Sponsors: Fund development of retry testing frameworks - Ecosystem: Coordinate retry behavior across interconnected services #### WordPress - Core: Implement idempotency for REST API endpoints, performance budget for retry delays - Core Performance: Profile retry overhead, optimize backoff algorithms - Core AI: Retry logic for AI service calls, evaluate model availability patterns - Core Program Team: Coordinate retry implementations across teams - Mobile: App retry policies for poor connectivity, telemetry on retry success rates - Playground: Labs for testing retry scenarios, educational examples - Community: Meetup presentations on WordPress scalability patterns - Training: Lesson plans on building resilient WordPress systems - Marketing: Case studies of WordPress.com retry strategies - Support: Escalation paths for retry-related issues - Design: UX for retry feedback, loading states during retries - Documentation: DevHub guidance on implementing retries - Meta: .org API reliability improvements, retry policies for plugin/theme directories - CLI: Command retry logic, network failure handling - Hosting: Cross-host retry recommendations, load balancer configuration - Openverse: Retry policies for external API integrations #### The WPCC - Programs: Fund fellowships for distributed systems expertise - Projects: Governance for retry behavior standards - Funding: Support retry testing infrastructure development #### FAIR - Technical independence: Retry policies that don't favor specific vendors - Policies: Transparency in retry behavior and failure modes - Governance artifacts: Standards for idempotent API design ## 🧪 Hypotheses - Systems with proper idempotency implementation have 50% fewer duplicate operations during outages - Combining idempotency keys with request fingerprinting reduces client protocol violations by 80% - Retry storms contribute to 30% longer outage duration in distributed systems - Organizations with retry testing frameworks experience 40% fewer production retry-related incidents ### 📊 Methodology - Measure duplicate operation rates before/after idempotency implementation - A/B test different retry backoff strategies and measure system load - Survey engineering teams on retry-related outage frequency and duration - Analyze correlation between retry configuration and system stability metrics ### ❓ Questions - How can we automatically detect when retry logic is contributing to system instability? - What are the optimal retry patterns for different types of distributed system failures? - How should retry behavior differ between internal service calls and external API integrations? ### 🔮 Future Research - Machine learning approaches to adaptive retry policies based on failure patterns - Formal verification methods for proving retry logic correctness - Cross-service retry coordination to prevent system-wide retry storms ### 💡 Implications - Practice: All API operations should be designed with idempotency from the start - Policy: Organizations need standards for retry behavior and testing requirements - Security: Retry amplification attacks become a new class of DoS vulnerabilities - Education: Distributed systems education must emphasize safe retry patterns - Funding: Investment needed in retry testing infrastructure and monitoring tools ### 🧠 Learning Styles Assessment Reading level estimate: 10th grade #### 👀 Visual learners - Present: System diagrams showing retry flows, timeline visualizations of failures - Improve: Network topology diagrams, retry storm visualization animations #### 🔊 Aural learners - Present: Conference talk format with clear verbal explanations - Improve: Audio summary of key retry patterns, podcast discussion format #### 🗣️ Verbal learners - Present: Clear definitions, analogies (payment examples), concrete examples - Improve: Story-driven explanations of outage scenarios, glossary of terms #### 🧩 Kinesthetic learners - Present: Code examples, step-by-step implementation guides - Improve: Hands-on labs for testing retry behavior, interactive failure simulations #### 🧠 Logical learners - Present: Clear cause-effect relationships, systematic problem breakdown - Improve: Decision trees for retry strategies, mathematical models of retry behavior #### 🧑‍🤝‍🧑 Social learners - Present: Conference setting with audience interaction - Improve: Team exercises for designing retry policies, peer review of retry implementations #### 🧘‍♀️ Solitary learners - Present: Individual reflection on system design choices - Improve: Self-assessment checklist for retry implementation, personal case study analysis #### 🌿 Naturalistic learners - Present: Ecosystem analogies for distributed systems - Improve: Biological metaphors for system resilience, natural failure recovery patterns ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive slide content - Improve: Alt text for system diagrams, code example descriptions #### Keyboard navigation - Present: Standard presentation controls - Improve: Keyboard shortcuts for demo navigation #### Contrast and readability - Present: High contrast slides, large text - Improve: Color-blind friendly diagrams, better code syntax highlighting #### Text resizing and reflow - Present: Scalable presentation format - Improve: Mobile-friendly slide layouts #### Media and captions - Present: Video with transcript provided - Improve: Audio descriptions of visual diagrams #### Forms and interactions - Present: No interactive elements requiring improvement - Improve: N/A for presentation format #### Standards reference - Map to WCAG 2.2 AA for future interactive materials ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global technology context, universal system challenges - Improve: Examples from different cultural contexts, international case studies #### Inclusive language - Present: Technical terms clearly defined, accessible examples - Improve: Avoid idioms like "down the rabbit hole", provide plain language summaries #### Neurodiversity support - Present: Clear structure, consistent formatting - Improve: Content warnings for technical complexity, summary sections #### Representation - Present: Focus on technical content over personal characteristics - Improve: Diverse case study sources, varied organizational examples #### Psychological safety - Present: Non-judgmental approach to technical failures - Improve: Emphasize learning from mistakes, blame-free problem solving ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Strong logical progression from distributed system constraints to retry necessity - Inductive strength: Good use of concrete examples (Square outage) to support general principles - Causal reasoning: Clear cause-effect relationships between retry behavior and system failures - Analogical strength: Effective analogies (payment example) to explain abstract concepts #### Fallacies - Appeal to authority: Minor issue with Einstein quote attribution, but corrected at end - False dichotomy: Could present more nuanced retry strategies beyond binary retry/no-retry #### Tone of voice - Overall tone: Conversational, self-deprecating, technically authoritative with humor to maintain engagement ### 🛡️ DARVO Analysis - Deny --- ## Full Transcript [00:01] [music] [00:05] Thank you very much. You are very much [00:07] my people. We've had two days of awesome [00:10] talks, an opportunity to network. You [00:12] could have gone home, you stayed. You [00:14] are my people. Or you've fallen asleep [00:16] and no one's woken you up. Either way, [00:18] I'm taking it as a win. So, thank you so [00:20] much. Um, as Pat mentioned, I'm I have [00:23] written some books in the past. I wrote [00:24] books about microservices. And despite [00:27] what I put in these books, people still [00:29] thought that microservices were a good [00:30] idea. And so went and started building a [00:33] lots of these architectures. And so as I [00:35] now feel duty bound to sort of try and [00:37] clean up some of the mess and so that's [00:39] why I'm working on my new book, which is [00:41] uh how you stop your system from falling [00:43] over once you've built it basically. And [00:45] this talk is very much taking some of [00:47] the ideas that I'm putting in that book [00:48] and some of the kind of I I've always [00:50] been trying to distill down some of the [00:52] lessons I've learned myself and that's [00:54] what I'm going to be sharing with you [00:55] today. and what are I think some kind of [00:57] fundamental building blocks that I think [00:58] everyone should really be aware of when [01:00] it comes to distributed systems. Uh and [01:02] we're going to focus on some narrow bits [01:04] in a way. We're going to talk about [01:05] retries. We're going to talk about item [01:07] potency, but fundamentally we're talking [01:10] about this in the context of distributed [01:12] systems. And we should probably define [01:14] what a distributed system is to start [01:16] off with because definitions are [01:18] important. I like defining things. So [01:20] we've got a very sort of I would say [01:22] sparse definition of what a distributed [01:25] system is which definitely uh avoids any [01:27] of the sociotechnical aspects of it. We [01:30] can think of a distributed system very [01:31] simply as two computers two or more [01:34] computers communicating with each other [01:37] over a network. Right? It seems very [01:39] simple. Computer A network computer B [01:43] and this is this seems quite technically [01:46] this is kind of a correct definition. [01:48] What it doesn't do is really speak to [01:51] the lived experience of having a [01:54] distributed system which is much more [01:56] painful than that isn't it right two [01:58] computers network what could go wrong [01:59] everything could go wrong right and [02:01] that's why my favorite definition of a [02:03] distributed system actually comes from [02:05] Leslie Lamport and it's a distributed [02:07] system is one in which the failure of a [02:09] computer that you didn't even know [02:11] existed can render your own computer [02:14] unusable this I feel is a more accurate [02:16] definition of what a distributed system [02:18] is. And of course, we had the outage a [02:21] couple weeks back of USD East1, right? [02:23] So, we all learned all about how [02:25] interconnected our systems were. We [02:27] learned, for example, that some of us, [02:29] if we had spent several thousand on a [02:31] mattress, now we couldn't use our [02:32] mattress for some reason. So, we should [02:34] probably update this quote to be more [02:37] accurate to say, "A distributed system [02:39] is one in which the failure of a [02:41] computer you didn't even know existed [02:42] can render your own mattress unusable. [02:46] The world is a wonderful place. So while [02:49] writing the book, I've been trying to [02:50] kind of create a book that anybody could [02:52] start with and start with the easy [02:54] things and get into the more [02:55] complicated. Distributed systems have a [02:57] lot of very complex topics within it. We [02:59] could talk about bisonantine consensus [03:01] theory. We could talk about cap theorem, [03:04] why distributed transactions are bad and [03:06] all of this stuff. And these are quite [03:07] complicated topics and can be quite hard [03:09] to understand. So I wanted to try and [03:12] drill down into what is the complexity [03:14] or what is the fundamental problems that [03:16] cause that complexities to emerge. So [03:19] I've kind of tried to come up with three [03:21] rules. I'd looked at the fallacies of [03:22] distributed computing and thought that's [03:23] eight. I can do better. I've come up [03:25] with three simple rules that explain why [03:27] distributed systems are miserable things [03:29] to work on. Uh the first problem is you [03:32] cannot beam information instantaneously [03:35] between two points. Now at this point [03:36] some smartass is saying oh but Sam what [03:39] about quantum entanglement? At which [03:41] point I'd say quantum entanglement is [03:43] not a networking protocol. It also only [03:45] works for read states. But if you have [03:47] solved using quantum entanglement as a [03:49] networking protocol congratulations you [03:51] are now rich. But in reality it takes [03:54] time and as developers as an architects [03:56] we are partly in control of how long [03:59] this takes. We can decide how big a [04:01] packet is or how big the payload is. We [04:03] might be able to choose our networking [04:05] protocols but we probably don't have [04:07] control over the networking equipment [04:08] nor what else is maybe going on in the [04:10] environment at the point in which we [04:12] send that packet along. So we have to [04:15] deal with this. There's going to be some [04:16] variability there as well. The second [04:18] problem with distributed systems is [04:20] sometimes we cannot reach the thing that [04:22] we want to talk to. Again we try and [04:25] control this a bit. We do things like [04:27] having multiple copies of a service for [04:29] example be a load balancer in an attempt [04:31] to ensure that if a single service [04:33] instance fails that we can still [04:34] maintain availability of that service [04:36] and these are sensible things to do but [04:38] we cannot fundamentally guarantee that [04:40] our service would always be available [04:42] things outside of our control do happen. [04:44] Things catch fire. US East1 fails and in [04:49] one situation I had rabbits eat your [04:51] network cable. I think rabbits ate my [04:53] network cable might be the name of my [04:55] autobiography. It happened to me. How am [04:57] I going to control for rabbits? Going [04:59] out with a shotgun is not what I thought [05:01] I would be doing when I became an [05:02] architect. Finally, [05:05] resources are not infinite. Resources [05:09] can and indeed will run out and this [05:11] often will bring down a distributed [05:13] system. Uh we're going to explore some [05:15] fundamental ideas about how we make our [05:16] systems more robust and see how these [05:18] elements will certainly number two and [05:20] three uh come into play. But even when [05:23] looking at the most complex aspects of [05:24] distributed systems, these three rules [05:26] have kind of not steered me wrong so [05:28] far. So let's start with something nice [05:30] and easy for today. We're going to be [05:32] talking about retries. [05:35] And the first thing is, is retrying [05:37] sensible? We have, of course, the famous [05:39] quote. The definition of insanity is [05:41] doing the same thing over and over again [05:43] and expecting a different result. Now, [05:45] this quote is flawed in many ways, but [05:48] specifically it's flawed in the context [05:50] of a distributed system because we [05:51] actually know that trying to do the same [05:53] thing over and over again sometimes is [05:54] exactly the right thing to do. Why? [05:57] Well, because sometimes you can't reach [05:59] the thing you want to talk to and the [06:01] thing and that might be a transient [06:03] issue. You've made a call. Your call has [06:06] been routed to an instance of your [06:08] service of choice which is right now [06:10] having a bit of a wobble, having a bit [06:12] of a midlife crisis. it can't serve your [06:14] traffic for whatever reason. It's been [06:16] wiped by the outofmemory killer on [06:18] Kubernetes, whatever else may have [06:19] happened. And so a second attempt to [06:22] establish connection with this service [06:24] might actually yield with a successful [06:27] answer. And when we have operations in [06:29] our system that we want to happen, if [06:31] they don't work the first time, try try [06:34] again. it might be a really good idea [06:36] because if we can't retry key operations [06:39] in our systems, well then our systems [06:40] will actually have more failure that we [06:42] actually have to pass back to the user. [06:43] So being able to retry is a key part of [06:46] trying to make a system more resilient. [06:50] So [06:51] we if we first we don't succeed, it's [06:54] definitely worth trying again sometimes. [06:57] Um so then of course we lead into the [06:59] conversation of this how many times [07:00] should we retry? How many times should [07:03] we retry? In fact, why are we even [07:05] bothered with how many times we retry? [07:07] Well, remember resources are not [07:09] infinite. The activity of retrying [07:12] consumes resources and resources are [07:14] valuable things. Uh if you're retrying [07:17] quite a lot, you might be tying up [07:19] resources. Certainly, a lot of times if [07:21] you maybe time out, you then retry, the [07:24] original call you make might still be [07:26] constraining resources on the server [07:27] you're talking to. Opening a connection, [07:30] holding open sockets, that is consuming [07:32] resources. And so if you're retrying too [07:34] much, you're tying up resources that [07:36] could be used for other things. And also [07:38] look, you know, people get bored of [07:40] waiting after a while. So having a [07:42] maximum number of retries is absolutely [07:44] something you need to do. Now, of [07:46] course, this kind of implies things that [07:48] the client is doing to keep the system [07:49] more resilient. Uh the server itself [07:51] should also be protecting itself. So [07:53] doing things like loading absolutely [07:54] makes sense. That's a topic for another [07:56] time. Retries though, can be dangerous [07:59] if you retry too much. Uh Mark Brook has [08:02] got a great blog post all about this and [08:04] if you're interested dig systems I'm [08:05] sure you already know who Mark is. Uh [08:07] but this graph I think sums it up quite [08:08] nicely. You start having a few bad [08:10] things happen. You start retrying that [08:12] overloads your system which causes more [08:14] failures which causes you to retry which [08:16] causes you to overload your system which [08:18] causes more retries. And this is not [08:20] good. As a concrete example back in 2017 [08:24] Square had a major outage. It was caused [08:26] by a restart in their multipass system. [08:29] Multipass was used for multiffactor [08:31] authentication. Multipass spun up. It [08:33] tried to pull data from Reddius. Reddis [08:35] was being a bit overloaded because of [08:37] all the multipass nodes spinning up. So [08:39] those calls were timing out. No worry. [08:41] The multipass had a little retry logic [08:43] loop in it. So it would retry again. It [08:45] would retry 500 times in a loop without [08:48] any delay between those retries. So [08:51] basically multipass pummeled Reddus into [08:54] submission. And so people thought, well, [08:56] the way to fix it is we'll restart. is [08:58] clearly overloaded and that just made [09:00] the problem worse. Now there's a great [09:01] postmortem around how they fixed all [09:03] this. It it's you know people this was [09:06] bit of naive code that had never had an [09:07] issue before. So I don't got don't blame [09:10] Square. I actually credit them for [09:11] sharing this stuff for us. Now so [09:14] retrying have a bit of a delay. We can [09:17] talk separately about jitter and all [09:18] those kinds of other things. I want to [09:20] come on to a more interesting topic [09:21] which is is it safe to retry? And this [09:24] is a bit problematic [09:26] because there's a situation that occurs [09:28] where we don't know if retrying is safe. [09:31] So let's imagine a situation here, [09:33] right? You really want to pay Sam, my [09:35] name's Sam by the way, £100. You're [09:37] committed to doing so. Maybe because [09:39] you're groggy after two days of learning [09:41] new things and you want to give Sam [09:43] £100. You say to the other to the order [09:45] service, please pay Sam £100. And you do [09:49] don't get a response. You don't get an [09:51] acknowledgement that the payment has [09:52] been made. Now, this is clearly [09:54] earthshattering for you because you [09:56] really want to make sure that I get my [09:57] money. Um, so what's going on here? [10:00] Well, from the point of view of the [10:01] client code, two things could have [10:03] happened. The first thing is that the [10:06] request to make that payment never [10:09] actually got through to the order [10:11] service. [10:12] It never arrived or maybe it did arrive, [10:16] but when in attempting to process it, [10:18] the poor payment service fell over. [10:20] Right? So in this situation, we've sent [10:22] the request of payment, but it hasn't [10:24] been processed. And that's why we didn't [10:26] get a response. It makes sense. The [10:27] payment wasn't made. Therefore, we [10:29] didn't get a response. That's fine. Um [10:32] the other possibility though is that we [10:34] sent the request saying, "Please pay Sam [10:36] £100." Again, remember that for later. [10:39] Please say, "Pay Sam £100." And the [10:42] payment was made, but the response never [10:44] made it back to you. So that's why we [10:46] didn't get the acknowledgement. So, at [10:48] this situation, you're not sat, you're [10:49] not sure if I've got my £100 or not. [10:52] We're very much in a cat in a box [10:54] situation, right? Is the cat alive? Is [10:56] the cat dead? We're not quite sure. Um, [10:58] the key thing is I put a picture of a [11:00] cat up there in case people are asleep, [11:01] but now there's a cat on the screen. [11:02] You're wondering why. Um, but let's move [11:04] on. So, how do we deal with this [11:06] situation? [11:07] >> Hope you're enjoying this video, but did [11:09] you know Lead Dev [music] has a lot more [11:11] to offer? [11:12] >> When I think about lead, I think about [11:14] bricks and hearts. [music] Bricks are [11:16] those foundational things that you need [11:18] to know and lead dev provides that [11:21] through webinars, through videos, [11:23] through cutting edge research, [music] [11:25] but hearts are the things that you need [11:27] to glue yourself to the people at [11:29] [music] work. And lead does that through [11:32] conferences and meetups. [11:33] >> Make sure you check out leadv.com. [11:34] [music] [11:41] When we have key operations in our [11:43] system that we need to make sure [11:44] happened and if we're not sure if they [11:47] happened, then being able to retry them [11:49] is a good idea. But that can result in [11:51] us doing things more than once. You say, [11:54] "I want to give Sam £100." You don't get [11:57] that response back. You really want to [11:59] make sure I get paid. And so, it's [12:01] natural to retry. The problem now is [12:04] that I've got paid twice. Now, for me, [12:06] this is not a problem. For you, you [12:08] might be a bit annoyed. How do we solve [12:10] this? Item potency is key. So an item [12:15] potent operation is an operation that [12:18] you can apply multiple times without [12:20] changing the result. Some operations are [12:23] item potent by default. I mean effect [12:26] read operation is by definition an item [12:28] potent operation. Most rights are not at [12:31] least in terms of how we've defined this [12:33] one. This particular operation we've [12:35] reattempted that operation. The payment [12:37] ends up getting made twice. it is not an [12:39] item potent operation. There are two key [12:41] ways to make operations item potent. [12:43] We've got item potency keys which are [12:46] what you want to do if you can and if [12:48] you can't do them you've got serverside [12:50] request fingerprinting which is [12:52] massively flawed but it might be better [12:54] than nothing and you might want to use [12:56] both item potency keys very simple [12:58] straightforward ID. This is where you [13:00] generate a unique ID for the operation [13:03] being carried out. This allows the [13:05] server to determine that we've seen the [13:07] operation before. So when sending the [13:10] payment now, I generate a UU ID. 456 is [13:13] a very bad UU ID. I could recommend a [13:16] version seven or maybe a version five of [13:19] the UUID specification for UU ID fans [13:21] out there. Um, so we generate that UU [13:24] ID, we send that to the server for the [13:26] payment. And on the server side now, we [13:27] store that along with the result. So we [13:29] carried out that payment. Here's the UU [13:31] ID for that operation. We carried it out [13:33] appropriately. Nice, simple, easy. We [13:36] get a subsequent request coming in where [13:39] the server has the client has [13:40] reattempted this operation. It sends [13:43] through the same ID. So we can see that [13:46] we've already processed that ID and we [13:48] can say don't worry about it. That [13:50] payment was made. We don't try and [13:51] reattempt payment. This is very simple [13:54] and straightforward and it's quite easy [13:56] to do if you can build it into the [13:57] protocol from the beginning. So a bit of [13:59] a question here is what happens? What's [14:02] the response you send back? If a server [14:04] has seen that attempted operation two, [14:07] three, four, five times, do we change [14:09] what we send back? And the answer is no. [14:12] The response you send back should always [14:14] just be the same semantic response. So [14:17] if I originally said, pay Sam, the [14:21] payment was made, I send you back a 200. [14:22] So in this example, I'm using HTB status [14:24] codes. So I send you back a 200, payment [14:27] accepted. Um, and then we reattempt the [14:30] operation anyway. Well, the payment was [14:32] accepted. So that is the answer I'm [14:33] going to send you back. The payment was [14:34] carried out is exactly the same [14:37] semantically. Now you can of course put [14:39] metadata in here to communicate to the [14:42] client that they've asked you this to [14:44] make this payment like 15 times and [14:46] could you please stop asking but [14:48] semantically we want the codes the [14:50] status codes to be the same. So [14:51] programmatically nothing needs to change [14:53] in the client side behavior. This [14:55] equally applies if it didn't work like [14:57] if it didn't work for good reasons [14:59] rather than being a system failure. So, [15:01] if you tried to make a payment and you [15:03] got back the you're poor status code, [15:05] you got to love HTTP, right? It's got a [15:07] HTTP status code for you having no cash. [15:09] So, you come back with a 402 cost of [15:11] living error which says you've got no [15:13] money because we couldn't carry out that [15:15] payment. If you reattempt that same [15:17] operation, well, that operation was [15:20] processed. You shouldn't like the [15:21] answer. You still send back the same [15:24] response. Now the issue with the state [15:28] with the item potency keys is it assumes [15:30] you've built them into your protocol [15:31] from the beginning. So if you're trying [15:33] to retrofit that into a system that's [15:35] going to require a backwards compatible [15:37] change in your client server protocol. I [15:39] know many of you are sort of modifying [15:41] or modernizing systems that used to run [15:44] on single processes. You're making them [15:45] more distributed. You may be in [15:47] situations where it's not easy for you [15:49] to change client server protocols. And [15:51] that's where we can use these things [15:52] called serverside request fingerprints. [15:54] They are flawed, but they can still be [15:56] useful. So rather than the client giving [15:58] us a unique ID, instead what the server [16:01] is going to do is going to look at the [16:03] request it's being sent and it's going [16:05] to calculate a hash based on that and [16:07] use that to determine uniqueness. So in [16:10] this example here, we're sending again [16:12] this is a HTTP based example, but this [16:13] would all work with a messagebased [16:14] system as well. So we got the header and [16:17] the footer, and we take that body, which [16:19] represents the kind of the operation [16:21] we're carrying out. And in this example, [16:23] we're just generating a simple simple [16:25] MD5 hash. Uh whether or not you'd use [16:27] MD5 or not depends on the nature of the [16:29] protocol here. But we've generated the [16:31] hash and then that's what we store. So [16:33] rather than using a UU ID given to us by [16:36] the client to see to look for uniqueness [16:38] of operation, instead we're using that [16:40] hash instead. That's actually quite nice [16:42] simple and easy thing to do and it only [16:44] needs to be implemented on the server [16:47] side. So this seems good, right? Until I [16:49] talk to you about all the ways in which [16:50] it's not good. Uh first problem, what if [16:54] you've got fields in the request you're [16:56] being sent that vary between attempts? [16:59] So in this example here, the protocol [17:01] gives us a time stamp that's actually in [17:03] the body of the request being sent and a [17:07] retry time moves on the time stamp [17:09] increases and so we end up with two very [17:12] different hashes because those [17:14] timestamps are different even though the [17:16] operation we're trying to carry out is [17:17] the same operation. So I see stuff like [17:20] this all the time, especially in older [17:22] applications. So straight away, if we [17:24] took a hash of that whole body, we would [17:27] now have a problem. We would now be [17:28] getting things that look like they were [17:30] different operations, even though they [17:32] are semantically the same operation. How [17:35] do we solve this? Well, then you have to [17:36] start doing things like masking out [17:38] certain fields. You say, well, these [17:40] fields are the ones I care about. These [17:41] fields I'm going to ignore. This kind of [17:43] implies that the payload you're being [17:45] sent is easily parsible. Um, and that's [17:48] not always the case and it can be quite [17:50] fiddly, especially again working with [17:52] older legacy systems to pick that out. [17:54] Now, of course, the right answer is to [17:55] say why the heck is the timestamp in the [17:58] body? We should move that up into the [18:01] header, which is the right answer. But [18:03] this implies that you can change the [18:04] client server protocol. And if you can [18:06] change the client server protocol, just [18:08] put an item potency key in in the first [18:10] place. Another problem. [18:14] You've paid Sam the £100 and a week has [18:17] gone by and you thought, do you know [18:18] what I haven't done recently? It's pay [18:21] £100. So I think I want to pay Sam [18:22] another £100. One week later you go and [18:25] make the same payment to the same [18:27] person, but unfortunately because the [18:29] parameters are the same and that's what [18:31] we're using to generate the hash, the [18:32] hashes are now identical. So now we're [18:35] going to have a problem where we're [18:36] going to have a false negative. that [18:37] second payment is going to be rejected [18:39] because it's going to look the same. I'm [18:41] bereff. You're crying. No one's having a [18:43] good time at this point. And this the [18:46] identical hash problem here is just [18:48] because two operations happening close [18:50] together look the same. So this is why [18:53] when using if you're only relying on [18:55] hashes, it's really important that you [18:57] understand how long to keep these hashes [18:59] for. So a simple way to do this would be [19:02] to add some kind of expiry on the end of [19:04] how long you keep those for. Now how [19:06] long that kind of window might be can [19:09] vary. I was doing a workshop a couple of [19:11] weeks ago some people in Denmark and [19:13] they explained to me how they had a [19:14] credit card payment system u was one of [19:17] the banks were using it was one of the [19:18] payment card companies used out there. [19:20] So if you went into a shop, you bought [19:21] something for 50 croner, um, and then a [19:24] couple of minutes later you bought you [19:26] then to carry out another payment for [19:28] exactly 50 croner, that second payment [19:30] would be rejected in the shop because [19:32] they're doing basically a sort of a hash [19:34] generation of the payment being carried [19:36] out. If you waited 20 minutes, it's [19:38] fine. You could do the second payment [19:39] again. So clearly they've come up with [19:41] an idea of, okay, well, how likely is it [19:43] that you're going to carry out two [19:44] payments that close together for exactly [19:47] the same amount of money? the chances [19:49] are quite low and so we'll keep you know [19:51] 5 10 15 minutes is kind of fine. So it [19:54] does actually come up in the real world [19:56] just quite rarely. But then if we shift [19:58] the domain this becomes even more [20:01] problematic. What if you have a [20:03] situation where you do legitimately want [20:05] to do the same thing over and over again [20:07] in a very short space of time and for [20:10] each of those things to be a different [20:11] operation. To give you an example, [20:14] imagine you're spinning up cloud [20:15] resources. So you're on AWS and you want [20:18] to spin up some EC2 instances. That's [20:20] the van managed virtual machines. So [20:22] it's quite valid for me to use the AWS [20:24] API to provision an EC2 instance uh with [20:28] you know the instance type I want and [20:29] the size and all of that other stuff and [20:31] then stick that you know line of code in [20:33] a for loop. So I'm going to iterate 20 [20:35] times through a for loop spinning up 20 [20:38] identical looking VMs. Now there are [20:41] more elegant ways to do that but it's a [20:43] completely legitimate thing to do with [20:44] the APIs. But now I've got 20 operations [20:47] that look exactly the same, which are [20:49] actually discrete operations, all of [20:51] which I want to apply. And if that's a [20:53] domain you're in, then this isn't going [20:55] to work for you. That, by the way, is [20:57] why the AWS APIs make use of item [20:59] potency keys for this purpose rather [21:02] than just relying on hashes. [21:04] So if you can change the client server [21:07] protocol, then item potency keys are far [21:09] and away the easiest ways to make [21:11] operations item potent. Just make sure [21:13] you get your UIDs right. You don't have [21:15] to go down the massive rabbit hole of [21:16] looking into how UU ID generation works. [21:18] Just stick with version seven if you [21:20] want something sortable. It's fine. If [21:22] you can't retrofit it, then despite the [21:24] downside, doing some kind of server side [21:26] fingerprinting is still a good approach [21:28] because ultimately it's better than [21:30] nothing. However, what about using these [21:33] two things together? This can actually [21:34] be really useful when thinking about the [21:37] resiliency of a system. We also want the [21:39] system to behave consistently [21:41] consistently and with clarity. [21:44] We also have this situation that [21:46] sometimes clients misbehave. There are [21:48] these things called bugs that crop up in [21:50] the system every now and then. Wouldn't [21:52] it be great if we could try and put some [21:53] things into our system to kind of deal [21:56] more elegantly with situations where [21:57] clients do misbehave because they often [21:59] do? And using both item potency keys and [22:03] serverside fingerprinting can be really [22:04] useful for this purpose. Let's consider [22:07] a scenario. We've got the order service. [22:09] We've tried to carry out a payment with [22:12] for £100 with the key of 456. [22:16] That payment was carried out. It's [22:17] happy. We're fine. And then the client [22:20] didn't get the response and thinks, I [22:22] better reattempt this. But for whatever [22:24] reason, we've now changed the amount of [22:26] money. So rather than trying to pay [22:28] £100, we're now paying £200. It's a [22:31] could be a weird coding bug. Maybe [22:33] somebody's trying to, you know, hack us [22:36] in some sort of weird way. The problem [22:38] is at this point if the payment service [22:40] responded and said that payment was [22:42] accepted because all we've done is [22:43] looked at the item potency key the [22:46] client is now in a confused state [22:48] because did we accept payment for £100 [22:51] or £200. If they didn't see the original [22:53] response and only saw the second [22:55] response they might now think we've [22:57] carried out a payment for £200 despite [22:59] us being the fact we did not. We [23:02] actually returned £100. In this [23:04] situation the client is actually wrong [23:06] here. the inherent kind of agreement of [23:09] the protocol here is if you keep the [23:10] item potency key the same the fields for [23:13] the operation themselves should also be [23:14] the same and so in this situation it is [23:17] much better if you're able to to tell [23:20] the client that they are wrong and again [23:23] coming back to using HTTP status codes [23:25] probably the right status code here [23:26] would be to say I'm giving you a 409 [23:29] right you've done something wrong here [23:31] this for me is all about giving about [23:33] providing clarity this is the reason why [23:35] most public facing With all the public [23:37] facing APIs I've been a to dive into in [23:38] detail in terms of how they work behind [23:40] the scenes, use a combination of item [23:42] potency keys and hashing. Certainly the [23:45] AWS APIs do. So if you change the [23:49] parameters but keep the item potency key [23:51] the same, the AWS APIs actually error to [23:54] you. Now whether or not you're doing a [23:56] public facing API or not, this can be a [23:58] useful thing to do for internal APIs as [24:00] well because I think this clear [24:02] communication around a client of a [24:03] protocol helps educate people. you're [24:05] explaining that you're doing something [24:07] wrong rather than just trying to [24:08] transparently pretend that everything's [24:10] okay. So using both of these things [24:13] together can be a really really good [24:14] idea. We just need to have them both in [24:16] a table. I mean it's not that much more [24:18] work to do, right? We're just adding on [24:20] all of these things in the status codes. [24:23] So what we've learned so far is that [24:25] doing the same thing over and over again [24:28] may well be sensible within reason. [24:30] Maybe not 500 times without any back off [24:32] in between, but only if you can do so [24:35] safely. Uh, speaking of doing the same [24:38] thing over and over again, um, we keep [24:41] using this quote, don't we, over and [24:43] over again. The definition of insanity [24:44] is doing the same thing over and over [24:46] again and expecting different results. [24:48] One thing we keep doing over and over [24:49] again is attributing this to Einstein. [24:52] There's no evidence he ever said this. [24:54] None. I didn't even say he said it, by [24:57] the way. I'll just let you lead [24:58] yourselves down that little merry path [25:00] by putting his picture on it. But yeah, [25:01] apparently there's no evidence [25:02] whatsoever that he ever said this. So in [25:05] summary, distributed systems are [25:07] complicated things. They are an example [25:09] of emergent complexity. But a lot of [25:10] that complexity comes from very simple [25:12] rules that I think most people can [25:14] understand. You cannot beam information [25:16] instantaneously between two points. [25:18] Sometimes you can't reach the thing you [25:20] want to talk to and resources are not [25:23] infinite. So if at first you don't [25:25] succeed, retry makes sense to a point. [25:29] Um but if you do want to retry, make [25:32] sure you make your operations item [25:34] potent. Uh if you want to know more, [25:35] I've got a larger longer version of this [25:37] talk available. You can find on the QR [25:39] code. You can also get information about [25:40] the book and find out how to read it on [25:42] early access. But thank you so much for [25:43] your time. [applause] [25:47] [music]