Your Blueprint for Automated Incident Recovery

Your Blueprint for Automated Incident Recovery ![rw-book-cover](https://img.youtube.com/vi/FPsuYDsHOBg/maxresdefault.jpg) 🔗 **Source:** [Your Blueprint for Automated Incident Recovery](https://www.youtube.com/watch?v=FPsuYDsHOBg) > [!note]+ Document Notes > tags: #incident-response #automation #devops #reliability #on-call #orchestration #ai-tools > author: LeadDev > source: LeadDev > reference: https://www.youtube.com/watch?v=FPsuYDsHOBg > date created: 2024-12-19 15:30 > > # 🏷️ Title: Your blueprint for automated incident recovery >  ## 🧰 Action Inbox - [H1] Implement game days and chaos engineering practice — build incident response muscle memory #incident-response - [H2] Audit automation for "who watches the watchers" problem — prevent automation becoming another source of toil #automation - [H3] Create feedback loop from postmortems to system improvements — encode learnings into circuit breakers and runbooks #reliability ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [LeadDev Panel Discussion](https://www.youtube.com/watch?v=FPsuYDsHOBg) - [Everbridge xMatters](https://www.everbridge.com/products/it-alerting/) ## 📝 Summary - Panel discussion on evolving incident response from chaotic firefighting to orchestrated automation - Key focus on building trust in automated systems through progressive implementation with human oversight - Major challenge: avoiding skill atrophy as automation reduces hands-on incident experience - Critical insight: speed without judgment is dangerous - automation should accelerate decision-making, not replace it - Bottom line: Best incident response minimizes incidents through proactive systems rather than just faster reaction ## 📖 Main Content The panel explores the tension between increasing system complexity and the push toward automated incident response. Four experts discuss how teams can move beyond reactive firefighting toward orchestrated, AI-assisted incident management while maintaining human judgment and avoiding automation pitfalls. Core themes include the importance of institutional knowledge capture, the need for progressive trust-building in automated systems, and the critical balance between speed and judgment during incidents. The discussion emphasizes that the goal isn't maximum automation but "maximum safe acceleration" that reduces toil while preserving human decision-making capability. Key challenges addressed: false positive alerts causing burnout, the "hot potato" problem of unclear incident ownership, and the risk of skill atrophy as automation handles more routine incidents. Solutions focus on game days for practice, shadow canaries for early detection, and encoding postmortem learnings into system improvements. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/FPsuYDsHOBg?t=0) — [H1] Panel introduction and expert backgrounds - Key idea: Four experts covering engineering leadership, platform operations, and incident management tooling - Evidence: "I've been dealing with incidents for nearly 20 years now since I was a one-person ops team" #### ⏱️ [03:15](https://youtu.be/FPsuYDsHOBg?t=195) — [H2] First 20 minutes of incidents - where breakdown occurs - Key idea: Organization and communication are more critical than technical debugging in early incident response - Evidence: "Engineers want to figure out what the technical problem is and they start rabbit holing and forget the people side of things, the customer side of things" #### ⏱️ [05:45](https://youtu.be/FPsuYDsHOBg?t=345) — [H3] Signal orchestration and context enrichment - Key idea: Effective incident response requires getting enriched context to the right people quickly - Evidence: "The alert that you receive can be pretty vague. What if we could enrich it with some more context so that you without having to log into five systems understand what's going on" #### ⏱️ [08:20](https://youtu.be/FPsuYDsHOBg?t=500) — [H1] Institutional knowledge problem in incident response - Key idea: Static runbooks don't scale - need dynamic, queryable systems that learn from incidents - Evidence: "If your incident system still depends on heroics it's already broken. It's about finding those sustainable systems that actually reduce cognitive load during incidents" #### ⏱️ [11:30](https://youtu.be/FPsuYDsHOBg?t=690) — [H2] Orchestration beyond alerting in practice - Key idea: Modern orchestration includes context aggregation, timeline management, and automated escalation decisions - Evidence: "It's easy for an engineer in the middle of the night to kind of decide I'm going to hang on to this for another five minutes because I don't think it's as bad as I think" #### ⏱️ [14:15](https://youtu.be/FPsuYDsHOBg?t=855) — [H3] AI opportunities before and after incidents - Key idea: AI can automate postmortem generation and prevent incidents through better CI/CD pipelines - Evidence: "Getting time to deal with incidents was always a struggle. AI is going to give us opportunities to have something thinking and chewing on some of these problems in the background" #### ⏱️ [16:45](https://youtu.be/FPsuYDsHOBg?t=1005) — [H1] Building trust in automation under pressure - Key idea: Engineers won't rely on automation during incidents unless they've seen it behave predictably beforehand - Evidence: "Our engineers won't rely on automation during a subzero unless they've seen it behave predictably before and most teams don't have an automation problem they have that confidence problem" #### ⏱️ [19:30](https://youtu.be/FPsuYDsHOBg?t=1170) — [H2] Shadow canaries for proactive incident prevention - Key idea: Running changes against production traffic without user impact shifts incident detection left - Evidence: "Shadow canaries running changes against real production traffic without impacting user seeing how that goes it shifts incident detection left so you're catching issues before they become incidents" #### ⏱️ [22:10](https://youtu.be/FPsuYDsHOBg?t=1330) — [H3] Understanding AI tools before automation - Key idea: Teams must understand how AI tools work internally before deploying them in production incident response - Evidence: "You have to understand kind of how they work what they're doing under the hood like learning that the context is going to get compacted" #### ⏱️ [25:40](https://youtu.be/FPsuYDsHOBg?t=1540) — [H1] Human cost of poor incident management - Key idea: Burnout from constant firefighting is the primary cost, with time off valued more than monetary compensation - Evidence: "Most people, if they had to choose between getting time off and getting more money in compensation for being on call, they all wanted the time off" #### ⏱️ [28:20](https://youtu.be/FPsuYDsHOBg?t=1700) — [H2] Breaking the reactive firefighting cycle - Key idea: Teams must stop the line and invest time in fixing root causes rather than just treating symptoms - Evidence: "Sometimes you have to stop the line, pump the brakes, say 'Okay, we have to get out in front of this. We have to actually change something now'" #### ⏱️ [31:55](https://youtu.be/FPsuYDsHOBg?t=1915) — [H1] What not to automate in incident response - Key idea: Never automate irreversible actions, ambiguous scenarios, or cross-service cascading changes - Evidence: "Speed without judgment is dangerous. Speed in incidents is not about doing things faster, it's about making fewer wrong decisions early" #### ⏱️ [35:30](https://youtu.be/FPsuYDsHOBg?t=2130) — [H2] Addressing skill atrophy through game days - Key idea: Regular practice through game days and failure injection prevents incident response skills from degrading - Evidence: "You get good at what you practice regularly and if you're not practicing how to respond to incidents it's going to get rusty" #### ⏱️ [39:45](https://youtu.be/FPsuYDsHOBg?t=2385) — [H3] Future of incident response teams in 2-3 years - Key idea: Best teams will shift from reactive response to proactive failure containment with AI-assisted decision making - Evidence: "Detection and mitigation will happen before a human is even paged through things like activating shadow canaries proactive validation or systems that can contain failures early" ## 📚 References - [LeadDev Panel Discussion](https://www.youtube.com/watch?v=FPsuYDsHOBg) - [Everbridge xMatters Platform](https://www.everbridge.com/products/it-alerting/) - [Mapbox Cloud Platform](https://www.mapbox.com/) ## 🗣️ Paraphrase - Effective incident response prioritizes organization and communication over immediate technical fixes, preventing engineers from "rabbit holing" into debugging while customers remain uninformed - Trust in automated incident response systems must be built progressively through low-stakes exposure and shadow testing, as engineers will bypass automation during critical moments if they haven't seen it work reliably - The future of incident management focuses on preventing incidents through proactive systems like shadow canaries and circuit breakers, rather than just responding faster when things break ## 📰 Newsletter **Breaking the Incident Response Firefighting Cycle** Teams are evolving beyond reactive incident management toward AI-assisted orchestration that prevents fires before they spread. Key insights from industry experts reveal that the biggest barrier isn't tooling—it's building trust in automated systems under pressure. **Key Takeaways:** - Start with "decision assistance, not decision replacement" when implementing AI in incident response - Use shadow canaries and game days to build confidence in automated systems during low-stakes scenarios - Encode postmortem learnings into circuit breakers and system improvements rather than static documentation **Practical Next Steps:** - Audit your current alerts for false positives that cause on-call burnout - Implement regular failure injection exercises to prevent skill atrophy - Focus automation on known, reversible actions while keeping humans in the loop for ambiguous scenarios Resources: [Site Reliability Engineering](https://sre.google/), [Incident Response Documentation](https://response.pagerduty.com/), [Chaos Engineering Principles](https://principlesofchaos.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create incident response best practices content, host game day workshops, track MTTR metrics across developer tools - Education: Develop curriculum on AI-assisted incident management, conduct office hours on automation trust-building - Developer experience: Improve incident response documentation, create SDK for incident automation tools - Developer marketing: Campaign on "preventing incidents vs faster response", content strategy around reliability engineering - Community management: Moderate discussions on incident response practices, survey community on automation adoption #### OSPO - Policy & Governance: Establish incident response automation policies, create charter for reliability working group - License Compliance: Review open source incident management tools, ensure compliance in automation frameworks - Security Management: Assess vulnerabilities in AI-assisted incident response, create security attestations for automation - Software Inventory: Maintain SBOM for incident response toolchain, track ownership of reliability components - Developer Support: Provide training on incident automation tools, consult on reliability patterns - Community Engagement: Fund upstream reliability projects, establish agreements with incident management communities - Strategic Alignment: Define KPIs for incident reduction, align OKRs with reliability goals #### Community - Maintainers: Support incident response roadmaps, create governance templates for reliability teams - Contributors: Onboard new contributors to incident management projects, create first-issue programs for reliability tools - Sponsors: Provide transparent budgets for reliability initiatives, publish impact reports on incident reduction - Ecosystem: Coordinate incident response standards across projects, facilitate cross-team reliability sharing #### WordPress - Core: Prioritize reliability improvements, triage performance-related incidents, establish performance budgets - Core Performance: Profile incident response systems, establish metrics for automated recovery - Core AI: Create guardrails for AI-assisted incident response, develop evaluation sets for automation - Hosting: Develop cross-host incident response checklists, implement reliability telemetry - CLI: Expand command coverage for incident management, improve incident response documentation - Meta: Improve .org reliability roadmap, enhance API reliability for incident management systems #### The WPCC - Programs: Create reliability fellowship programs, provide microgrants for incident prevention tools - Projects: Host reliability governance clinics, sponsor incident response documentation sprints - Funding: Establish matched grants for reliability improvements, create milestone payouts for incident reduction #### FAIR - Technical independence: Ensure incident response metadata independence, implement provenance checks for reliability tools - Policies: Maintain funding transparency for reliability initiatives, prevent vendor capture in incident management - Governance artifacts: Create charters for reliability working groups, establish operator agreements for shared incident response ## 🧪 Hypotheses - Teams that practice incident response through game days will have better outcomes during real incidents compared to teams that only learn through production failures - AI-assisted context enrichment in incident alerts will reduce mean time to resolution by providing responders with pre-filtered relevant information - Shadow canary deployments will catch 60-80% of potential incidents before they impact users, reducing overall incident volume - Progressive trust-building in automation (recommendations → guided actions → autonomous responses) will achieve higher adoption rates than immediate full automation ### 📊 Methodology - Study design to test hypothesis 1: Compare incident response metrics (MTTR, escalation rates, repeat incidents) between teams with regular game day practice vs those without - Data sources and analysis for hypothesis 2: A/B test enriched vs standard alerts, measuring time to accurate diagnosis and resolution - Experiments to validate hypothesis 3: Deploy shadow canaries alongside traditional monitoring, track incident prevention vs detection ratios ### ❓ Questions - How do we measure the ROI of incident prevention versus incident response speed improvements? - What are the optimal handoff points between human and automated decision-making during different incident severity levels? - How can organizations maintain incident response expertise as automation reduces hands-on experience opportunities? ### 🔮 Future Research - Long-term effects of AI-assisted incident response on engineer skill development and career growth - Cross-industry patterns in incident automation adoption and trust-building strategies - Integration challenges between legacy incident response processes and modern AI-powered orchestration tools ### 💡 Implications - Practice: Teams need structured game days and failure injection to maintain incident response capabilities as automation increases - Policy: Organizations should establish clear boundaries around what incident actions can be automated without human approval - Security: AI-assisted incident response introduces new attack vectors and requires additional security controls and monitoring - Education: Engineering education must include training on AI tool limitations and failure modes, not just capabilities - Funding: Budget allocation should balance incident prevention technology with maintaining human expertise and practice opportunities ### 🧠 Learning Styles Assessment Reading level estimate: 12th Grade (Flesch-Kincaid Grade Level) #### 👀 Visual learners - Present: Discussion format with clear speaker identification and role context - Improve: Incident response flow diagrams, automation decision trees, visual timeline of incident progression #### 🔊 Aural learners - Present: Full audio discussion with multiple expert perspectives and real-world examples - Improve: 5-minute audio summary highlighting key takeaways, chapter markers for easy navigation #### 🗣️ Verbal learners - Present: Rich analogies (rabbit holing, hot potato problem), technical term definitions, story-driven examples - Improve: Glossary of incident response terms, comparison tables between traditional vs AI-assisted approaches #### 🧩 Kinesthetic learners - Present: Practical examples of game days, hands-on automation implementation stories - Improve: Step-by-step incident response checklists, interactive game day planning worksheets #### 🧠 Logical learners - Present: Clear decision frameworks for automation boundaries, logical progression from problem to solution - Improve: Decision matrices for automation vs human judgment, incident severity classification systems #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format, community Q&A integration, team-based solutions - Improve: Discussion prompts for team retrospectives, collaborative game day planning guides #### 🧘‍♀️ Solitary learners - Present: Individual reflection opportunities through expert insights and personal experience sharing - Improve: Self-assessment questionnaires for incident response --- ## Full Transcript [00:12] Welcome everyone. My name is Nesla. I am [00:16] really excited to be uh hosting this [00:18] panel today. This is a um panel called [00:23] your blueprint for automated incident [00:25] recovery uh which is in done in [00:27] partnership with lead dev and everbridge [00:31] x matters. So the discussion is going to [00:33] last about 45 minutes after which both [00:36] myself and all the panelists are going [00:38] to head over to the lead dev slack. [00:40] We'll post the um link if you don't have [00:43] that already and you can ask more [00:46] questions there uh to the panel in the [00:48] Q&A channel. We might also have time to [00:51] answer a few questions live. So please [00:53] do submit those through the Q&A on Zoom [00:56] and I'll be actively monitoring that to [00:58] uh make sure we can answer them if if [01:01] there's time. [01:02] And first I want to get started with [01:04] some introductions. So like I said my [01:06] name is Neesla. I'm joined by three [01:08] amazing panelists here. A little bit [01:10] about myself. I'm a director of [01:11] engineering at a startup called Align [01:15] based in the US, although I'm based in [01:16] Berlin, Germany. I'm also a leadership [01:19] coach and I've been working with Lee [01:20] Devub for a few few years. I love doing [01:22] these panels because I always learn a [01:24] lot. So, I'll pass it over to Ren to [01:27] introduce themselves first. [01:30] Hey everyone, I'm Ren. I'm the tech lead [01:33] of the cloud platform team at Mapbox and [01:36] I've been dealing with incidents for [01:39] nearly 20 years now since I was a [01:42] one-person ops team at my first startup [01:44] tech job in New York where I was, you [01:46] know, the one person with badges to the [01:49] data center. Uh, so I've seen kind of [01:51] the good, the bad, and the ugly over the [01:53] years and I'm really looking forward to [01:54] this panel talking about how we can get [01:56] some more of the the good in there. [02:01] Thanks Ren. Malikica. [02:05] >> Uh hi you all. Uh I'm Malika. Um I am a [02:10] engineering leader at Zorkdoc. Uh and [02:12] most recently uh I I've also been at [02:15] Netflix. Uh previously at Walmart and [02:17] Twitter and I've spent a lot of time [02:20] building and operating distributed [02:21] systems in general particularly in [02:23] environments where reliability, velocity [02:26] and stability are constantly in tension. [02:28] And a big part of my focus has been [02:31] thinking about incident response not [02:32] just as a process problem but as a [02:35] systems and decision making layer um uh [02:39] under you know all the uncertaintity [02:41] that is there and how do we get the [02:43] right context to the right people at the [02:45] right time. So most recently I've been [02:47] exploring a lot about how AI can shift [02:49] that from static runbooks to dynamic [02:52] context aware systems. So I'm very [02:54] excited to like dig deeper into uh those [02:56] constructs with this group and um you [02:59] know for this conversation. [03:02] >> Thank you. And last but not least, Dan. [03:05] >> Hi there. I'm I'm Dan. I'm a product [03:07] manager from Everbridge X Matters. Uh so [03:09] I used to be on the tech side. So I used [03:11] to be an engineer and engineering team [03:13] lead uh back in the day like six or [03:15] seven years ago. But for the last six or [03:16] seven I've been working uh on kind of [03:18] solutions that are related to incident [03:20] management. So, I've had an opportunity [03:21] to talk to lots of customers about uh [03:23] their problems, lots of folks about what [03:24] they're seeing. So, excited to kind of [03:26] to chat about that with you all today. [03:29] >> And I I will say I'm really excited to [03:31] have a product manager, which I haven't [03:33] had an elite dev panel before. And I [03:36] think it's actually really like [03:37] pertinent to the moment because our [03:39] roles are really starting to converge. A [03:41] lot is happening and PMs are getting [03:43] more technical. Engineers have to think [03:45] more like PMs. So, it's great. things [03:49] only get blurriier, right? [03:50] >> Exactly. So, we're talking about [03:52] incident response today. How teams move [03:55] from those first chaotic minutes of an [03:58] incident towards a more orchestrated and [04:00] effective approach. So, [04:03] systems that we're working with are [04:04] actually more complex than ever. As we [04:06] all know, it's making things harder to [04:08] diagnose. It's making the ownership more [04:10] unclear. And at the same time, AI and [04:13] automation are becoming a a newer and [04:16] bigger part of the bigger part of the [04:18] process. But it does raise questions [04:20] around trust and how often we need human [04:23] judgment. So we're going to explore all [04:25] this today. Uh we're going to where [04:27] orchestration um can be automated, where [04:31] it shouldn't be automated, what this [04:32] looks like in practice. Let's dig in. [04:34] And I want to start with Ren. So, can [04:37] you tell us what actually happens in the [04:39] first 20 minutes of an incident and [04:42] where does that typically break down in [04:44] your experience? [04:47] >> Yeah, I mean the the first thing is well [04:49] figuring out is this actually an [04:51] incident? You know, you've you've gotten [04:53] paged. It's maybe the workday, maybe the [04:56] middle of the night, who knows? You're [04:57] you're figuring out all right is this is [04:59] this actually a problem? And then kind [05:01] of kind of what you need to start doing [05:03] after that is figuring out what what is [05:04] the actual impact. Um and then [05:07] organization is really what should be [05:10] happening first. Um getting [05:12] communication in in gear because as a [05:14] customer there's nothing that's more [05:16] frustrating than going to, you know, the [05:18] status page when you know that there's [05:20] something wrong and it just says [05:21] everything's green. All of the check [05:23] marks, everything is happy. You want to [05:26] make sure that we're we're communicating [05:28] to our customers as soon as possible. [05:31] And since one person cannot do all of [05:34] the things all the time, that means [05:36] assigning different roles to people. You [05:37] know, get your incident commander or [05:39] whatever you're talking calling that [05:40] role in place, dividing up the work. I [05:45] think the place where I've seen that go [05:47] wrong most in practice is you get [05:49] engineers who they want to figure out [05:51] what the technical problem is and they [05:53] start rabbit holeing and forget the [05:55] people side of things, the customer side [05:57] of things, the business side of things [05:59] like are we bleeding money that we need [06:01] to you know get that bandated up before [06:04] we you know dig into the technical [06:07] problems. So keeping keeping focus on, [06:10] you know, the business side, the [06:11] customer side, what's actually important [06:14] while trying to organize everything else [06:16] that's going on in an increasingly [06:18] complex system? [06:22] Yeah. And speaking of people in complex [06:25] systems, Dan, how do you figure out when [06:29] an incident is happening? Uh who owns [06:32] the incident? [06:34] How like how the work is going to get [06:35] distributed? How do you avoid this like [06:37] hot potato problem or like oh it's not [06:39] me get it off my plate? [06:41] >> For sure. A lot of this is about kind of [06:43] finding the right ways to uh prep for [06:46] incidents as well, right? So how do we [06:47] make sure that we have the correct that [06:48] we're getting the correct signals? How [06:50] do we orchestrate those signals across [06:52] that we make sure that they get to the [06:53] right people and that we can also raise [06:55] the right context to to those folks to [06:57] know is this something that I need to [06:58] deal with today and now. Um, so, uh, [07:01] usually it's about kind of meeting [07:02] people within the tools that that make [07:04] the most sense for them, but also [07:05] finding ways to to build build some kind [07:08] of process over top as well, so that [07:10] there's some consistency between teams [07:11] because I think we all have teams that [07:13] want to use different tools. Um, so how [07:14] do we make sure that those signals that [07:16] are kind of specific to one team um, can [07:19] be there as context for the for the [07:20] larger group to be able to make those [07:21] calls early as well. And how do you make [07:23] it so that uh, they they can take that [07:26] action quickly and feel confident about [07:28] what to do next. Um, so a lot of it is [07:29] about like the alert that you receive [07:31] can be pretty vague. Um, what if we [07:32] could enrich it with some more context [07:34] so that you without having to like log [07:35] into five systems understand what's [07:37] going on and what your what your part of [07:39] the world looks like and maybe what some [07:40] of the other parts of the world look [07:42] like. How do we make it easy for you to [07:43] be like that's not enough information. [07:45] Where do I go next? Um, and of course, [07:47] you know, the more that we can do some [07:48] of these pieces, the more we can [07:49] automate behind the scenes and take some [07:51] low-risk actions on behalf of our teams, [07:54] um, that they would likely be doing if [07:55] they were being woken up in the middle [07:57] of the night or, you know, in focus time [07:59] during the day and their flow. So, so a [08:01] lot of it is about kind of like slowly [08:02] building towards this this this new [08:04] model of the auto remediation kind of [08:06] side by building up that context, making [08:08] confident decisions and making those uh [08:11] decisions actions available to folks as [08:13] quickly as possible so they can kind of [08:15] take the next step. So hopefully by [08:17] doing more and more of that and by by [08:19] learning from the incidents we have, [08:20] we're able to kind of find find the [08:22] right people faster um identify what was [08:24] what information was missing and make [08:26] better decisions earlier. [08:29] Yeah, that's great. [08:31] I think you're touching on like how [08:32] important process is like okay like [08:35] teams are all operating independently [08:38] humans are inherently different like [08:39] let's start there right so people are [08:41] going to think teams are different [08:42] dynamics but then the tooling might be [08:43] different from team to team [08:46] >> um the like what you might deem severe [08:48] versus not severe that even is [08:50] subjective [08:52] >> um and it you're talking about building [08:54] a system where people can cleanly like [08:56] interface with one another and it's [08:58] legible and [08:59] >> exactly [09:00] >> yeah and I'm curious uh from you [09:02] Malikica like as that's happening and [09:04] like information is being tossed around [09:06] how do you make sure that the [09:08] institutional knowledge that needs to be [09:10] retained by certain people actually [09:12] stays there over time it actually gets [09:14] to the right people maybe not in the [09:16] moment of the incident but also before [09:18] and after. [09:20] >> Yeah. Uh that's a great question and I [09:23] uh I I do think a lot of that incident [09:25] response today still relies on exactly [09:28] what you uh I love the word that you use [09:30] that institutional knowledge right uh [09:33] the one engineer who just knows where to [09:35] look um and the I think the mistake the [09:40] teams make is trying to flatten that [09:42] into static runbooks that doesn't scale [09:44] to a large extent if your incident [09:46] system still depends on heroics it's [09:49] already broken It's about finding those [09:52] sustainable systems that actually reduce [09:55] cognitive load during incidents and not [09:57] necessarily increase it and reaching out [10:00] to multiple humans does increase it. Um [10:03] what we have seen work better is [10:05] capturing decision trees not just steps [10:08] uh linking signals to hypothesis and [10:10] actions continuously updating this from [10:13] postmotms and increasingly AI can help [10:16] here not just by replacing expertise but [10:19] by retrieving and contextualizing it in [10:21] the moment. Um for example even how we [10:25] think about postmortems right if your [10:27] postmortems don't change the system [10:29] they're just storytelling [10:31] um the bar should be what did we encode [10:33] back into the system so this class of [10:35] issue gets easier next time um and not [10:39] just becomes uh you know buried in those [10:41] postmortm logs right another example is [10:44] circuit breakers they are underrated um [10:47] right if we learn something from a [10:49] subzero can we bring it back to you know [10:52] maybe editing or calibrating our circuit [10:55] breakers uh not just as a resilience [10:57] pattern but as an incident containment [11:00] strategy itself right so I do feel like [11:02] good incident systems don't just detect [11:04] failures they limit that blast radius [11:07] automatically so that shift is really [11:09] from documentation to living queriable [11:12] system knowledge and that is how we can [11:15] kind of take all of that institutional [11:17] knowledge and slowly encoded part of the [11:20] system um uh so that uh it's kind of a [11:24] default how do we want to go about um [11:27] managing when it happens again. Um yeah. [11:33] >> Yeah, I totally agree. I think that so [11:35] much of it is about how do you how do [11:36] you learn and how do you kind of adapt [11:37] to those things because there's always [11:39] going to be new new like our systems are [11:41] con constantly changing. Um the world [11:43] around us changes. I feel like you know [11:44] we can control what we can control. We [11:46] can't control what our partners and our [11:47] vendors are doing. So being able to [11:49] adapt and being able to improve is just [11:51] really important with these kinds of [11:52] things. [11:56] >> Yeah. It's almost like, you know, in the [11:58] before times you would do postmortem [12:00] and, you know, you might lag. There [12:02] might be a long lag to actually get to [12:04] all the to-do items to create that [12:07] institutional knowledge to get people to [12:09] even read the postmortem document. But [12:11] now it's like we have less of an excuse, [12:14] right? like the postmortem should be [12:16] sent into whatever ticketing system that [12:18] you have. It should be queryable, right, [12:20] in a way by others to in order to learn. [12:24] Um, so there's a lot more um power that [12:27] we can unlock from this documentation, [12:29] it sounds like. And another thing I'm [12:32] curious about is what does orchestration [12:36] look like in practice beyond just [12:39] alerting the next person nowadays? like [12:41] what is what is the actual like flow [12:43] that we need to care about? [12:45] >> For sure. Yeah. So, I mean, I think I've [12:46] kind of I touched on some of those [12:48] pieces in the last one because I I like [12:50] to get ahead, but um the a lot of it is [12:52] about kind of like the context. Um so, I [12:54] think, you know, within a lot of [12:55] alerting systems that we we we use [12:57] today, um receiving that kind of initial [13:00] information usually is pretty thin. Like [13:02] it has the context of like the original [13:04] signal, but doesn't kind of pull [13:05] anything else in. So part of that [13:06] orchestration is bringing that right [13:08] information and what those next steps [13:09] might be to to users directly to to [13:12] people that are responding. Um and then [13:14] from there it is I think Malikica you [13:16] kind of touched on just the fact that [13:18] like how do we how do we prepare for [13:20] what the next step is and how do we like [13:21] what are what are some of these circuit [13:22] breakers sometimes it it can also be to [13:25] like make quick decisions about when to [13:27] notify more folks to get more context [13:29] and to to I think NA you mentioned kind [13:31] of the stakeholder comms. how do we get [13:33] some folks that are there for that [13:35] purpose as well? Um so being able to [13:37] decide when those decisions have to be [13:39] made. Um being able to to make that [13:41] decision for the user sometimes like oh [13:42] it's been 10 minutes we know that things [13:43] are bad let's move faster. Um because [13:45] the you know it's it's easy for for an [13:47] engineer in the middle of the night to [13:48] kind of decide I'm going to hang on to [13:49] this for another five minutes because I [13:51] don't think it's as bad as I as I think. [13:53] Um so a lot of it is about like setting [13:54] setting those timelines um and [13:56] connecting all the different tools that [13:57] you need to in order to have that kind [13:59] of response and connecting the people. [14:00] Um so being prepared uh with with kind [14:02] of the the right escalations and and [14:05] getting that context out together. Um [14:07] over time as well it is how do you [14:09] automate those pieces so that users [14:10] don't have to make the choices as often. [14:12] Um and of course that's that kind of [14:14] gets to those points around we have more [14:16] and more tools at our disposal to make [14:18] decisions so to speak with AI. So how do [14:20] we what's the right line with some of [14:22] those pieces is always interesting. [14:26] I think there can be some really [14:28] interesting opportunities for [14:30] orchestration and automation before and [14:32] after incidents as well. Like during is [14:35] great like that's you know fantastic but [14:37] I think there's you know before like can [14:39] we stop incidents from happening? Can we [14:42] do better things with our CI pipelines, [14:44] our test pipelines, making sure that [14:47] issues get caught before they go into [14:49] production or afterwards putting [14:52] together you know the postmortem [14:54] document. I think it's one failure mode [14:56] I've seen is the details get lost of who [14:59] did what when and what graph were you [15:02] looking at and what made you decide to [15:04] do that thing versus that other thing. [15:06] And I think this is something where you [15:08] know if you can get you know Gemini to [15:10] take the transcript of the huddle that [15:12] your engineers were in and pull out [15:14] those details in a more automated [15:16] fashion so that information isn't lost [15:18] and then you can you know codify that [15:20] into you know future run books, future [15:23] documentation, future automation, that [15:25] sort of thing. [15:28] >> Yeah, I think we live in an exciting [15:29] time where I feel like get having time [15:31] to deal with incidents was always a [15:33] struggle. So that meant that everyone [15:34] was kind of firefighting because that [15:36] was always kind of the um even though [15:37] it's the highest priority thing when [15:39] you're in an incident, it's the lowest [15:40] priority thing to think about when [15:41] you're not. It's, you know, how are we [15:43] delivering value to customers and not [15:44] worrying about that stuff as much. Um it [15:46] seems like AI is going to give us a lot [15:47] of opportunities to have something [15:49] thinking and chewing on some of these [15:50] problems and and building up that [15:52] context. Um so hopefully in the future, [15:54] um even though we may not still have [15:56] more and more time for those things, [15:57] we'll have we'll have someone working on [15:59] in the background, which is great. [16:02] Well, let's let's talk about the a word [16:06] and the two a a words AI and automation. [16:09] So, something that Ren said earlier was [16:12] uh uh like a failure mode is when an [16:15] engineer tries to go too deep, right? [16:17] And that's because I think as engineers, [16:20] we're so like curious and we want to [16:22] understand this thing, but it might not [16:23] be like the it might not be what the [16:26] situation warrants. So maybe now we have [16:28] the ability to to ask something to [16:31] automate like the next step for us. How [16:35] how can we actually trust all this [16:36] automation that we're building into [16:38] these systems? And I'll kick it off with [16:40] you Malikica, but I'm sure everybody has [16:42] some ideas here. [16:45] >> Yeah. Um I think the limiting factor [16:49] isn't tooling, right? It's actually [16:51] trust under pressure for our engineers. [16:54] Um our engineers won't rely on [16:56] automation during a subzero unless [16:58] they've seen it behave predict [17:01] predictably before and that's to uh you [17:04] know what Naja you uh alluded uh Dan and [17:07] Ren pointed out as well and most teams [17:10] don't have an automation problem they [17:12] have that confidence problem where if [17:14] our engineers don't trust the system [17:16] they'll bypass it at the exact moment it [17:19] actually matters. M [17:21] >> um so you know like there are a few ways [17:24] to build that trust proactively right [17:26] like start with recommendations not [17:28] actions have more human in the loop at [17:31] least in the uh more initial iterations [17:34] make systems explain their reasonings [17:37] not just outputs um and then invest very [17:40] heavily in guardrails and blast radius [17:43] control just so that you know incidents [17:45] or like uh metrics degradations from our [17:49] upstream downstream same customers don't [17:51] start affecting us and I think most [17:54] importantly teams need to see these [17:56] systems work during low stakes incidents [17:58] first or even simple patterns like you [18:01] know one pattern I found really [18:03] effective is shadow canaries running [18:05] changes against real production traffic [18:08] without impacting user seeing how that [18:10] goes it shifts incident detection left [18:13] right so you're catching issues before [18:14] they become incidents it's one of the [18:17] cleanest cheapest ways to actually [18:18] reduce subzer without slowing teams [18:21] down. So investing in uh uh uh processes [18:27] systems which can actually build trust [18:30] technically through the systems and it [18:32] also I think creates a feedback loop [18:34] where automation has real data to learn [18:36] from. It's not just synthetic tests [18:39] running in the background. So I do feel [18:41] like trust isn't designed right it's [18:43] earned through repeated safe exposure [18:46] and our engineers need to see that in [18:48] action. uh before something really bad [18:51] happens and it's going to happen. So [18:53] it's you know like can we accept the [18:55] fact and like prepare for that future [19:01] >> and I'll say making sure you understand [19:04] what the tools are doing and how they [19:06] work like back when we were writing you [19:08] know bash scripts for everything you [19:10] have to know how to do something [19:11] manually before you can automate it [19:13] before you can write a script for it and [19:15] kind of a similar principle applies with [19:17] AI tools like you have to understand [19:19] kind of how they work what they're doing [19:21] under the hood like learning that oh you [19:24] know the context is going to get [19:26] compacted. So if you have you know in [19:27] your claude MD file don't run [19:30] destructive commands against prod that [19:32] might get lost due to how the tool works [19:34] and if you don't understand that you're [19:36] going to be surprised when claude goes [19:38] and you know deletes some production [19:39] infrastructure that you weren't [19:41] expecting whereas if you know like okay [19:43] here's how hooks work in you know the [19:46] the life cycle of the tool you're going [19:48] to be better prepared. So making sure [19:50] that we're not just like jumping on the [19:52] hype train of all of these shiny new [19:54] tools without really understanding what [19:56] they're doing and how to use them [19:58] effectively. [20:01] >> Yeah, human trust is kind of funny. Um [20:03] because I do feel like we we jump to [20:05] trust pretty quick in certain cases when [20:06] you show us that you so I feel like with [20:08] AI we're kind of in that riding the wave [20:10] of like you know go I think I every [20:12] every week I feel like I hear different [20:14] stories of like it's amazing and it does [20:16] everything and I never have to worry and [20:17] then on the other side actually maybe [20:19] not maybe we need to pump the brakes. [20:21] Um, but I I think there's also just so [20:23] much room like uh we talked about this a [20:25] lot um with with with with X matters but [20:28] um just uh what how do you kind of like [20:32] slowly adapt and I think we Mala you [20:34] touched on this really well like human [20:36] human guidance at first right like we're [20:38] going to give you more information about [20:39] what you should think about doing this [20:41] is what what AI thinks we're going to do [20:43] or what what the automations suggest [20:45] based on just rules-based automation and [20:47] that kind of learning. Um like there's [20:49] so much that you can get out of this in [20:50] the incident management side today that [20:52] we probably like you're just scratching [20:53] the surface. You don't have to jump [20:55] fully into like Gen AI the uh AI S is [20:59] going to do all this stuff without [21:00] without a human in the loop right away. [21:02] Um so it's like how do you take those [21:03] steps to build towards that future and [21:06] that vision and and build that trust [21:07] slowly instead of kind of jumping to the [21:09] end state first. Definitely lots of [21:11] options for that. [21:15] >> Yeah, I love that. I mean ultimately [21:17] like the what we what we have here is a [21:19] relationship with the tools, the [21:21] technology and relationships can be [21:23] toxic. They can be up and down [21:24] emotionally, right? Or they can feel [21:26] clear and transparent. And um I love [21:30] talking about incident response because [21:31] it really is one of those things that [21:33] touches people, process, and technology [21:36] all at the same time. And uh when you're [21:39] really in the incident, the people [21:41] aspect kind of trumps everything because [21:42] you need to mobilize everyone so quickly [21:44] and and get to that actual resolution [21:47] that is needed in the moment. Um and [21:51] Ren, based on your experience being in [21:54] the trenches, what what do you see as [21:56] this like human cost for poor incident [21:59] management? [22:00] And [22:02] how do you go back to being sustainable [22:05] in terms of how people are feeling? [22:07] Burnout, being able to actually fix [22:09] things. Yeah, go. Please go ahead. Yeah, [22:12] burnout is definitely the biggest thing. [22:15] Um I mean I the size of your on call roa [22:20] obviously matters. Like I've been a [22:22] oneperson ops team, you know, the one [22:24] person on call 24/7 365 not getting a [22:28] break. But even even a a small team [22:31] where you know every other weekend [22:33] you're getting pulled away from you know [22:35] your kids, your family, your hobbies, [22:37] whatever, it's going to take a toll. Um [22:40] I think I I forget where I read this, [22:41] but I feel like I I read something that [22:43] said that most people, you know, if they [22:45] had to choose between like getting time [22:47] off and getting more money in in [22:49] compensation for being on call, they all [22:51] wanted the time off cuz you know, money [22:53] can't give us back, you know, the the [22:56] vacations that we missed out on or [22:57] whatever. So thinking about what are the [23:01] biggest pain points in your particular [23:05] incident response system like for some [23:07] people it might be you know false [23:10] positives. You get these alarms that [23:12] didn't actually mean anything. The blips [23:15] that resolved by the time you got your [23:17] laptop out to go look at them. Or maybe [23:20] it's, you know, feeling like postmortems [23:22] are useless because the action items [23:25] just go into a Jira void and nobody ever [23:27] looks at them again. That's like, why [23:29] are we going through this process if [23:31] nothing ever actually changes? Like, oh [23:33] yes, that it's that SEV one again that [23:36] we did last month and the month before [23:38] that it's the same problem. Um, so make [23:42] sure that teams have the time in your in [23:46] your sprints, in your schedules, [23:48] whatever, to be proactive and to [23:50] actually fix those pain points. Um, [23:53] talking to the engineers who are the [23:55] ones on call and asking them like, hey, [23:57] if you could fix one thing about how we [23:59] do incidents, what would it be? Compile [24:02] those answers and then actually make [24:04] time to fix them. because I feel like [24:05] that's kind of the trap that most teams [24:08] fall into is they get into just this [24:10] treading water really reactive state [24:13] where you just can't ever get ahead of [24:16] everything that is always on fire. So [24:18] sometimes you have to, you know, stop [24:20] the line, pump the brakes, say, "Okay, [24:22] we have to get out in front of this. We [24:24] have to actually change something now." [24:26] But without, you know, support to [24:28] actually let teams do that, you're just [24:30] going to end up with more fire. [24:36] I totally um agree with Ren's point [24:39] here. It's just uh surprising to me how [24:43] many lowhanging fruits there are [24:44] typically uh as part of the process you [24:48] know uh weekly rituals or bi-weekly [24:51] rituals where you know looking at our [24:53] noisy alerts or our triage process right [24:57] are they even going into the right [24:59] buckets are we treating everything the [25:01] wrong bugs as se zeros and se ones um [25:04] and are we tweaking that process [25:06] continually where you know as part of [25:09] our quarterly retros um or any team [25:12] rituals there is a set system where we [25:15] can edit these um ways of how we do our [25:20] on call that is crucial and I feel like [25:23] every incident story um is a powerful [25:26] story exactly because there is a human [25:29] behind it at least so far right like we [25:31] we've been relying so much uh on the [25:34] human in the loop and that's why those [25:36] stories have so much of data and [25:38] knowledge in them. So having a way to [25:40] actually act on them to Ren's point um [25:43] is a very lowhanging fruit uh for uh [25:45] teams to kind of um jump on. [25:50] >> Yeah, I think how do how do you kind of [25:51] get get that buyin from everyone to [25:54] create space and time to work on some of [25:56] this stuff? Right. So, as a as a PM, but [25:59] our organization is is thankfully pretty [26:01] um I I always I was from the engineering [26:04] side, I was always kind of a mature [26:05] organization when you actually had time [26:07] to work on stuff that wasn't just [26:08] product features um where you actually [26:10] had the opportunity to kind of work on [26:12] some of that debt side. Um so, we've [26:14] we've always been able to kind of [26:15] maintain some of those pieces, but um [26:17] hopefully I think also within this [26:19] incident world, the the better tools you [26:21] have to be able to kind of identify look [26:23] at this group that's going through this [26:24] pain. Um, right? Like we can see that [26:26] this person has responded to this many [26:27] things. They're getting you can see the [26:29] the charts on like how where they're [26:30] getting uh they were woken up this many [26:33] times overnight. Like some of those [26:34] types of hopefully the more data that we [26:35] can provide to uh folks like me or folks [26:38] that that might be controlling kind of [26:39] how much time you can spend on this [26:40] stuff, the more you're able to kind of [26:42] empower them with that data as well, [26:44] hopefully there's some some change you [26:45] can make because yeah, the the human [26:46] toll is is significant with uh with [26:49] being being on call. So how how do we [26:52] make that better? Um, and that the other [26:53] part of that is as well is like making [26:55] sure that folks are using the tools [26:56] correctly. Making sure that they've [26:58] defined the devices that that will [26:59] actually wake them up. Um, so that you [27:01] know, you don't have the one person that [27:02] set things up correctly and is getting [27:04] woken up every day because the other [27:05] folks know that they're going to be [27:06] there as the back stop behind them. So, [27:08] how do we make sure that folks are [27:10] encouraged to to kind of do the right [27:11] thing together too, right? [27:14] Not that I think anybody seems to do [27:16] that, but [27:19] >> Yeah, that's great. And you know, it's [27:21] funny, Dan, is I used to be a product [27:23] manager. So, I actually, you know, if I [27:26] put my PM hat on, I think, well, [27:28] everything is a funnel. And when you [27:30] have a funnel problem, you're always [27:32] looking at the constraints in the funnel [27:33] and and you tackle one problem at a [27:35] time. And so, within with incident [27:37] response, [27:38] you know, back to Ren's point, the the [27:40] best way to handle incidents is to not h [27:42] have them. [27:43] >> 100%. [27:44] >> And now we can Yeah. Now we get to now [27:47] we can through queries ask like hey [27:49] these are the incidents we saw in the [27:51] last 3 months um scan my codebase and [27:55] tell me what could you know knock out [27:59] 20% of this right how do we reduce this [28:01] dramatically so we actually get this [28:04] opportunity now that I think if you [28:06] didn't have time before it's become much [28:07] easier if you didn't have that like [28:09] maybe PM mindset [28:11] >> cultivated for so long in order to do [28:13] this type of analysis Right. And so in [28:15] this way, we're all blending quite a [28:17] bit. [28:18] >> Every so often, I hear some great [28:20] stories of folks where it's kind of like [28:21] they talk about their teams just being [28:23] kind of um so focused on firefighting. [28:26] Um and it's just amazing kind of the [28:27] next tier of problems and and just the [28:29] more time that you have to spend on [28:30] other things that you can get to if you [28:32] can deal with these things well. You can [28:34] orchestrate these issues so they don't [28:35] become major fires that last hours and [28:38] hours or days. So it's it is pretty [28:40] amazing kind of the difference that it [28:41] can make. [28:46] Yeah. And on a similar vein, you know, [28:50] there maybe are are some things that [28:52] maybe some things we should automate, [28:54] some things we shouldn't. In y'all's [28:56] opinions, uh, Mal Malik, we'll start [28:59] with you. Like, what are the things that [29:01] we definitely should not be automating? [29:03] And what's a clear yes to you? [29:07] Yeah, I I feel like one of the costliest [29:10] mistakes I've seen is treating [29:12] automation as inherently good. Um, [29:16] right. Uh, and especially in incident [29:18] response. Um, speed without judgment is [29:22] dangerous and it has a big cost and [29:25] speed comes up a lot and that's why I [29:27] talk about speed and in incidents. It's [29:31] just not about doing things faster, [29:33] right? It's also about making fewer [29:35] wrong decisions early uh so that they [29:38] don't cascade into our decision making [29:40] later. So there are really three areas [29:42] that I'm very cautious about when it [29:44] comes to automating any irreversible [29:47] actions right like any uh data deletion [29:50] or large scale failovers. Uh it could [29:52] also be any ambiguous root cause [29:54] scenarios where signals are still noisy [29:57] just waiting a little bit more before [29:58] they stabilize. any cross service [30:01] cascading actions. This is huge you know [30:03] like if you have uh a big blast radius [30:06] lots of downstream upstream [30:08] implications. So automation should [30:11] handle known reversible well understood [30:13] parts. For anything else it's actually [30:15] okay to watch for a little bit and wait [30:18] for the system to stabilize. Everything [30:20] else should be decision assisted not [30:22] decision replaced. Um and the highest [30:25] leverage automation um actually happens [30:28] before incidents. uh right like Ren [30:31] mentioned this as well things like uh [30:33] can we block bad deploys validating [30:35] configs catching anomalies early and can [30:38] we codify this uh in the system and I [30:41] think it gets easier when fewer [30:43] incidents exist right so creating that [30:45] space where we can go towards that [30:48] northstar where we just have fewer [30:50] incidents um so I feel like that goal [30:53] shouldn't be maximum automation um and [30:56] it's actually maximum safe acceleration [30:58] so teams can unlock that velocity and [31:01] like Dan said we can redirect them to [31:03] more meaningful places either in the [31:05] product or on the inside but being [31:07] really mindful as we do that. Uh curious [31:10] to hear other thoughts as well. [31:16] >> I think making sure that we know what [31:19] our automation is doing and how it is [31:22] behaving like kind of who watches the [31:24] watchers. If you have some nice, you [31:26] know, auto remediation setup, if that [31:28] thing fires, I don't know, once a week, [31:30] once a month, it's probably fine. If [31:33] that's going off 20 times a day, you [31:35] probably want to know what is being [31:37] automatically remediated that often. But [31:40] too often I've seen teams just, you [31:42] know, get so excited and set up all the [31:44] automation and then forget about it and [31:46] it goes into another void like the Jira [31:48] void where maybe it's doing the right [31:51] thing, maybe it's not, but nobody's [31:52] really paying that much attention to it [31:55] anymore and it can become too easily [31:57] another source of toil. [32:03] >> Definitely there. So there's also some [32:05] interesting stuff here with kind of I [32:06] think we we we touched on the kind of [32:08] the communication aspect pretty early. [32:10] Um and sometimes with automation, you [32:12] know, and and some of those pieces uh [32:14] for internal columns, it can be fine for [32:16] those pieces to be automated, but [32:18] sometimes with external audiences, uh [32:20] you know, we do want to be able to have [32:21] some some humanized just to make sure [32:22] we're not saying the wrong thing. So I [32:24] think those things will get better over [32:25] time, but for now at least, I think a [32:27] lot of the comm stuff is also still good [32:29] to at least have a bit of guidance human [32:31] in the loop stuff there. Um the one [32:34] thing that I I find fascinating with [32:35] this stuff is I feel like things have [32:36] shifted so much in six months even with [32:38] some of these pieces. Um uh so I I like [32:41] I I really appreciate Mala that kind of [32:43] like it's not just speed but doing the [32:46] right thing and not making the wrong [32:47] decisions that pen in. Um so I think [32:49] right now with automation a lot of it is [32:51] about doing those those types of actions [32:53] that um [32:55] would be low risk. uh right like so so [32:58] uh in complex systems that that does get [33:01] harder because you don't necessarily [33:02] know how your change might impact [33:04] another change and the dependencies [33:06] between things get harder. So maybe roll [33:07] backs aren't safe all the time. But if [33:09] there are places where you can do safe [33:11] roll backs to previous versions the more [33:13] mature kind of our infrastructure is [33:14] with this it also makes those pieces [33:15] easy. I'm like we'll talk to customers [33:17] where you know it's it's a [33:20] a lot easier to roll back or fail over [33:23] in some of those pieces and that's [33:24] great. then you can then you can really [33:25] like make use of automation in a lot of [33:27] interesting ways. Um but but yeah, for [33:29] for kind of crustier or simpler uh uh [33:33] infrastructure, it can be really [33:34] challenging to automate those pieces and [33:36] that's okay. It just takes time for us [33:37] to kind of understand where we're at and [33:38] slowly grow and build the evidence to [33:40] say, hey, we should make this better. So [33:43] um yeah, but I think in six in a few [33:45] months it might change as far as what we [33:47] think our tools are able to do. So it's [33:49] it's an interesting time. [33:51] >> I'm sure that it will. [33:54] Um, we have a question from the [33:56] audience. I want to get to that. This is [33:58] from Colette and anyone can answer. [34:01] There's some scientific research that [34:03] points to the fact that when teams don't [34:05] have regular practice at incidents, it [34:07] can lead to skill atrophy for when in [34:11] incidents for when insulin do actually [34:14] happen. With the rise of automation and [34:16] a push to have less incidents, how [34:17] should teams and leaders address this [34:19] atrophy? [34:23] I think game days is a big one like yeah [34:27] it's very true I mean that that was kind [34:28] of the impetus behind like the original [34:31] you know how what was what was that the [34:33] talk like nine deploys per day 12 [34:36] deploys per day you get good at what you [34:38] practice regularly and yeah if you're [34:40] not practicing how to respond to [34:42] incidents it's it's going to get rusty [34:44] so yeah game days chaos monkey whatever [34:47] you want to do within your organization [34:50] and depending on kind of the size and [34:52] structure of your org. You might want to [34:53] handle this differently. Um, you know, [34:56] have have teams set up different puzzles [34:58] for each other. Have people jump on a [35:00] Zoom and watch troubleshooting. Um, [35:03] we've done things on my team where [35:05] somebody, you know, the one person who [35:08] who knows that one system and gets woken [35:11] up all the time because it's it's in [35:13] their head. Great. Next time we get paid [35:16] for that system, let's everyone hop on a [35:19] call and watch and just do, you know, [35:21] whether it's a real run through or take [35:24] take us through this thing you did last [35:26] time. You know, remember what you did [35:27] did a couple months ago, but making sure [35:29] that you're taking advantage of all the [35:32] different resources that you have on [35:33] your team. And I think, you know, [35:35] probably this is something where AI [35:37] tools can help come up with game day [35:39] scenarios. You know, look at our [35:41] incidents over the past 6 to 12 months. [35:44] you know what happened the most, what [35:46] should we be practicing. So, it's it's [35:48] going to be interesting to see where [35:49] that goes. [35:53] >> Yeah, totally. Uh big plus one to game [35:56] days. Um [35:58] uh and uh it's a really good question. [36:00] Uh like avoiding incidents entirely can [36:03] make teams worse at handling them, [36:05] right? So, uh I think the question is [36:07] very nuanced, but the fix isn't more [36:09] outages, right? It's more intentional [36:11] practice and that's where uh heavy plus [36:15] one to game days it could even be [36:16] failure injection premotms replaying [36:19] past incidents so engineers can build [36:22] that muzzle in a much more safe [36:24] environment and I think most importantly [36:26] not also fully automating the thinking [36:28] behind how we solve these problems and [36:30] that's where no matter how uh much of AI [36:33] or agentic your systems become uh on how [36:36] you handle this keeping that human in [36:39] the loop for diag agnosis and decision [36:41] making even in the future so that um a [36:44] team stay sharp I think is still [36:46] valuable so it's probably more [36:48] deliberate exposure to failure modes [36:50] right that's the way I think about it [36:54] >> yeah for sure and [36:56] >> even in your game days like what happens [36:58] when the tooling doesn't work I mean [37:00] cloud had an outage a few days ago right [37:02] and then everyone on the internet was [37:03] like what do we do now so it's important [37:06] to test the different scenarios and Um, [37:09] a bit more on automation from the [37:11] audience. Someone asked, "What are your [37:13] takes on automated postmortems which we [37:16] mentioned earlier? [37:20] I think there kind of is is some [37:21] similarity to this in the previous one [37:23] um where I think you can automate a lot [37:25] of the um the the effort of pulling [37:28] together information, deciding like and [37:30] suggesting what root causes were in some [37:31] of those pieces. But for now at least, I [37:33] think it is it's still probably [37:34] important for humans to really process [37:36] that information and do something about [37:37] it or else or else it probably doesn't [37:39] benefit anybody really. Um but I think [37:41] over time again like the the more that [37:43] we have kind of um tools that will then [37:45] process those and then kind of suggest [37:47] changes the our rate of change might go [37:49] up in the future. Um so those things may [37:51] may change but at least I I yeah I think [37:54] there's still a lot of reason to keep [37:56] humans in the loop on all these types of [37:57] pieces. [37:59] >> Definitely. I think there's post-mortems [38:02] that tend to focus on more the technical [38:04] side and then there's postmortems that [38:06] kind of dig deeper and go more into the [38:08] like systemic side, the human factors [38:10] side. And that's where you you have to [38:12] have the humans like [38:14] tools are not going to be able to answer [38:17] why a human took a particular action. [38:20] Like you you've got to talk to talk to [38:21] the people about that. like what what [38:23] were you thinking when not in a you know [38:25] obviously you know blameless postmortems [38:27] but just like what were you thinking you [38:28] know what made you go look at this [38:30] signal instead of that signal and yeah [38:33] you know we're not going to be able to [38:35] automate that sort of thing away [38:41] some interesting stuff there too I feel [38:42] like the people element there is also [38:44] the business element and that's probably [38:46] hard for for our tools to be able to [38:48] pull that information right now um so [38:49] maybe I'll change um but really being [38:51] able to understand kind of that full [38:52] scope of impact is is still still [38:55] valuable. [38:58] >> Totally. [39:00] All right. If anyone in the audience has [39:02] any more questions, please write them [39:03] in. Otherwise, I have um another [39:06] question for the crew here. So, this is [39:09] for everybody. [39:11] in two to three years [39:14] after this, you know, up and down with [39:17] emotional roller coaster that we're all [39:19] on with AI. Although I don't know if [39:21] there's going to be an after or we're [39:22] just going to be in this indefinitely, [39:24] but once things settle, where do what do [39:26] you think the best incident response [39:28] teams will do differently than uh they [39:30] did will do differently than they do [39:32] today? [39:37] >> Anybody want to go first? Okay. I'm I'm [39:39] not on mute so I can um so the I think [39:42] that I mean it's going to look entirely [39:44] different probably for the for the most [39:45] part. Um we will probably trust the [39:48] tools a lot more um because they we will [39:50] understand better how to give them the [39:51] correct guard rails and the correct [39:52] context. Um so the the hope I think is [39:56] that we have uh fewer humans having to [39:59] be impacted by things and detect these [40:01] things faster so they don't become [40:02] business impacting as soon as they do [40:05] today. Um, now the I think you you [40:09] brought up a good point that what [40:10] happens when the tools go down. Um, so [40:11] you know, we're still going to have to [40:12] be able to understand what how how can a [40:15] human interject in these things and [40:16] still take actions as well. We're going [40:18] to or or else like you know, how many [40:20] how many tools down do you have to worry [40:22] about in order to make sure that you [40:23] have redundancy with something like this [40:24] that can be so critical. Um, so I think [40:26] you know the the talk the topic of like [40:28] the uh game days staying staying kind of [40:32] active with this that's only going to be [40:34] more important at the rate of change [40:35] we're going to move forward. So you have [40:36] to be you have to be ready and I think a [40:39] lot of us is going to be this is going [40:40] to be kind of a preparing ourselves for [40:42] those times where things go down as [40:44] opposed to preparing for when our [40:45] systems go down. It'll be preparing for [40:46] when our tooling goes down because [40:48] that's going to be this crazy point of [40:50] failure that we all have to be prepared [40:51] for. So maybe we shift around kind of in [40:53] the process, but I think humans are [40:54] going to be there doing a lot of things [40:55] and hopefully the AI is there to take [40:57] care of some of the kind of everyday [40:59] fires that we're in today. [41:02] >> Yeah, totally agree. Um I think also the [41:07] I feel like uh if you're talking two to [41:09] three years, it's so hard to predict, [41:11] but I think the best teams won't just [41:12] respond to incidents faster, right? [41:14] They'll fundamentally shift where [41:16] incidents are handled. And that goes to [41:19] you know um where we'll be spending our [41:22] human capacity agency on and I think a [41:25] lot more of that detection and [41:27] mitigation will happen before a human is [41:29] even paged through things like [41:31] activating shadow canaries proactive [41:33] validation or um systems that can [41:36] contain failures early uh with any [41:40] patterns uh you know like circuit [41:42] breakers or how do we activate our uh [41:45] runbooks to more context aware systems [41:47] that can surface likely causes uh [41:51] impacted services and next steps. Um and [41:54] I think it's a big win if our engineers [41:56] are making um uh decisions with a much [42:00] more narrower search space, right? And [42:03] not just like exposed to so much [42:05] information. Um and I think we have a [42:08] good uh chance that the potential uh [42:11] especially when we use more agentic [42:12] systems are getting there for sure. And [42:15] I think like lastly to all the good [42:17] points that were made around postmotms [42:19] um similar line of thought they won't [42:22] just produce docs right they'll [42:23] continuously feed back into the system [42:26] so every incident makes the platform [42:28] smarter even if just by 1% either [42:31] through your operational reviews or some [42:33] rhythm like that where we can feed it [42:35] back to the system. So I feel like to [42:37] your point Dan um you know if we can [42:39] move from that reactive firefighting to [42:41] more adaptive learning systems that can [42:43] assist decision making for our engineers [42:46] that's a future that I'm kind of betting [42:48] on I guess in the next couple of years. [42:53] >> Yeah, very much thinking the same thing. [42:55] I mean who's to say what things are [42:57] going to look like two to three years [42:58] from now? I don't know. I for one [42:59] welcome our new robot overlords whatever [43:01] they may look like. But keeping in mind, [43:05] yeah, everything that we've been saying, [43:07] you can't automate away the human side [43:10] of things. So, making sure that we're [43:11] keeping the focus on the human side, the [43:14] customer side, the business side, like [43:15] what are we we're not just playing with [43:18] computers because it's so fun like we [43:20] are trying to solve problems for people. [43:22] So getting rid of the toil, seeing if we [43:25] can, you know, automate away a lot of [43:26] the toil, a lot of that reactive work so [43:28] that we can get to the more proactive [43:30] side of like trying to remember what [43:33] problems we are actually trying to solve [43:35] in the first place. [43:37] >> We kind of we kind of talk about a [43:38] future where a lot of folks instead of [43:40] becoming like the incident commander [43:41] today is like pretty active. Um, but we [43:44] kind of always like we see this picture [43:45] of of folks kind of moving further up, [43:47] right? where it's kind of like the human [43:48] is still involved, but they get to [43:50] orchestrate more instead of having to uh [43:52] be on the front lines of making those [43:54] decisions. They can kind of see what's [43:55] going on and change. So, there's some [43:57] pretty cool um pretty cool future that [43:59] we could have with being able to move [44:01] ourselves slightly up in the in the food [44:03] chain for instance. [44:06] >> Thank you. That's a great way to close [44:08] for now. We actually got a great [44:09] question from the audience uh which we [44:11] don't have time to answer, but we we can [44:14] answer in the lead Slack. So you can [44:16] head over to the lead dev Slack and join [44:18] the panel Q&A channel and feel free to [44:23] ask our panelists any more questions, [44:25] chat about this discussion. Thank you so [44:28] much for joining us and we hope to see [44:30] you soon. [44:32] >> Thank you all. [44:33] >> Thanks everyone.  📄 **Full text:** ![[Full Document Contents/Articles/Your Blueprint for Automated Incident Recovery.md|Your Blueprint for Automated Incident Recovery by YouTube]] ## Highlights - have a product manager, which I haven't had an elite dev panel before. And I think it's actually really like pertinent to the moment because our roles are really starting to converge. A lot is happening and PMs are getting more technical. Engineers have to think more like PMs. So, it's great. things only get blurriier, right? >> Exactly. So, we're talking about incident response today. How teams move from those first chaotic minutes of an incident towards a more orchestrated and ([View Highlight](https://read.readwise.io/read/01kqzg1z2g4g165hyg7ncngyv2)) - Tags: 👻 ai highlighted - lot of it is about kind of like slowly building towards this this this new model of the auto remediation kind of side by building up that context, making confident decisions and making those uh decisions actions available to folks as quickly as possible so they can kind of take the next step. So hopefully by doing more and more of that and by by learning from the incidents we have, we're able to kind of find find the right people faster um identify what was what information was missing and make better decisions earlier. Yeah, that's great. ([View Highlight](https://read.readwise.io/read/01kqzg1z3bp1rz64qh01raz9sa)) - Tags: 👻 ai highlighted - of the stakeholder comms. how do we get some folks that are there for that purpose as well? Um so being able to decide when those decisions have to be made. Um being able to to make that decision for the user sometimes like oh it's been 10 minutes we know that things are bad let's move faster. Um because the you know it's it's easy for for an engineer in the middle of the night to kind of decide I'm going to hang on to this for another five minutes because I don't think it's as bad as I as I think. Um so a lot of it is about like setting setting those timelines um and connecting all the different tools that you need to in order to have that kind of response and connecting the people. ([View Highlight](https://read.readwise.io/read/01kqzg1z5q8vhvf7qwga6rg53w)) - Tags: 👻 ai highlighted - orchestration and automation before and after incidents as well. Like during is great like that's you know fantastic but I think there's you know before like can we stop incidents from happening? Can we do better things with our CI pipelines, our test pipelines, making sure that issues get caught before they go into production or afterwards putting together you know the postmortem document. I think it's one failure mode I've seen is the details get lost of who did what when and what graph were you ([View Highlight](https://read.readwise.io/read/01kqzg1z8eyk6rc87zhw7vwayj)) - Tags: 👻 ai highlighted - slowly adapt and I think we Mala you touched on this really well like human human guidance at first right like we're going to give you more information about what you should think about doing this is what what AI thinks we're going to do or what what the automations suggest based on just rules-based automation and that kind of learning. Um like there's so much that you can get out of this in the incident management side today that we probably like you're just scratching the surface. You don't have to jump fully into like Gen AI the uh AI S is going to do all this stuff without ([View Highlight](https://read.readwise.io/read/01kqzg1zcc37ww2vgca9wjwgj4)) - Tags: 👻 ai highlighted - >> Definitely there. So there's also some interesting stuff here with kind of I think we we we touched on the kind of the communication aspect pretty early. Um and sometimes with automation, you know, and and some of those pieces uh for internal columns, it can be fine for those pieces to be automated, but sometimes with external audiences, uh you know, we do want to be able to have some some humanized just to make sure we're not saying the wrong thing. So I think those things will get better over time, but for now at least, I think a lot of the comm stuff is also still good to at least have a bit of guidance human ([View Highlight](https://read.readwise.io/read/01kqzg1zfq4xkvq14v6dgm1ybv)) - Tags: 👻 ai highlighted - in the loop stuff there. Um the one thing that I I find fascinating with this stuff is I feel like things have shifted so much in six months even with some of these pieces. Um uh so I I like I I really appreciate Mala that kind of like it's not just speed but doing the right thing and not making the wrong decisions that pen in. Um so I think right now with automation a lot of it is about doing those those types of actions that um would be low risk. uh right like so so uh in complex systems that that does get ([View Highlight](https://read.readwise.io/read/01kqzg1zr5f81ms1wxv38j242f)) - Tags: 👻 ai highlighted