Your Blueprint for Automated Incident Recovery  🔗 **Source:** [Your Blueprint for Automated Incident Recovery](https://www.youtube.com/watch?v=FPsuYDsHOBg) Welcome everyone. My name is Nesla. I am really excited to be uh hosting this panel today. This is a um panel called your blueprint for automated incident recovery uh which is in done in partnership with lead dev and everbridge x matters. So the discussion is going to last about 45 minutes after which both myself and all the panelists are going to head over to the lead dev slack. We'll post the um link if you don't have that already and you can ask more questions there uh to the panel in the Q&A channel. We might also have time to answer a few questions live. So please do submit those through the Q&A on Zoom and I'll be actively monitoring that to uh make sure we can answer them if if there's time. And first I want to get started with some introductions. So like I said my name is Neesla. I'm joined by three amazing panelists here. A little bit about myself. I'm a director of engineering at a startup called Align based in the US, although I'm based in Berlin, Germany. I'm also a leadership coach and I've been working with Lee Devub for a few few years. I love doing these panels because I always learn a lot. So, I'll pass it over to Ren to introduce themselves first. Hey everyone, I'm Ren. I'm the tech lead of the cloud platform team at Mapbox and I've been dealing with incidents for nearly 20 years now since I was a one-person ops team at my first startup tech job in New York where I was, you know, the one person with badges to the data center. Uh, so I've seen kind of the good, the bad, and the ugly over the years and I'm really looking forward to this panel talking about how we can get some more of the the good in there. Thanks Ren. Malikica. >> Uh hi you all. Uh I'm Malika. Um I am a engineering leader at Zorkdoc. Uh and most recently uh I I've also been at Netflix. Uh previously at Walmart and Twitter and I've spent a lot of time building and operating distributed systems in general particularly in environments where reliability, velocity and stability are constantly in tension. And a big part of my focus has been thinking about incident response not just as a process problem but as a systems and decision making layer um uh under you know all the uncertaintity that is there and how do we get the right context to the right people at the right time. So most recently I've been exploring a lot about how AI can shift that from static runbooks to dynamic context aware systems. So I'm very excited to like dig deeper into uh those constructs with this group and um you know for this conversation. >> Thank you. And last but not least, Dan. >> Hi there. I'm I'm Dan. I'm a product manager from Everbridge X Matters. Uh so I used to be on the tech side. So I used to be an engineer and engineering team lead uh back in the day like six or seven years ago. But for the last six or seven I've been working uh on kind of solutions that are related to incident management. So, I've had an opportunity to talk to lots of customers about uh their problems, lots of folks about what they're seeing. So, excited to kind of to chat about that with you all today. >> And I I will say I'm really excited to have a product manager, which I haven't had an elite dev panel before. And I think it's actually really like pertinent to the moment because our roles are really starting to converge. A lot is happening and PMs are getting more technical. Engineers have to think more like PMs. So, it's great. things only get blurriier, right? >> Exactly. So, we're talking about incident response today. How teams move from those first chaotic minutes of an incident towards a more orchestrated and effective approach. So, systems that we're working with are actually more complex than ever. As we all know, it's making things harder to diagnose. It's making the ownership more unclear. And at the same time, AI and automation are becoming a a newer and bigger part of the bigger part of the process. But it does raise questions around trust and how often we need human judgment. So we're going to explore all this today. Uh we're going to where orchestration um can be automated, where it shouldn't be automated, what this looks like in practice. Let's dig in. And I want to start with Ren. So, can you tell us what actually happens in the first 20 minutes of an incident and where does that typically break down in your experience? >> Yeah, I mean the the first thing is well figuring out is this actually an incident? You know, you've you've gotten paged. It's maybe the workday, maybe the middle of the night, who knows? You're you're figuring out all right is this is this actually a problem? And then kind of kind of what you need to start doing after that is figuring out what what is the actual impact. Um and then organization is really what should be happening first. Um getting communication in in gear because as a customer there's nothing that's more frustrating than going to, you know, the status page when you know that there's something wrong and it just says everything's green. All of the check marks, everything is happy. You want to make sure that we're we're communicating to our customers as soon as possible. And since one person cannot do all of the things all the time, that means assigning different roles to people. You know, get your incident commander or whatever you're talking calling that role in place, dividing up the work. I think the place where I've seen that go wrong most in practice is you get engineers who they want to figure out what the technical problem is and they start rabbit holeing and forget the people side of things, the customer side of things, the business side of things like are we bleeding money that we need to you know get that bandated up before we you know dig into the technical problems. So keeping keeping focus on, you know, the business side, the customer side, what's actually important while trying to organize everything else that's going on in an increasingly complex system? Yeah. And speaking of people in complex systems, Dan, how do you figure out when an incident is happening? Uh who owns the incident? How like how the work is going to get distributed? How do you avoid this like hot potato problem or like oh it's not me get it off my plate? >> For sure. A lot of this is about kind of finding the right ways to uh prep for incidents as well, right? So how do we make sure that we have the correct that we're getting the correct signals? How do we orchestrate those signals across that we make sure that they get to the right people and that we can also raise the right context to to those folks to know is this something that I need to deal with today and now. Um, so, uh, usually it's about kind of meeting people within the tools that that make the most sense for them, but also finding ways to to build build some kind of process over top as well, so that there's some consistency between teams because I think we all have teams that want to use different tools. Um, so how do we make sure that those signals that are kind of specific to one team um, can be there as context for the for the larger group to be able to make those calls early as well. And how do you make it so that uh, they they can take that action quickly and feel confident about what to do next. Um, so a lot of it is about like the alert that you receive can be pretty vague. Um, what if we could enrich it with some more context so that you without having to like log into five systems understand what's going on and what your what your part of the world looks like and maybe what some of the other parts of the world look like. How do we make it easy for you to be like that's not enough information. Where do I go next? Um, and of course, you know, the more that we can do some of these pieces, the more we can automate behind the scenes and take some low-risk actions on behalf of our teams, um, that they would likely be doing if they were being woken up in the middle of the night or, you know, in focus time during the day and their flow. So, so a lot of it is about kind of like slowly building towards this this this new model of the auto remediation kind of side by building up that context, making confident decisions and making those uh decisions actions available to folks as quickly as possible so they can kind of take the next step. So hopefully by doing more and more of that and by by learning from the incidents we have, we're able to kind of find find the right people faster um identify what was what information was missing and make better decisions earlier. Yeah, that's great. I think you're touching on like how important process is like okay like teams are all operating independently humans are inherently different like let's start there right so people are going to think teams are different dynamics but then the tooling might be different from team to team >> um the like what you might deem severe versus not severe that even is subjective >> um and it you're talking about building a system where people can cleanly like interface with one another and it's legible and >> exactly >> yeah and I'm curious uh from you Malikica like as that's happening and like information is being tossed around how do you make sure that the institutional knowledge that needs to be retained by certain people actually stays there over time it actually gets to the right people maybe not in the moment of the incident but also before and after. >> Yeah. Uh that's a great question and I uh I I do think a lot of that incident response today still relies on exactly what you uh I love the word that you use that institutional knowledge right uh the one engineer who just knows where to look um and the I think the mistake the teams make is trying to flatten that into static runbooks that doesn't scale to a large extent if your incident system still depends on heroics it's already broken It's about finding those sustainable systems that actually reduce cognitive load during incidents and not necessarily increase it and reaching out to multiple humans does increase it. Um what we have seen work better is capturing decision trees not just steps uh linking signals to hypothesis and actions continuously updating this from postmotms and increasingly AI can help here not just by replacing expertise but by retrieving and contextualizing it in the moment. Um for example even how we think about postmortems right if your postmortems don't change the system they're just storytelling um the bar should be what did we encode back into the system so this class of issue gets easier next time um and not just becomes uh you know buried in those postmortm logs right another example is circuit breakers they are underrated um right if we learn something from a subzero can we bring it back to you know maybe editing or calibrating our circuit breakers uh not just as a resilience pattern but as an incident containment strategy itself right so I do feel like good incident systems don't just detect failures they limit that blast radius automatically so that shift is really from documentation to living queriable system knowledge and that is how we can kind of take all of that institutional knowledge and slowly encoded part of the system um uh so that uh it's kind of a default how do we want to go about um managing when it happens again. Um yeah. >> Yeah, I totally agree. I think that so much of it is about how do you how do you learn and how do you kind of adapt to those things because there's always going to be new new like our systems are con constantly changing. Um the world around us changes. I feel like you know we can control what we can control. We can't control what our partners and our vendors are doing. So being able to adapt and being able to improve is just really important with these kinds of things. >> Yeah. It's almost like, you know, in the before times you would do postmortem and, you know, you might lag. There might be a long lag to actually get to all the to-do items to create that institutional knowledge to get people to even read the postmortem document. But now it's like we have less of an excuse, right? like the postmortem should be sent into whatever ticketing system that you have. It should be queryable, right, in a way by others to in order to learn. Um, so there's a lot more um power that we can unlock from this documentation, it sounds like. And another thing I'm curious about is what does orchestration look like in practice beyond just alerting the next person nowadays? like what is what is the actual like flow that we need to care about? >> For sure. Yeah. So, I mean, I think I've kind of I touched on some of those pieces in the last one because I I like to get ahead, but um the a lot of it is about kind of like the context. Um so, I think, you know, within a lot of alerting systems that we we we use today, um receiving that kind of initial information usually is pretty thin. Like it has the context of like the original signal, but doesn't kind of pull anything else in. So part of that orchestration is bringing that right information and what those next steps might be to to users directly to to people that are responding. Um and then from there it is I think Malikica you kind of touched on just the fact that like how do we how do we prepare for what the next step is and how do we like what are what are some of these circuit breakers sometimes it it can also be to like make quick decisions about when to notify more folks to get more context and to to I think NA you mentioned kind of the stakeholder comms. how do we get some folks that are there for that purpose as well? Um so being able to decide when those decisions have to be made. Um being able to to make that decision for the user sometimes like oh it's been 10 minutes we know that things are bad let's move faster. Um because the you know it's it's easy for for an engineer in the middle of the night to kind of decide I'm going to hang on to this for another five minutes because I don't think it's as bad as I as I think. Um so a lot of it is about like setting setting those timelines um and connecting all the different tools that you need to in order to have that kind of response and connecting the people. Um so being prepared uh with with kind of the the right escalations and and getting that context out together. Um over time as well it is how do you automate those pieces so that users don't have to make the choices as often. Um and of course that's that kind of gets to those points around we have more and more tools at our disposal to make decisions so to speak with AI. So how do we what's the right line with some of those pieces is always interesting. I think there can be some really interesting opportunities for orchestration and automation before and after incidents as well. Like during is great like that's you know fantastic but I think there's you know before like can we stop incidents from happening? Can we do better things with our CI pipelines, our test pipelines, making sure that issues get caught before they go into production or afterwards putting together you know the postmortem document. I think it's one failure mode I've seen is the details get lost of who did what when and what graph were you looking at and what made you decide to do that thing versus that other thing. And I think this is something where you know if you can get you know Gemini to take the transcript of the huddle that your engineers were in and pull out those details in a more automated fashion so that information isn't lost and then you can you know codify that into you know future run books, future documentation, future automation, that sort of thing. >> Yeah, I think we live in an exciting time where I feel like get having time to deal with incidents was always a struggle. So that meant that everyone was kind of firefighting because that was always kind of the um even though it's the highest priority thing when you're in an incident, it's the lowest priority thing to think about when you're not. It's, you know, how are we delivering value to customers and not worrying about that stuff as much. Um it seems like AI is going to give us a lot of opportunities to have something thinking and chewing on some of these problems and and building up that context. Um so hopefully in the future, um even though we may not still have more and more time for those things, we'll have we'll have someone working on in the background, which is great. Well, let's let's talk about the a word and the two a a words AI and automation. So, something that Ren said earlier was uh uh like a failure mode is when an engineer tries to go too deep, right? And that's because I think as engineers, we're so like curious and we want to understand this thing, but it might not be like the it might not be what the situation warrants. So maybe now we have the ability to to ask something to automate like the next step for us. How how can we actually trust all this automation that we're building into these systems? And I'll kick it off with you Malikica, but I'm sure everybody has some ideas here. >> Yeah. Um I think the limiting factor isn't tooling, right? It's actually trust under pressure for our engineers. Um our engineers won't rely on automation during a subzero unless they've seen it behave predict predictably before and that's to uh you know what Naja you uh alluded uh Dan and Ren pointed out as well and most teams don't have an automation problem they have that confidence problem where if our engineers don't trust the system they'll bypass it at the exact moment it actually matters. M >> um so you know like there are a few ways to build that trust proactively right like start with recommendations not actions have more human in the loop at least in the uh more initial iterations make systems explain their reasonings not just outputs um and then invest very heavily in guardrails and blast radius control just so that you know incidents or like uh metrics degradations from our upstream downstream same customers don't start affecting us and I think most importantly teams need to see these systems work during low stakes incidents first or even simple patterns like you know one pattern I found really effective is shadow canaries running changes against real production traffic without impacting user seeing how that goes it shifts incident detection left right so you're catching issues before they become incidents it's one of the cleanest cheapest ways to actually reduce subzer without slowing teams down. So investing in uh uh uh processes systems which can actually build trust technically through the systems and it also I think creates a feedback loop where automation has real data to learn from. It's not just synthetic tests running in the background. So I do feel like trust isn't designed right it's earned through repeated safe exposure and our engineers need to see that in action. uh before something really bad happens and it's going to happen. So it's you know like can we accept the fact and like prepare for that future >> and I'll say making sure you understand what the tools are doing and how they work like back when we were writing you know bash scripts for everything you have to know how to do something manually before you can automate it before you can write a script for it and kind of a similar principle applies with AI tools like you have to understand kind of how they work what they're doing under the hood like learning that oh you know the context is going to get compacted. So if you have you know in your claude MD file don't run destructive commands against prod that might get lost due to how the tool works and if you don't understand that you're going to be surprised when claude goes and you know deletes some production infrastructure that you weren't expecting whereas if you know like okay here's how hooks work in you know the the life cycle of the tool you're going to be better prepared. So making sure that we're not just like jumping on the hype train of all of these shiny new tools without really understanding what they're doing and how to use them effectively. >> Yeah, human trust is kind of funny. Um because I do feel like we we jump to trust pretty quick in certain cases when you show us that you so I feel like with AI we're kind of in that riding the wave of like you know go I think I every every week I feel like I hear different stories of like it's amazing and it does everything and I never have to worry and then on the other side actually maybe not maybe we need to pump the brakes. Um, but I I think there's also just so much room like uh we talked about this a lot um with with with with X matters but um just uh what how do you kind of like slowly adapt and I think we Mala you touched on this really well like human human guidance at first right like we're going to give you more information about what you should think about doing this is what what AI thinks we're going to do or what what the automations suggest based on just rules-based automation and that kind of learning. Um like there's so much that you can get out of this in the incident management side today that we probably like you're just scratching the surface. You don't have to jump fully into like Gen AI the uh AI S is going to do all this stuff without without a human in the loop right away. Um so it's like how do you take those steps to build towards that future and that vision and and build that trust slowly instead of kind of jumping to the end state first. Definitely lots of options for that. >> Yeah, I love that. I mean ultimately like the what we what we have here is a relationship with the tools, the technology and relationships can be toxic. They can be up and down emotionally, right? Or they can feel clear and transparent. And um I love talking about incident response because it really is one of those things that touches people, process, and technology all at the same time. And uh when you're really in the incident, the people aspect kind of trumps everything because you need to mobilize everyone so quickly and and get to that actual resolution that is needed in the moment. Um and Ren, based on your experience being in the trenches, what what do you see as this like human cost for poor incident management? And how do you go back to being sustainable in terms of how people are feeling? Burnout, being able to actually fix things. Yeah, go. Please go ahead. Yeah, burnout is definitely the biggest thing. Um I mean I the size of your on call roa obviously matters. Like I've been a oneperson ops team, you know, the one person on call 24/7 365 not getting a break. But even even a a small team where you know every other weekend you're getting pulled away from you know your kids, your family, your hobbies, whatever, it's going to take a toll. Um I think I I forget where I read this, but I feel like I I read something that said that most people, you know, if they had to choose between like getting time off and getting more money in in compensation for being on call, they all wanted the time off cuz you know, money can't give us back, you know, the the vacations that we missed out on or whatever. So thinking about what are the biggest pain points in your particular incident response system like for some people it might be you know false positives. You get these alarms that didn't actually mean anything. The blips that resolved by the time you got your laptop out to go look at them. Or maybe it's, you know, feeling like postmortems are useless because the action items just go into a Jira void and nobody ever looks at them again. That's like, why are we going through this process if nothing ever actually changes? Like, oh yes, that it's that SEV one again that we did last month and the month before that it's the same problem. Um, so make sure that teams have the time in your in your sprints, in your schedules, whatever, to be proactive and to actually fix those pain points. Um, talking to the engineers who are the ones on call and asking them like, hey, if you could fix one thing about how we do incidents, what would it be? Compile those answers and then actually make time to fix them. because I feel like that's kind of the trap that most teams fall into is they get into just this treading water really reactive state where you just can't ever get ahead of everything that is always on fire. So sometimes you have to, you know, stop the line, pump the brakes, say, "Okay, we have to get out in front of this. We have to actually change something now." But without, you know, support to actually let teams do that, you're just going to end up with more fire. I totally um agree with Ren's point here. It's just uh surprising to me how many lowhanging fruits there are typically uh as part of the process you know uh weekly rituals or bi-weekly rituals where you know looking at our noisy alerts or our triage process right are they even going into the right buckets are we treating everything the wrong bugs as se zeros and se ones um and are we tweaking that process continually where you know as part of our quarterly retros um or any team rituals there is a set system where we can edit these um ways of how we do our on call that is crucial and I feel like every incident story um is a powerful story exactly because there is a human behind it at least so far right like we we've been relying so much uh on the human in the loop and that's why those stories have so much of data and knowledge in them. So having a way to actually act on them to Ren's point um is a very lowhanging fruit uh for uh teams to kind of um jump on. >> Yeah, I think how do how do you kind of get get that buyin from everyone to create space and time to work on some of this stuff? Right. So, as a as a PM, but our organization is is thankfully pretty um I I always I was from the engineering side, I was always kind of a mature organization when you actually had time to work on stuff that wasn't just product features um where you actually had the opportunity to kind of work on some of that debt side. Um so, we've we've always been able to kind of maintain some of those pieces, but um hopefully I think also within this incident world, the the better tools you have to be able to kind of identify look at this group that's going through this pain. Um, right? Like we can see that this person has responded to this many things. They're getting you can see the the charts on like how where they're getting uh they were woken up this many times overnight. Like some of those types of hopefully the more data that we can provide to uh folks like me or folks that that might be controlling kind of how much time you can spend on this stuff, the more you're able to kind of empower them with that data as well, hopefully there's some some change you can make because yeah, the the human toll is is significant with uh with being being on call. So how how do we make that better? Um, and that the other part of that is as well is like making sure that folks are using the tools correctly. Making sure that they've defined the devices that that will actually wake them up. Um, so that you know, you don't have the one person that set things up correctly and is getting woken up every day because the other folks know that they're going to be there as the back stop behind them. So, how do we make sure that folks are encouraged to to kind of do the right thing together too, right? Not that I think anybody seems to do that, but >> Yeah, that's great. And you know, it's funny, Dan, is I used to be a product manager. So, I actually, you know, if I put my PM hat on, I think, well, everything is a funnel. And when you have a funnel problem, you're always looking at the constraints in the funnel and and you tackle one problem at a time. And so, within with incident response, you know, back to Ren's point, the the best way to handle incidents is to not h have them. >> 100%. >> And now we can Yeah. Now we get to now we can through queries ask like hey these are the incidents we saw in the last 3 months um scan my codebase and tell me what could you know knock out 20% of this right how do we reduce this dramatically so we actually get this opportunity now that I think if you didn't have time before it's become much easier if you didn't have that like maybe PM mindset >> cultivated for so long in order to do this type of analysis Right. And so in this way, we're all blending quite a bit. >> Every so often, I hear some great stories of folks where it's kind of like they talk about their teams just being kind of um so focused on firefighting. Um and it's just amazing kind of the next tier of problems and and just the more time that you have to spend on other things that you can get to if you can deal with these things well. You can orchestrate these issues so they don't become major fires that last hours and hours or days. So it's it is pretty amazing kind of the difference that it can make. Yeah. And on a similar vein, you know, there maybe are are some things that maybe some things we should automate, some things we shouldn't. In y'all's opinions, uh, Mal Malik, we'll start with you. Like, what are the things that we definitely should not be automating? And what's a clear yes to you? Yeah, I I feel like one of the costliest mistakes I've seen is treating automation as inherently good. Um, right. Uh, and especially in incident response. Um, speed without judgment is dangerous and it has a big cost and speed comes up a lot and that's why I talk about speed and in incidents. It's just not about doing things faster, right? It's also about making fewer wrong decisions early uh so that they don't cascade into our decision making later. So there are really three areas that I'm very cautious about when it comes to automating any irreversible actions right like any uh data deletion or large scale failovers. Uh it could also be any ambiguous root cause scenarios where signals are still noisy just waiting a little bit more before they stabilize. any cross service cascading actions. This is huge you know like if you have uh a big blast radius lots of downstream upstream implications. So automation should handle known reversible well understood parts. For anything else it's actually okay to watch for a little bit and wait for the system to stabilize. Everything else should be decision assisted not decision replaced. Um and the highest leverage automation um actually happens before incidents. uh right like Ren mentioned this as well things like uh can we block bad deploys validating configs catching anomalies early and can we codify this uh in the system and I think it gets easier when fewer incidents exist right so creating that space where we can go towards that northstar where we just have fewer incidents um so I feel like that goal shouldn't be maximum automation um and it's actually maximum safe acceleration so teams can unlock that velocity and like Dan said we can redirect them to more meaningful places either in the product or on the inside but being really mindful as we do that. Uh curious to hear other thoughts as well. >> I think making sure that we know what our automation is doing and how it is behaving like kind of who watches the watchers. If you have some nice, you know, auto remediation setup, if that thing fires, I don't know, once a week, once a month, it's probably fine. If that's going off 20 times a day, you probably want to know what is being automatically remediated that often. But too often I've seen teams just, you know, get so excited and set up all the automation and then forget about it and it goes into another void like the Jira void where maybe it's doing the right thing, maybe it's not, but nobody's really paying that much attention to it anymore and it can become too easily another source of toil. >> Definitely there. So there's also some interesting stuff here with kind of I think we we we touched on the kind of the communication aspect pretty early. Um and sometimes with automation, you know, and and some of those pieces uh for internal columns, it can be fine for those pieces to be automated, but sometimes with external audiences, uh you know, we do want to be able to have some some humanized just to make sure we're not saying the wrong thing. So I think those things will get better over time, but for now at least, I think a lot of the comm stuff is also still good to at least have a bit of guidance human in the loop stuff there. Um the one thing that I I find fascinating with this stuff is I feel like things have shifted so much in six months even with some of these pieces. Um uh so I I like I I really appreciate Mala that kind of like it's not just speed but doing the right thing and not making the wrong decisions that pen in. Um so I think right now with automation a lot of it is about doing those those types of actions that um would be low risk. uh right like so so uh in complex systems that that does get harder because you don't necessarily know how your change might impact another change and the dependencies between things get harder. So maybe roll backs aren't safe all the time. But if there are places where you can do safe roll backs to previous versions the more mature kind of our infrastructure is with this it also makes those pieces easy. I'm like we'll talk to customers where you know it's it's a a lot easier to roll back or fail over in some of those pieces and that's great. then you can then you can really like make use of automation in a lot of interesting ways. Um but but yeah, for for kind of crustier or simpler uh uh infrastructure, it can be really challenging to automate those pieces and that's okay. It just takes time for us to kind of understand where we're at and slowly grow and build the evidence to say, hey, we should make this better. So um yeah, but I think in six in a few months it might change as far as what we think our tools are able to do. So it's it's an interesting time. >> I'm sure that it will. Um, we have a question from the audience. I want to get to that. This is from Colette and anyone can answer. There's some scientific research that points to the fact that when teams don't have regular practice at incidents, it can lead to skill atrophy for when in incidents for when insulin do actually happen. With the rise of automation and a push to have less incidents, how should teams and leaders address this atrophy? I think game days is a big one like yeah it's very true I mean that that was kind of the impetus behind like the original you know how what was what was that the talk like nine deploys per day 12 deploys per day you get good at what you practice regularly and yeah if you're not practicing how to respond to incidents it's it's going to get rusty so yeah game days chaos monkey whatever you want to do within your organization and depending on kind of the size and structure of your org. You might want to handle this differently. Um, you know, have have teams set up different puzzles for each other. Have people jump on a Zoom and watch troubleshooting. Um, we've done things on my team where somebody, you know, the one person who who knows that one system and gets woken up all the time because it's it's in their head. Great. Next time we get paid for that system, let's everyone hop on a call and watch and just do, you know, whether it's a real run through or take take us through this thing you did last time. You know, remember what you did did a couple months ago, but making sure that you're taking advantage of all the different resources that you have on your team. And I think, you know, probably this is something where AI tools can help come up with game day scenarios. You know, look at our incidents over the past 6 to 12 months. you know what happened the most, what should we be practicing. So, it's it's going to be interesting to see where that goes. >> Yeah, totally. Uh big plus one to game days. Um uh and uh it's a really good question. Uh like avoiding incidents entirely can make teams worse at handling them, right? So, uh I think the question is very nuanced, but the fix isn't more outages, right? It's more intentional practice and that's where uh heavy plus one to game days it could even be failure injection premotms replaying past incidents so engineers can build that muzzle in a much more safe environment and I think most importantly not also fully automating the thinking behind how we solve these problems and that's where no matter how uh much of AI or agentic your systems become uh on how you handle this keeping that human in the loop for diag agnosis and decision making even in the future so that um a team stay sharp I think is still valuable so it's probably more deliberate exposure to failure modes right that's the way I think about it >> yeah for sure and >> even in your game days like what happens when the tooling doesn't work I mean cloud had an outage a few days ago right and then everyone on the internet was like what do we do now so it's important to test the different scenarios and Um, a bit more on automation from the audience. Someone asked, "What are your takes on automated postmortems which we mentioned earlier? I think there kind of is is some similarity to this in the previous one um where I think you can automate a lot of the um the the effort of pulling together information, deciding like and suggesting what root causes were in some of those pieces. But for now at least, I think it is it's still probably important for humans to really process that information and do something about it or else or else it probably doesn't benefit anybody really. Um but I think over time again like the the more that we have kind of um tools that will then process those and then kind of suggest changes the our rate of change might go up in the future. Um so those things may may change but at least I I yeah I think there's still a lot of reason to keep humans in the loop on all these types of pieces. >> Definitely. I think there's post-mortems that tend to focus on more the technical side and then there's postmortems that kind of dig deeper and go more into the like systemic side, the human factors side. And that's where you you have to have the humans like tools are not going to be able to answer why a human took a particular action. Like you you've got to talk to talk to the people about that. like what what were you thinking when not in a you know obviously you know blameless postmortems but just like what were you thinking you know what made you go look at this signal instead of that signal and yeah you know we're not going to be able to automate that sort of thing away some interesting stuff there too I feel like the people element there is also the business element and that's probably hard for for our tools to be able to pull that information right now um so maybe I'll change um but really being able to understand kind of that full scope of impact is is still still valuable. >> Totally. All right. If anyone in the audience has any more questions, please write them in. Otherwise, I have um another question for the crew here. So, this is for everybody. in two to three years after this, you know, up and down with emotional roller coaster that we're all on with AI. Although I don't know if there's going to be an after or we're just going to be in this indefinitely, but once things settle, where do what do you think the best incident response teams will do differently than uh they did will do differently than they do today? >> Anybody want to go first? Okay. I'm I'm not on mute so I can um so the I think that I mean it's going to look entirely different probably for the for the most part. Um we will probably trust the tools a lot more um because they we will understand better how to give them the correct guard rails and the correct context. Um so the the hope I think is that we have uh fewer humans having to be impacted by things and detect these things faster so they don't become business impacting as soon as they do today. Um, now the I think you you brought up a good point that what happens when the tools go down. Um, so you know, we're still going to have to be able to understand what how how can a human interject in these things and still take actions as well. We're going to or or else like you know, how many how many tools down do you have to worry about in order to make sure that you have redundancy with something like this that can be so critical. Um, so I think you know the the talk the topic of like the uh game days staying staying kind of active with this that's only going to be more important at the rate of change we're going to move forward. So you have to be you have to be ready and I think a lot of us is going to be this is going to be kind of a preparing ourselves for those times where things go down as opposed to preparing for when our systems go down. It'll be preparing for when our tooling goes down because that's going to be this crazy point of failure that we all have to be prepared for. So maybe we shift around kind of in the process, but I think humans are going to be there doing a lot of things and hopefully the AI is there to take care of some of the kind of everyday fires that we're in today. >> Yeah, totally agree. Um I think also the I feel like uh if you're talking two to three years, it's so hard to predict, but I think the best teams won't just respond to incidents faster, right? They'll fundamentally shift where incidents are handled. And that goes to you know um where we'll be spending our human capacity agency on and I think a lot more of that detection and mitigation will happen before a human is even paged through things like activating shadow canaries proactive validation or um systems that can contain failures early uh with any patterns uh you know like circuit breakers or how do we activate our uh runbooks to more context aware systems that can surface likely causes uh impacted services and next steps. Um and I think it's a big win if our engineers are making um uh decisions with a much more narrower search space, right? And not just like exposed to so much information. Um and I think we have a good uh chance that the potential uh especially when we use more agentic systems are getting there for sure. And I think like lastly to all the good points that were made around postmotms um similar line of thought they won't just produce docs right they'll continuously feed back into the system so every incident makes the platform smarter even if just by 1% either through your operational reviews or some rhythm like that where we can feed it back to the system. So I feel like to your point Dan um you know if we can move from that reactive firefighting to more adaptive learning systems that can assist decision making for our engineers that's a future that I'm kind of betting on I guess in the next couple of years. >> Yeah, very much thinking the same thing. I mean who's to say what things are going to look like two to three years from now? I don't know. I for one welcome our new robot overlords whatever they may look like. But keeping in mind, yeah, everything that we've been saying, you can't automate away the human side of things. So, making sure that we're keeping the focus on the human side, the customer side, the business side, like what are we we're not just playing with computers because it's so fun like we are trying to solve problems for people. So getting rid of the toil, seeing if we can, you know, automate away a lot of the toil, a lot of that reactive work so that we can get to the more proactive side of like trying to remember what problems we are actually trying to solve in the first place. >> We kind of we kind of talk about a future where a lot of folks instead of becoming like the incident commander today is like pretty active. Um, but we kind of always like we see this picture of of folks kind of moving further up, right? where it's kind of like the human is still involved, but they get to orchestrate more instead of having to uh be on the front lines of making those decisions. They can kind of see what's going on and change. So, there's some pretty cool um pretty cool future that we could have with being able to move ourselves slightly up in the in the food chain for instance. >> Thank you. That's a great way to close for now. We actually got a great question from the audience uh which we don't have time to answer, but we we can answer in the lead Slack. So you can head over to the lead dev Slack and join the panel Q&A channel and feel free to ask our panelists any more questions, chat about this discussion. Thank you so much for joining us and we hope to see you soon. >> Thank you all. >> Thanks everyone., Your Blueprint for Automated Incident Recovery, YouTube, 01kqze72hsd42seq0ha01mz7qf.