Get Smarter With Every Incident  All right, let's get started. So, hello everyone. My name is Najla. I am a director of engineering at a company called Aligned. I'm calling in from Berlin, Germany. And I'm also an engineering leadership coach. and I'm really excited to have this panel today. Um, this panel is going to be about 45 minutes uh where the panelists and I are going to discuss some questions. Uh, but we're also really open to you writing in questions in the chat. So, please get those in and I'll try to weave those in. And if anybody has any questions at the end, we'll all head over to the lead dev Slack in the panel Q&A channel which I'll remind us of at the end. Um, so first let's get started with introducing the rest of our panelists. I'm going to start with Dave. Dave, hello. >> Hi everyone. Uh, thank you for the intro the short introduction. I'm Dave. I manage two reliability teams here at Pager Duty responsible for everything from CI/CD tooling to uh our Kubernetes clusters and network core networking etc. And I also manage our developer experience function here as well. Thanks Dave Arjett. >> Thanks. Thanks. So yeah, I'm Arjett. Uh I am a senior engineering manager with delivery hero in Berlin as well. So I lead the engineering for the quickcommerce side and uh I'm here to talk about incidents and I've spent a lot of time thinking about incident response. So uh yeah, I'm here to talk about that today. >> And last but not least, Spencer. Hey everybody. Uh my name is Spencer. I'm the CTO at Privy. We do uh marketing automation and email and SMS. And I've spent a lot of time both in and managing incidents. Uh excited to talk with you all. >> Great. So yeah, the title of this panel is called get smarter with every incident and we have a nice group of panelists here with a variety of experience and urgency uh really keeps systems alive during an incident. That's kind of the baseline level that we're all used to operating in. But when that really becomes the sustained default, teams get stuck in firefighting, burnout occurs. Um, and yeah, we not don't necessarily resolve the underlying problem. So, we're going to get into that today. How to break out of those loops, how to go from reactivity to um actually maybe fixing some things in the on call systems and where AI is involved. That's a pretty hot topic right now. So, first I want to start with Spencer. Spencer, can you talk about how you're using AI to catch early warning signs and tune alerts? >> I think this is the dominant topic really of the year and probably the last several years and it's pervading every part of of software engineering. And so, you know, when we start to think about incidents, it's, you know, natural and inevitable about how we can use AI to support those incidents, how to detect things earlier. And I think that's one of the areas that we've really been focused at Privy is figuring out how we can use AI tools. And we're really using the tools we already use, which have a lot of these uh kind of AI layers built into them. You know, for us, our stack is is New Relic and Sentry and Pedag Duty. Shout out to Dave's team. Uh but uh things like uh Sentry right now have a lot of the AI tools already built in and so we can turn those on. And one of the ones that we use right now is anomaly detection. And it's cool, but there's a lot of anomalies. And I think one of the challenges for us is figuring out how do we tune the the error detection or this kind of AI uh alerting so that we are not over alerting that we're not creating a bunch of noise for our team. And that's something we haven't nailed yet. You know, I think for us uh we are still defaulting to right now we're defaulting to uh a lot of these AI tools to notify rather than page. And so that's our default. That's the kind of I guess the pragmatic choice we have made is that until we figure that the right way to tune these tools, we'll notify. So we'll, you know, notify our operational Slack channel, but if it's 3:00 a.m. and we get an anomaly detection, at least right now, unless our other uh monitoring tools also go off, we're not going to be paging the engineer on call. >> Uh, go ahead, Dave. Right. The that I'm going to try not to talk too much about page duty, but in terms of AI, how AI fits into incident management, it's something we've been thinking about for a long time, starting with like the basics, like how do we group alert, use AI to group alerts so that we reduce the noise. But I think a lot of it comes down to when you want to apply AI to this problem, no matter how good the AI is, it starts with the data set it has access to. So if you aren't at the beginning of the pipe, if you aren't cautious or in some way discerning about which events you're sending to the AI to build its data set from, then it's going to be its output is going to be noisier for you too. So if you're sending everything from staging, from a development environment, from prod to the same place, then it's going to start anomaly detecting things for you and say like, hey, hey, this is a problem. even though it might not actually be a problem that impacts is worth waking you up for or is customer impacting. So I think some of it has to be in partnership with the AI itself. You have to be you have to find a good way to give it the data that then is going to result in the outputs that you need if that makes sense. >> Great. Thank you. And you can continue Dave because so that was like pre-inccident. I'm curious for once we're in the middle of an incident and you have to triage, what are you how are you triaging? What's slowing you down the most? Um, and how are you like leveraging tools or practices to get context more quickly during an incident? this I I hate to start out qu answer a question with it depends but the two things that the one thing I think that matters in terms of when you're triaging is it is it we like to say is it new and novel or is it well understood because I think the way you triage an incident depends on you know if it's well understood you probably already have existing runbooks and automation and best practices to kind of reduce the time it takes to resolve the particular incident so I'm not necessarily going to spend a ton of time talking about that, but I'm going to assume something's new and novel because those are the ones that take up the more brain space. So like the other part is where is your triage occurring? Is it, you know, is it via Zoom call? Is it via in Slack or some other, you know, like where because you want to bring what you want to do is bring the information and the tools to to the responder to the place where they're working. So like are they able to access all these things in one place or do they have to open 17 tabs or or like you know have have a bunch of things open on their desktop that they have to be looking at at the same time or this is my ambivalent feelings about dashboards too. I think dashboards are great but also they can be overwhelming like I think you need to cut down people's reliance on dashboards and give them more like a refund and this is where again AI can help too is like give them the data and the context at their fingertips. And then uh lastly is how can you reduce the number of humans required for resolution? And I don't mean that in an AI sense, but like if if you have an incident where you need to bring in 15 people, that's already a problem, right? Like how can you get it down to where you only need like one or two people or even like some automation to help out the one person that's that's there. So I think reducing the the friction to triage is is the key. And it could be, but the friction comes in many forms. So I think you have to analyze where it is specifically for your or organization or your team. >> Can I add on to this a little bit because I I love Dave's answer and I think one of the things that we've noticed at Privy and some of my previous roles is that having very specific roles for responders can help a lot with triage. If everybody who's involved is actively researching a different thread and there's nobody who's coordinating and nobody who is kind of, you know, recontextualizing the information that's coming in, it's really easy to have the entire team heads down for maybe longer than your entire budget to get things back online. And you know, engineers who go heads down on triage, on an investigation, it's really good to be able to leave them in that mode and have somebody else who's operating at like an incident commander level or a scribe level or something else. Uh, and I think that's something that a lot of bigger companies have established really great culture around. But if you're operating with a smaller team, even though you may only have, you know, three or five or 10 engineers, being able to separate those roles and identify who is responsible for what during an incident can, you know, greatly speed up the overall response. >> Yeah, absolutely. Like I would definitely also like concer exactly to what Spencer was saying. So for example for us also internally we try to have like let's say the incident commander role itself between different parts of the earth. So you know your fintech incident commander is speaking to your logistics incident commander. So that way essentially you reduce the number of nodes of different people talking to each other. So at least there are specific directed roles and like a a lot more streamlined uh flow for these incidents rather than a lot of noise. Yeah, thanks Arjette. And um yeah, let's hear more from you. So the last part of this is after an outage so or an incident like can you talk about why we see the same problems come up over and over again and how you move from just like reacting to the same problem to actually learning and actually changing what's happening. Yeah, I mean I think one part of the problem is like this we start we see the same problems because sometimes we treat post incident reviews more like a confession space rather than experiments and and what I mean by that is like teams genuinely write like good postmortms. They're like there's clear action items. We have owners but then those action items sort of start competing with your road map and then they get dep prioritized and then a couple of months later we see the same incident or like a variation of that incident with different people and then we we are left wondering why didn't we learn so one thing that sort of worked with our part of like the or that where we are where we started like we were like okay let's treat these post incident learnings like actual features and not always as tech deaths. So we we changed our whole flow from what should we fix to >> is there like the smallest thing that I can do to make this less painful next time. So like if an action item in your incident report is >> move this database X to Y. It's a great action item but it's not going to get prioritized because it requires you getting sponsorship from your leadership from product and all of that. So like what's the smallest thing that you can sort of do to make it less painful. So I think that helped and then uh figuring out if there is repetition of certain incidents. So we like how do you do like not just another postmortm but like more like a deeper architectural review that hey we are seeing a lot more incidents from this part of the system. Is there something that we need to think about a little more in depth or is is that something which is something else needs attention there? And I think one other thing that has also worked out well uh for us is just thinking about how many or how often is someone getting paged outside business hours because we we need to like figure out like active capacity so that there is remediation happening right because if our action items like keep on getting stalled in Jira then our on call also gets stuck in slack and page of duty and everywhere else. So I think that was one of the biggest breakthrough for us to stop treating some of these things as nice to haves and more of more along along the lines of this is how we want to prevent the next outage that essentially breaks the users trust in us and and that sort of helped >> that's great Arjett and because I'm passionate about this I'll also add that like yeah if you treat like if you have access to pager duty you have a road map of work that you can now pursue to fix underlying issues. It's just like when you have an app and you go inside crash litics and you say, "What's my top crash caused by?" Okay, great. If you if you fix that one thing, you've made progress. And um you can treat pages the same way. Like what is the what is which alert went off the most in the last 3 months in the last quarter? And if I tackle that one thing, how much burden am I going to reduce off the the entire team operationally and also in the tech and for customers? Uh Spencer, go ahead. >> Yeah. So, one thing that I think is is helpful, and this is coming kind of from the the leadership side of things, but I think one of the responsibilities of a a senior or or more senior engineer is to learn how to advocate for the work that needs to be done. It's not enough just to understand that it needs to be done. It's not enough to create a ticket for it. You kind of have to learn how to talk the business language to be able to explain why something is important. And maybe you're selling that to your manager who's then going to sell it to a product team or to a leadership team. But learning how to be really articulate about both the I guess the pain or the the impact of leaving a situation as it is and that can be cost to the road map that can be distraction of engineers that can be burnout that can be uh you know product experience there's all kinds of different uh areas where there's impact in this. Uh but it's it's not the manager's job. It's not you know it is my you know job as a CTO but I also push that back on to the engineers who really have the context about why something is happening over and over again or you know the the reason that we need to fix it. And it's really your job as an engineer to be able to make the case for why why we should work on something. >> Dave, >> I could spend days talking about this subject because it's really interesting. But I I a few years ago we went through a revamp internally of how we handle post incident actions which I think was really successful. But one thing that we discovered was it was a local optimization meaning within the incident itself we were actually pretty good about tackling the the high priority items afterwards. But what we didn't what was missing and what I think I you know I'm also open to ideas on this is what about patterns? What about things across the major incidents? Like who's looking at like systemic things that are coming across that may be not within the local of that one incident but may be recurring across these incidents as well. And I think that's something that sometimes those are the big ones, right? Like because you can look at the big you can look at the incident itself and often yeah maybe it was a major outage but usually there's something it's tying back to something else. And that's the that's the other thing. You have to not just look for the you know the small ticket like and which is really important but like what is the big pattern that we need to tackle architecturally that may prevent some of these incidents from occurring in the future. >> Yeah, that's great. And I want to move to a question from the audience. Uh I I think it was you Dave that mentioned dashboards and like all this information. So, Florian, hi Florian asks, "One key factor in dealing with incidents is live business metrics. Any learnings you'd like to share on how you came up with or used a business metric in a novel way? That's a I I think ideally and I I don't know if we're there yet, but ideally you if it if it's a business problem, then you should have some easy way to discern like a tie ar tie your outage to ARR, right? Like see like what's the revenue impact of the thing that that's happening right now and who's being impacted is it like our crown jewel customers like is there depending on if you have sharding or if you have like some sort of dis how how where is the impact and who is it impacting and and I'm not saying practically speaking like all customers are great but some customers are more valuable than others to your business right so like I think that's the end goal would be to and it's hard to get right because you need distributed tracing you need to have some sort of like account linking and it's very it's very complicated but I think ultimately that is the goal and in in in the meantime I think you can probably use like some sort substitute uh kind of metrics in in between. So if it's not the customer ID, it's it's it's it's something that's related to customers, but ultimately it's about the end users. It's about what is the customer impact. So that is the thing I'll be focusing on. So because if you're like if your CPU on one of your clusters is at 100% but there's like it doesn't it's there's no customer impact then it's not I'm not saying it's not a problem but it's not an urgent problem to solve. So I think that that that's the that's the thing I've seen change over time is like from the Nagios days of like I got I got a PEG CPU I got to fix it to now people are thinking more in terms of customer impact >> hardship. Yeah, I mean completely agreed with Dave like like if you see something like a 5% error rate you're like okay sure not great but also not catastrophic but if you see like oh 100 customers are impacted now suddenly that is a different story right so it's it's more along the lines of figuring out what is your like what's the easiest way for you to figure out your blast radius like some sort of a blast radius dashboard in terms of your actual impact in terms of business or customers because like yeah 100% CPU to uh like shouting at you, you're like okay sure there is something I need to do but is it actually impacting you probably don't know so yeah >> and just to add on one last thing to this but uh there are tools that allow you to instrument this so that the tool itself will tell you how many different customers have experienced a particular error or particular bug and you can set thresholds for alerting if you know the bug is experienced or if the errors experienced by you know whether it's number of users whether it's number you know if you tenant by customer or something like that but as long as you have the ability to identify you know which end user was affected by an error or you know what is the the meta information that caused that. You can build a lot of instrumentation that will allow you to notify on this very specifically and to you know cause a page if a a threshold was breached. >> Thank you all. I'm going to take another audience question. This one I'm This one is very fun. How do you get your engineers excited about S sur and putting the effort into aspects like monitoring? >> Can I jump in here on this one for a second? Yes, please. >> I don't I don't know if I have an answer for the first part because not everybody loves being on the front line. But for the second part, my answer is that everybody is responsible for keeping the the app online. Everybody needs to be responsible for monitoring, for operations, for those kinds of things. And the way that I have done that in my last several organizations is everybody's on call and you share an on call rotation. And when you're on call, well, you start to learn to care about things like monitoring and logs and alerting because it affects your life on a day-to-day basis. And so those things do start to get better when everybody has a responsibility. On the excitement side, harder to assign that. So, I don't know if Arjett or Dave have ideas on on how to drive that. >> I agree with everything you said because I think it's more self-preservation than excitement. Like, how many times do you want to be woken up at 3:00 in the morning, right? like it's not healthy for you and it's not frankly it's not healthy for the business because if you're getting woken up presumably some customers being impacted. >> Yeah, for sure. I think I've honestly stopped trying to get them excited about monitoring and I'm just I've moved from excitement to more like like just making them angry about not having it like and like Dave said, right? If you wake up at 3:00 a.m. and you're like, "Oh, I wish there was that dashboard which would have told me how many customers are getting impacted." Now you are angry. you want to go and figure that out because you spent 1 hour waking up at 3:00 a.m. just to find something which if that if that would have existed would have taken you 5 minutes to figure out if this is a real issue or not. So I think at least for me I've moved a little away from excitement after failing around it and now more along the lines of how to get some emotion out of it. >> You you'll also build better tools when you're building them out of need rather than building them out of theory. You know, if you have a team that's just responsible for building all of your internal tools, you'll end up with some really great tools and a bunch of tools that nobody ever uses because they were built with some theoretical backing rather than as a response to a particular painoint. >> I guess the perennial question is how do you get an engineer excited about anything? But we'll save that for another panel. Um, okay. So, let's see. Arjett, when does on call fatigue stop being an individual problem and become a systemic one and then how do you work with that? Uh I mean I think like on call fatigue sort of becomes systemic the moment like you hear one person or like a bunch of engineers talking about that being a factor in their decisions in their career decisions in the projects they're thinking about or just their fatigue itself right I think a lot of people get caught up in oh we would do like a full-fledged burnout survey or a retention metric or some of that but I mean your signals can be soft your signals can be small that hey people are talking about it they're thinking about not taking up a project, not being on a team or just being tired of it. And I think I mean that is where the question around like just the fairness of how do you improve some of this on call right and like your second question how do we improve the systemic part of it I'd say one we need to figure out like are your engineers getting paged multiple times in a single on call shift and I think if if they're getting multiple times paged in a single night itself you know people are already starting to dread their on call week and like some sort of a rec- recovery time and then are your incidents even predictable that you know it's a deployment change or it's a config change or is it random where you will have to bring in 10 teams to figure something out because those are the ones that take a lot more out of you and like that that sort of starts compounding your fatigue and creates that helplessness right so I think for us what has sort of worked is like of course we improve on tooling but like also just acknowledging sustainable on call means that like sometimes it will it will impact of course your sleep and everything else but is that is that like distributed well enough or is there like good enough cognitive load distribution and if that's not the case and like I think Spencer was saying out you need 20 people in an incident to resolve something you have a lot more deeper systemic issues rather than the lack of a particular tool or one person not being involved Yeah. And I'm actually curious, this is something that we discussed prior to the call, how in places like Germany, you actually cannot mandate that everybody is on call like Spencer is doing in his or I I'm American so I've now worked in both ecosystems and it's fascinating. So I'm curious like if anybody has insights on, you know, on how to work with that constraint. >> I mean I'm in Germany right now so I can probably at least some some part of it and I think which is where exactly your question becomes even more important Asha because you have to have better systems in place. You cannot be firefighting all the time if you have a limited pool of engineers also there. So like you have to at some point in time you have to sit down and like hey do we need to invest in SLOs's is a 2% CPU higher than CPU threshold worth bringing an engineer into this and like let letting them spend time on this or is this going to auto resolve in 15 minutes. So I think you start thinking about things in a lot more longer fashion because you cannot use all of your engineers as a defense mechanism to all the different alerts that are happening. And I think that like as as an example last year at some point in time we had only three people in our rotation and this was still covering like a bunch of services. Now within three of us if we have to take care of seven or eight different services we cannot be reacting to everything. So you pick your battles carefully and then you navigate that and then you the other part of it is you figure out what is causing people to not be a part of on call because these are services that they are maintaining and that is the part which is your systemic issue which you tend to like help them resolve build SLOs's build confidence tell them that this work is appreciated it's not extra burden work that you or your manager are just piling on these people right let people let compensate people if possible right? If monetarily or by compost or whatever things are applicable in your company. So at least for us there is monetary compensation and then if you're on call for a longer night or for a weekend then you can of course talk to your manager get a day off like figure those things out because if it's not appreciated you're not going to have a healthy set of people who are going to be interested in doing this. >> Yeah, that's that's great advice. And I guess on a similar note like when we don't have that constraint like Spencer you have everyone on call but even though everyone's on call you might still implicitly like overrely on certain people to be the fixers not just rather than the triagers. So as your team is scaling since you work at a startup how are you preventing that reliance on like a a certain number of engineers while maintaining strong ownership? It's a tough question to answer because I think the reality is that certain engineers do tend to establish themselves as the experts for systems, for databases, for the types of things that tend to break when you have a an incident. You know, it's rarely a front-end component that's acting in a really mysterious way. It's you know usually something to do with the connection between distributed systems or you know a load that uh is higher than you were anticipating which tends to tends to be you know a few engineers. I think the the main thing for me is for those engineers that tend to find themselves in let's call it like lynchpin situations where they are the one engineer who knows I try to work with that engineer to make one of their particular goals to to be to spread that knowledge. So it's me for me it's like a three-step system. You you start off as the expert. You're the lynch bin. You're the person who really has to be looked at to fix. And I think it's inevitable that you will have situations like that on your team. I don't think you can just permanently avoid that because creating new software, creating new systems tends to involve somebody doing something either novel or something that the rest of the team doesn't know, you know, from day one. But then the job is to then go from being a lynchpin to let's say a knowledge sharer. You bring other people in into that. So, you know, as a leader, I try to assign different people to the same systems. So maybe I have somebody who's the, you know, the Stripe or the billing or the subscriptions expert. The next time we have a small project there, I'm going to assign somebody who is not that person. Not because it will get done faster, but because we will start to spread that knowledge. And I think being willing to take a slight penalty in velocity in order to avoid having the exact same people responsible for a single system can be helpful. And then you can move from being, let's say, a a knowledge sharer to being a multiplier. And I think that's the really the end goal for a lot of these really generally more senior engineers is how do you move from lynchpin to ultimately knowledge sharer to then multiplier where you're not just you know spreading the knowledge of what you've done but you're enabling other people to create systems in a similar way. >> Thank you. I'm going to take another question from the audience that I love. So this I'm gonna paraphrase this one. Uh this person is asking how do you treat the first occurrence of an incident because in their experience and this has also been my experience. Um they take it to like leadership or product manager and and the person replies well how likely is this to reoccur? Um and it's usually hard to answer because in incidents are inherently unpredictable. So, how do you how do you usually answer that? How do you treat that first occurrence and like maybe provide the right level of risk to a stakeholder that doesn't really understand how systems inherently work? And usually they're looking for the answer of like, oh, it's low probability, so I don't have to fix it. That's what they want to hear. Uh, how do you work with that? >> The the first thing that comes to my mind here is this is really about trade-offs. It's about risk analysis. And I I don't even know if the frequency or the likelihood of something recurring is the most important uh metric or the the most important factor here. But if it does recur, what is the impact to the business? Is it going to mean is it system has a major outage and is down for many customers or for the entire customer base? If so, even a very small chance of recurrence is probably worth addressing. if it's a minor nuisance and you don't think it's going to recur, you know, I think there can be a business case to either deferring or I I mean, honestly, I like a wait and see approach sometimes. You know, maybe you don't know how likely it is that something's going to recur and it's not that big of a deal and so you do say, "We're going to defer this, but if it happens again next week, then we're going to put some time on the calendar and make sure that we fix this." Yeah, I think I absolutely agree with Spencer and I think I I also feel like the probability of reoccurrence is sort of a misleading question. Like it's one of those things that should I get a fire insurance but my house has never caught fire, right? Like I mean you're not going to get a fire insurance after the house catches fire. So I feel the conversation needs to be like what did this incident cost you? And sometimes the cost of the fix versus the cost of if this happens again, right? That is a simpler trade-off like because the fix could mostly be a lot more cheaper than bringing 20 engineers into an incident call managers, directors, whatnot. And if you literally like start figuring out the cost of managing that incident, you realize, you know what, the fix is probably much cheaper rather than doing the whole shebang. Yeah, that's great. I think in general sometimes we end up in a state of paralysis overanalyzing when just doing the thing would be the faster thing and make everybody feel better also. So that's a great point. Um I want to ask a little bit about leadership. So Spencer already mentioned a little bit of like what needs to happen after an incident to make space. Dave, I'm curious to hear from you. Like what else do you think engineering leaders should be doing differently to make on call sustainable? How do they protect remediation time? How do they stand up for their teams and the people that are actually trying to improve these systems and the on call process? Um especially like when we all have a bunch of planned roadmap work that we have to negotiate. >> Um this is tough because I think this is as leaders it's always a problem, right? because you have a people issue, right? If especially when you're close to burnout for a team or when you're seeing a lot of on call, but and and even though it is a human problem, I think you need to frame it as a business problem. So because that's essentially and it's not to say that leadership is unempathetic or not human, but you need to when you're talking about trade-offs and you're talking about investments, you need to understand what the cost could be to the business. So to me if half my team is burned out I can actually quantify that with metrics in terms of like hey what's the if the attrition happen if this kind of attrition happens this is the impact of the projects that and deliverables that we're going to have so I'd like to prevent that or if my team is getting paged a lot that's not just a human problem that's that also implies that our systems are unstable so what is the customer impact of that so what you can do is actually put an argument together that's based on business value and and business outcomes and that then feeds into actually helping fix the human problem because if you just if you just go and say like my engineers are burned out it's not I mean even if it if someone was coming to me I would be really empa sympathetic to that but it's it's hard because then you know you're a CTO you're probably talking to your product counterparts all the time you're probably talking to the CEO and they're giving you pressure for like hey I wanted you know we got this product stuff to release over here and so if you're not able to make the case at that then I don't think you're going to get the outcomes that you need. But I think you can make the case at that level because the burnout actually does lead to business issues and instability actually does lead to business issues. But that's the level you need to be talking at when you're making those arguments in my opinion. >> Yeah. Anyone else have thoughts on leadership's role in all of this? I think you know and I'll look at myself here but certainly if you're in an executive seat you do share responsibility for making sure that you are advocating for things that will help your team work in a sustainable way and you know while I do look to my engineers senior engineers staff engineers etc to help me make that case ultimately I think it is you know the leadership's responsibility to you know avoid putting engineers in a situ situation where they're going to burn out. Uh or avoid putting people, you know, Arjett talked about multiple pages within the same on call rotation, you know, like do you have a contingency plan for what happens if you have an incident that is recurring several times in a row? Can you run rotation early? You know, is there a way to do that so that you don't have the exact same people responding day after day? Uh, you know, one of the things that we'll do is for our our known busy periods, we work in e-commerce and and marketing. So, Black Friday, Cyber Monday, some big holidays, they're just busier. They're higher stress. We have the highest load we'll have for the entire year. So, we have a different on call rotation. It's more like I think it's a 24-hour rotation rather than a week-long rotation for those periods because it's much more stressful and if you just draw the wrong straw and you happen to end up with the worst on call rotation, that's not great. So I think it's really business to business, but can you analyze the different patterns for your business and come up with something that doesn't, you know, randomly put people in a situation that they're much more likely to to be stressed or to, you know, have a lot of higher higher stress incidents. >> Okay, I'm going to take this might be our last question. Let's see uh another one from the audience. So, incidents affect reputation and trust. How do you maintain that trust with your customers after an incident? Um, do you call out your vendors in the I'm not sure what RCA stands for. Sorry. Maybe someone else does. >> Maybe. >> Oh, yeah. Okay. Acronyms are hard. Okay. Do you call out your vendors? What other te techniques do you use to maintain trust? >> I have I have opinions about this. I first off never call out your vendors. You you made the choice to select that vendor, right? That's your responsibility. So ultimately, yes, they may have contributed to whatever it is that you did, but you made that choice, right? So, you know, you don't abdicate responsibility by handing it off to someone else. That's I feel pretty strongly about that. And secondly, I think be transparent. You know, I think that I was pretty proud. We had a really significant incident back in uh end of August, beginning of September and I was really proud of how we and the team came together to publish probably our best postmortem doc that we have in like the last four years and we just decided to approach it from a very transparent and technically rigorous perspective. So we had like five, you know, I was in a room with like principal and staff engineers saying like how much do we want to describe about the way our systems look and we we just said let's do everything we can like let's be honest about what we what broke how our architecture is like where the failure points were here and what we're going to do to fix it next time. And I think that transparency and honesty actually helps build trust because I I I know that I feel the same as someone who manages a lot of vendors. I that's how I feel when I want to see a a postmortem doc or an incident review from one of my vendors. I want to see I don't want to see just like two sentences thing going like oh you know kind of papering over it. I want to see here's a here's a real technical issue we ran into and here's how we're going to prevent this from happening again. >> Thanks Arjette. >> Yeah I mean huge plus one to Dave. I I I mean from a customer point of view I don't think the customers don't they don't actually care whose fault it was. They care whether if you are going to let that happen again. If you go ahead and write in your like incident that hey you know what AWS went down and that is what happened. It's the same problem that from the customer's perspective they have a contract with you not AWS. So AWS going down sure it's your problem not their problem. So I I do think like completely huge plus one to Dave and that is exactly how I feel. So they have basically put in your trust to have resilient systems and vendors are always going to be imperfect. So that's not something you can probably escape. >> I'll add not to Dave and Arjett's answer which was perfect. Don't call out your vendors, but it's really important to build relationships if you're on the engineering side with the go-to market counterparts. And I think one of the most important things you can do to build trust is to learn how you communicate incidents, downtime, other unexpected scenarios directly to the affected customers. And so for for Privy, we have what we call support on call or support alert. I can't remember what it's called. It's in our Slack channel. And we will tag our support alert team anytime there's something that affects customers. And the goal is not that we have paged your duty and that you know support is in this on call rotation, but more that as we get into a spot where we're actually trying to communicate the impact or the expected duration or or something like that to somebody who is affected by the incident. The engineering can describe it however we will. you know, we're going to use engineering terms and then it can be translated by a customer success rep who has a relationship with that customer already. We also have a status page and those kinds of things. But we found that generally, you know, especially more complex or ongoing incidents, it's much better to have just that onetoone communication either from our support team and you know, we'll measure an incident by support volume as well. You know, so when we have an incident, it's like, okay, well, how many tickets have we had about this? And for any real incident, you see that support volume start to go up. So really important to be able to enable your support team or empower your support team to give good answers that don't beat around the bush, but also, you know, you don't reveal confidential information. And and the more you can do to to help that uh that group out, the better. >> Yeah. And I think just to round that out, I'll say that what Spencer just said is where, you know, showing accountability and ownership is where you can shine as a leader. That really separ separates like managers from true leadership. Um, we have one more amazing question, but we're not going to have time to answer it. And so, please take it to the the chat. Um, oh, wait, actually, we have two more minutes, so I'll I'll throw it out for someone to give a lightning round answer. What do you consider a good ratio of on call engineers to application domain etc. I mean I'd say six to eight engineers is probably okay but it's more about the like if you are an engineer who's touching that system every single day and you're on call once in every two months it's probably fine but if you're an engineer who touches that particular system once in six months and you're on call every two months that basically means you're literally relearning the system again during your on call shift right so I think it's also a little more tied to how frequently you work on that system and how frequently are you on call for that And I think it's also has to do with how operationally excellent your team is and how wellmaintained the services are because I gota have a team with eight people on it with 40 services and on call is quiet because we you know you're proactively taking care of those issues ahead of time. So I think it's really hard to say in the abstract that there's a magic number. I think you could have like a one team of eight people with one service and that service is paging you every day like seven times a day. then that's bad, right? So, I think it it really depends. >> I'll take this from the >> Sorry, go ahead. >> I was gonna wrap up, but if you have any last thoughts, Spencer, >> I was just going to take this from the other side, which is I think it's important to have enough people on call that the rotation isn't every other week, you know, or something that's unsustainable. So I would rather have a rotation that has responsibility for say a wider surface area of of services and has enough people that you're not on call more than let's say one every six one out of every six weeks than one where everybody has a really deep knowledge of everything that they are operating and you're on call every other week. >> Yeah, thank you for that. and and I think our friend uh Simon who asked this question. If you have like a more specific question about your scenario, great thing to bring up in the Slack channel as well and you can get like real advice from this group. So, thank you all so much for the good insights. I definitely learned some things. So, uh we're going to head over to the panel Q&A channel in Slack. We just messaged it in the chat in case you didn't catch that. And uh yeah, please head over there to continue the discussion. Thank you for joining. Thank you for our panelists. and have a great day. Bye., Get Smarter With Every Incident, YouTube, 01kg84raq2mxxwa2yyfpbke5hk.