Get smarter with every incident - CourtneyR.dev's 2nd Brain

Get smarter with every incident 🔗 **Source:** [Get smarter with every incident](https://www.youtube.com/watch?v=lC8VkQRGzeE) > [!note]+ Document Notes > tags: [incident-management, on-call, site-reliability, devops, leadership] > author: The Lead Dev > source: YouTube > reference: https://www.youtube.com/watch?v=lC8VkQRGzeE > date created: 2024-12-19 16:45 > > # 🏷️ Title: Get smarter with every incident >  ## 🧰 Action Inbox - [H1] Implement AI-powered anomaly detection with notifications only, not pages — early warning systems #incident-response - [H2] Create specific incident response roles to reduce triage time — incident commander pattern #leadership - [H3] Treat post-incident actions as features, not tech debt — roadmap integration #process-improvement ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [The Lead Dev Panel Discussion](https://www.youtube.com/watch?v=lC8VkQRGzeE) ## 📝 Summary - What happened: Engineering leaders from PagerDuty, Privy, and Delivery Hero discussed breaking out of incident firefighting cycles - Why it matters: Teams stuck in reactive mode burn out and fail to address underlying system issues - Key stakeholders: Engineering managers, SRE teams, on-call engineers, business leadership - Risks and tradeoffs: AI tools reduce noise but require careful data curation; shared on-call builds ownership but may not be legally mandated everywhere - Bottom line takeaway: Transform incidents from reactive firefighting into systematic learning opportunities by treating post-incident actions as roadmap features ## 📖 Main Content The panel explores how engineering teams can evolve from reactive incident response to proactive system improvement. Key themes include using AI for early warning systems while avoiding alert fatigue, establishing clear incident response roles to speed triage, and most critically, treating post-incident learnings as product features rather than deprioritized tech debt. The discussion covers practical constraints like German labor laws that prohibit mandatory on-call rotations, forcing teams to build more resilient systems. Panelists emphasize measuring business impact rather than just technical metrics, and stress the importance of engineering leaders advocating for remediation work using business language rather than technical arguments. Trust-building with customers after incidents focuses on transparency and accountability rather than vendor blame. The conversation addresses systematic patterns across multiple incidents, not just individual post-mortems, as the key to architectural improvements. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/lC8VkQRGzeE?t=0) — [H1] Panel introduction and context - Key idea: Moving from firefighting urgency to systematic incident learning - Evidence: "Teams get stuck in firefighting, burnout occurs, and we don't necessarily resolve the underlying problem" #### ⏱️ [02:30](https://youtu.be/lC8VkQRGzeE?t=150) — [H2] AI for early warning systems - Key idea: Use AI tools for notification, not immediate paging until properly tuned - Evidence: "We're defaulting to notify rather than page until we figure out the right way to tune these tools" #### ⏱️ [05:45](https://youtu.be/lC8VkQRGzeE?t=345) — [H3] Data quality drives AI effectiveness - Key idea: AI output quality depends on clean input data excluding staging/dev environments - Evidence: "If you're sending everything from staging, from development environment, from prod to the same place, it's going to start anomaly detecting things that aren't actually problems" #### ⏱️ [08:15](https://youtu.be/lC8VkQRGzeE?t=495) — [H1] Triage optimization through role separation - Key idea: Assign specific roles during incidents to prevent everyone going heads-down - Evidence: "Having very specific roles for responders can help a lot with triage... incident commander level or scribe level" #### ⏱️ [11:30](https://youtu.be/lC8VkQRGzeE?t=690) — [H2] Focus on customer impact over technical metrics - Key idea: Prioritize incidents based on customer impact rather than CPU usage - Evidence: "If your CPU is at 100% but there's no customer impact, it's not an urgent problem to solve" #### ⏱️ [14:20](https://youtu.be/lC8VkQRGzeE?t=860) — [H1] Treating post-incident actions as features - Key idea: Change from "what should we fix" to "smallest thing to make this less painful next time" - Evidence: "We changed our whole flow from what should we fix to what's the smallest thing that I can do to make this less painful next time" #### ⏱️ [18:45](https://youtu.be/lC8VkQRGzeE?t=1125) — [H2] Business impact metrics for incidents - Key idea: Tie outages directly to revenue and customer count rather than abstract technical metrics - Evidence: "5% error rate you're like okay sure not great but if you see 100 customers are impacted now suddenly that is a different story" #### ⏱️ [22:10](https://youtu.be/lC8VkQRGzeE?t=1330) — [H3] Shared on-call responsibility drives ownership - Key idea: Everyone on call creates self-preservation motivation for better monitoring - Evidence: "When you're on call, you start to learn to care about monitoring and logs because it affects your life" #### ⏱️ [25:40](https://youtu.be/lC8VkQRGzeE?t=1540) — [H2] Legal constraints force better systems - Key idea: German labor laws preventing mandatory on-call require more resilient architecture - Evidence: "You cannot be firefighting all the time if you have a limited pool of engineers" #### ⏱️ [28:55](https://youtu.be/lC8VkQRGzeE?t=1735) — [H1] Knowledge spreading to prevent lynchpin dependencies - Key idea: Move from lynchpin to knowledge sharer to multiplier through deliberate assignment - Evidence: "Assign different people to the same systems... take a slight penalty in velocity to avoid having the exact same people responsible" #### ⏱️ [32:20](https://youtu.be/lC8VkQRGzeE?t=1940) — [H2] Risk analysis over probability for first incidents - Key idea: Focus on business impact of recurrence rather than likelihood prediction - Evidence: "Even a very small chance of recurrence is probably worth addressing if it means major outage" #### ⏱️ [36:15](https://youtu.be/lC8VkQRGzeE?t=2175) — [H1] Leadership framing incidents as business problems - Key idea: Frame burnout and instability in business terms to get leadership support - Evidence: "You need to frame it as a business problem... burnout actually does lead to business issues" #### ⏱️ [40:30](https://youtu.be/lC8VkQRGzeE?t=2430) — [H2] Customer trust through transparency and accountability - Key idea: Never blame vendors in post-mortems; own the vendor selection decision - Evidence: "Never call out your vendors. You made the choice to select that vendor, that's your responsibility" #### ⏱️ [43:45](https://youtu.be/lC8VkQRGzeE?t=2625) — [H3] On-call rotation sizing based on system familiarity - Key idea: Balance rotation frequency with system knowledge rather than fixed ratios - Evidence: "If you're an engineer who touches that system once in six months and you're on call every two months, you're literally relearning the system" ## 📚 References - [PagerDuty Platform](https://www.pagerduty.com/) - [Sentry Error Monitoring](https://sentry.io/) - [New Relic Observability](https://newrelic.com/) - [The Lead Dev Community](https://theleaddev.com/) ## 🗣️ Paraphrase - AI tools help with early incident detection but need careful tuning to avoid alert fatigue - start with notifications before graduating to pages - Structured incident response roles (commander, scribe, investigators) prevent chaos and speed up resolution compared to everyone researching simultaneously - Post-incident actions fail when treated as tech debt competing with roadmaps; success comes from framing them as product features solving business problems ## 📰 Newsletter 🚨 **Level Up Your Incident Response Game** Tired of the same incidents waking you up at 3 AM? This panel from The Lead Dev shows how to break the firefighting cycle and actually learn from every outage. **Key Takeaway**: Stop treating post-incident action items like tech debt. Frame them as product features that solve real business problems - they'll actually get prioritized and completed. **Quick Wins**: - Set up AI anomaly detection for notifications (not pages) while you tune thresholds - Assign specific incident roles: commander, scribe, investigators - no more chaos calls - Measure customer impact, not just CPU usage - 100 affected customers trumps 100% CPU **The German Advantage**: Teams in Germany can't legally mandate everyone on-call, forcing better system design. Sometimes constraints breed innovation! **Trust Builder**: When incidents hit customers, own your vendor choices completely. Transparency about technical failures builds more trust than finger-pointing. Resources: [PagerDuty Documentation](https://support.pagerduty.com/), [Incident Response Guide](https://response.pagerduty.com/), [SRE Workbook](https://sre.google/workbook/) ## ✅ Action Items #### DevRel - Developer education: Create incident response workshops covering role assignments and business impact measurement - Developer experience: Build dashboards that surface customer impact alongside technical metrics - Community management: Facilitate knowledge sharing sessions between teams to prevent lynchpin dependencies #### OSPO - Policy & Governance: Establish incident response policies that comply with local labor laws regarding on-call requirements - Security Management: Integrate incident learnings into security threat models and response procedures - Community Engagement: Share anonymized post-mortems with upstream communities to improve ecosystem reliability #### Community - Maintainers: Provide incident response templates and role definitions for open source projects - Contributors: Create onboarding materials that include operational responsibility expectations - Sponsors: Fund development of incident response tooling and post-mortem process improvements #### WordPress - Core: Implement business impact metrics for WordPress.org service incidents affecting plugin/theme developers - Core Performance: Create early warning systems for performance regressions that could impact hosting providers - Community: Develop incident communication templates for WordCamp organizers handling event disruptions - Training: Create lesson plans covering incident response best practices for WordPress agencies - Support: Build escalation procedures that include customer impact assessment criteria - Meta: Establish transparent incident reporting for WordPress.org infrastructure affecting developer workflows ## 🧪 Hypotheses - Teams using AI-powered anomaly detection with notification-first policies will reduce alert fatigue while maintaining incident detection speed - Organizations implementing structured incident response roles will reduce mean time to resolution by 30-50% - Treating post-incident actions as product features rather than tech debt will increase completion rates from typical 20-30% to 70-80% - Engineering teams in jurisdictions with on-call labor restrictions will develop more resilient systems due to constraint-driven design ### 📊 Methodology - Study design: Compare incident metrics before/after implementing structured response roles across multiple organizations - Data sources: PagerDuty incident data, post-mortem completion tracking, burnout surveys, customer satisfaction scores - Experiments: A/B test AI notification thresholds vs traditional alerting across similar service teams ### ❓ Questions - How can we quantify the business value of incident prevention work to justify roadmap prioritization? - What specific legal frameworks around on-call work create the most effective constraints for system design? - How do we scale knowledge sharing practices to prevent lynchpin dependencies as teams grow rapidly? ### 🔮 Future Research - Investigate correlation between post-incident action completion rates and long-term system reliability improvements - Study effectiveness of different incident communication strategies on customer trust and retention - Explore optimal team sizing and rotation patterns for various types of system complexity ### 💡 Implications - Practice: Incident response becomes a structured discipline with defined roles and business impact focus rather than ad-hoc firefighting - Policy: Labor regulations around on-call requirements may drive better system architecture and reliability practices - Security: Incident response patterns apply directly to security incident handling and post-breach learning - Education: Engineering curricula should include incident response, business communication, and operational responsibility - Funding: Post-incident remediation work needs dedicated budget allocation as product features rather than technical debt ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Panel discussion format with multiple speakers - Improve: Add incident flow diagrams, role responsibility matrices, customer impact dashboards #### 🔊 Aural learners - Present: 45-minute audio discussion with Q&A - Improve: Chapter markers for key topics, supplementary podcast episodes #### 🗣️ Verbal learners - Present: Clear definitions of incident roles, business terminology - Improve: Glossary of incident response terms, analogies for technical concepts #### 🧩 Kinesthetic learners - Present: Action-oriented advice, specific implementation steps - Improve: Incident simulation exercises, hands-on tooling workshops #### 🧠 Logical learners - Present: Business impact analysis, cost-benefit discussions - Improve: Decision trees for incident prioritization, metrics frameworks #### 🧑‍🤝‍🧑 Social learners - Present: Panel format with multiple perspectives, audience Q&A - Improve: Follow-up discussion forum, peer learning groups #### 🧘‍♀️ Solitary learners - Present: Individual takeaways, self-assessment opportunities - Improve: Personal incident response checklists, reflection questions #### 🌿 Naturalistic learners - Present: System ecosystem thinking, organizational patterns - Improve: Incident lifecycle models, system health metaphors ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Video has speakers identified, structured conversation - Improve: Full transcript with speaker attribution, chapter headings #### Keyboard navigation - Present: Standard YouTube player controls - Improve: Transcript navigation, timestamp jumping #### Contrast and readability - Present: Standard video player interface - Improve: High contrast transcript options, adjustable font sizes #### Text resizing and reflow - Present: Responsive video player - Improve: Transcript formatting that reflows properly #### Media and captions - Present: Auto-generated captions available - Improve: Professional captions with technical term accuracy, downloadable transcript #### Forms and interactions - Present: Basic video controls and comments - Improve: Accessible feedback forms, structured Q&A interface ### 🌍 DEIB Assessment #### Cultural breadth - Present: International perspectives (Germany vs US labor laws) - Improve: More diverse geographic examples, cultural approaches to on-call work #### Inclusive language - Present: Professional technical terminology - Improve: Plain language alternatives, avoid cultural idioms #### Neurodiversity support - Present: Clear structure with time-bounded segments - Improve: Content warnings for stress-related topics, processing time considerations #### Representation - Present: Mixed gender panel, leadership roles - Improve: More diverse company sizes, industry representation #### Psychological safety - Present: Respectful discussion of burnout and stress - Improve: Explicit acknowledgment of mental health impacts, resource links ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Business impact → investment priority (strong logical connection) - Inductive strength: Multiple company examples support general principles - Causal reasoning: Clear cause-effect relationships between practices and outcomes - Analogical strength: Fire insurance analogy effectively illustrates risk management #### Fallacies - Appeal to authority: --- ## Full Transcript [00:12] All right, let's get started. So, hello [00:14] everyone. My name is Najla. I am a [00:17] director of engineering at a company [00:19] called Aligned. I'm calling in from [00:23] Berlin, Germany. And I'm also an [00:26] engineering leadership coach. and I'm [00:28] really excited to have this panel today. [00:30] Um, this panel is going to be about 45 [00:33] minutes uh where the panelists and I are [00:36] going to discuss some questions. Uh, but [00:38] we're also really open to you writing in [00:42] questions in the chat. So, please get [00:43] those in and I'll try to weave those in. [00:46] And if anybody has any questions at the [00:48] end, we'll all head over to the lead dev [00:50] Slack in the panel Q&A channel which [00:53] I'll remind us of at the end. [00:55] Um, so first let's get started with [00:57] introducing the rest of our panelists. [00:59] I'm going to start with Dave. Dave, [01:02] hello. [01:03] >> Hi everyone. Uh, thank you for the intro [01:05] the short introduction. I'm Dave. I [01:07] manage two [01:10] reliability teams here at Pager Duty [01:12] responsible for everything from CI/CD [01:15] tooling to uh our Kubernetes clusters [01:18] and network core networking etc. And I [01:20] also manage our developer experience [01:23] function here as well. [01:27] Thanks Dave Arjett. [01:30] >> Thanks. Thanks. So yeah, I'm Arjett. Uh [01:33] I am a senior engineering manager with [01:35] delivery hero in Berlin as well. So I [01:37] lead the engineering for the [01:39] quickcommerce side and [01:41] uh I'm here to talk about incidents and [01:44] I've spent a lot of time thinking about [01:45] incident response. So uh yeah, I'm here [01:48] to talk about that today. [01:51] >> And last but not least, Spencer. Hey [01:54] everybody. Uh my name is Spencer. I'm [01:56] the CTO at Privy. We do uh marketing [01:59] automation and email and SMS. And I've [02:02] spent a lot of time both in and managing [02:05] incidents. Uh excited to talk with you [02:07] all. [02:10] >> Great. So yeah, the title of this panel [02:12] is called get smarter with every [02:13] incident and we have a nice group of [02:15] panelists here with a variety of [02:17] experience and urgency uh really keeps [02:21] systems alive during an incident. That's [02:23] kind of the baseline level that we're [02:24] all used to operating in. But when that [02:27] really becomes the sustained default, [02:29] teams get stuck in firefighting, burnout [02:31] occurs. Um, and yeah, we not don't [02:36] necessarily resolve the underlying [02:38] problem. So, we're going to get into [02:39] that today. How to break out of those [02:41] loops, how to go from reactivity to um [02:44] actually maybe fixing some things in the [02:46] on call systems and where AI is [02:48] involved. That's a pretty hot topic [02:50] right now. So, first I want to start [02:52] with Spencer. Spencer, can you talk [02:55] about how you're using AI to catch early [02:57] warning signs and tune alerts? [03:01] >> I think this is the dominant topic [03:04] really of the year and probably the last [03:06] several years and it's pervading every [03:09] part of of software engineering. And so, [03:11] you know, when we start to think about [03:12] incidents, it's, you know, natural and [03:15] inevitable about how we can use AI to [03:19] support those incidents, how to detect [03:21] things earlier. And I think that's one [03:22] of the areas that we've really been [03:23] focused at Privy is figuring out how we [03:26] can use AI tools. And we're really using [03:29] the tools we already use, which have a [03:31] lot of these uh kind of AI layers built [03:33] into them. You know, for us, our stack [03:36] is is New Relic and Sentry and Pedag [03:39] Duty. Shout out to Dave's team. Uh but [03:43] uh things like uh Sentry right now have [03:46] a lot of the AI tools already built in [03:48] and so we can turn those on. And one of [03:50] the ones that we use right now is [03:51] anomaly detection. And it's cool, but [03:53] there's a lot of anomalies. And I think [03:55] one of the challenges for us is figuring [03:57] out how do we tune [03:59] the the error detection or this kind of [04:02] AI [04:04] uh alerting so that we are not over [04:07] alerting that we're not creating a bunch [04:08] of noise for our team. And that's [04:10] something we haven't nailed yet. You [04:12] know, I think for us uh we are still [04:14] defaulting to right now we're defaulting [04:16] to uh a lot of these AI tools to notify [04:20] rather than page. And so that's our [04:22] default. That's the kind of I guess the [04:23] pragmatic choice we have made is that [04:26] until we figure that the right way to [04:28] tune these tools, we'll notify. So [04:30] we'll, you know, notify our operational [04:31] Slack channel, but if it's 3:00 a.m. and [04:34] we get an anomaly detection, at least [04:35] right now, unless our other uh [04:38] monitoring tools also go off, we're not [04:40] going to be paging the engineer on call. [04:46] >> Uh, go ahead, Dave. Right. The that I'm [04:49] going to try not to talk too much about [04:50] page duty, but in terms of AI, how AI [04:53] fits into incident management, it's [04:55] something we've been thinking about for [04:56] a long time, starting with like the [04:58] basics, like how do we group alert, use [04:59] AI to group alerts so that we reduce the [05:01] noise. But I think a lot of it comes [05:03] down to when you want to apply AI to [05:06] this problem, [05:08] no matter how good the AI is, it starts [05:09] with the data set it has access to. So [05:11] if you aren't at the beginning of the [05:13] pipe, if you aren't cautious or in some [05:17] way discerning about which events you're [05:20] sending to the AI to build its data set [05:22] from, then it's going to be its output [05:25] is going to be noisier for you too. So [05:27] if you're sending everything from [05:28] staging, from a development environment, [05:30] from prod to the same place, then it's [05:33] going to start anomaly detecting things [05:36] for you and say like, hey, hey, this is [05:38] a problem. even though it might not [05:39] actually be a problem that impacts is [05:41] worth waking you up for or is customer [05:43] impacting. So I think some of it has to [05:46] be in partnership with the AI itself. [05:47] You have to be you have to find a good [05:50] way to give it the data that then is [05:53] going to result in the outputs that you [05:54] need if that makes sense. [06:00] >> Great. Thank you. And you can continue [06:03] Dave because so that was like [06:05] pre-inccident. [06:07] I'm curious for once we're in the middle [06:08] of an incident and you have to triage, [06:12] what are you how are you triaging? [06:14] What's slowing you down the most? Um, [06:16] and how are you like leveraging tools or [06:18] practices to get context more quickly [06:20] during an incident? this [06:23] I I hate to start out qu answer a [06:27] question with it depends but the two [06:28] things that the one thing I think that [06:30] matters in terms of when you're triaging [06:32] is it is it we like to say is it new and [06:35] novel or is it well understood because I [06:37] think the way you triage an incident [06:38] depends on you know if it's well [06:40] understood you probably already have [06:42] existing runbooks and automation and [06:43] best practices to kind of reduce the [06:46] time it takes to resolve the particular [06:48] incident so I'm not necessarily going to [06:50] spend a ton of time talking about that, [06:52] but I'm going to assume something's new [06:53] and novel because those are the ones [06:54] that take up the more brain space. So [06:57] like the other part is where is your [06:59] triage occurring? Is it, you know, is it [07:01] via Zoom call? Is it via in Slack or [07:04] some other, you know, like where because [07:05] you want to bring what you want to do is [07:07] bring the information and the tools to [07:10] to the responder to the place where [07:13] they're working. So like are they able [07:15] to access all these things in one place [07:16] or do they have to open 17 tabs or or [07:20] like you know have have a bunch of [07:21] things open on their desktop that they [07:23] have to be looking at at the same time [07:24] or this is my ambivalent feelings about [07:27] dashboards too. I think dashboards are [07:29] great but also they can be overwhelming [07:31] like I think you need to cut down [07:33] people's reliance on dashboards and give [07:35] them more like a refund and this is [07:37] where again AI can help too is like give [07:38] them the data and the context at their [07:41] fingertips. And then uh lastly is how [07:44] can you reduce the number of humans [07:45] required for resolution? And I don't [07:48] mean that in an AI sense, but like if if [07:51] you have an incident where you need to [07:52] bring in 15 people, that's already a [07:55] problem, right? Like how can you get it [07:57] down to where you only need like one or [07:58] two people or even like some automation [08:00] to help out the one person that's that's [08:02] there. So I think reducing the the [08:07] friction to triage is is the key. And it [08:10] could be, but the friction comes in many [08:12] forms. So I think you have to analyze [08:13] where it is specifically for your or [08:15] organization or your team. [08:20] >> Can I add on to this a little bit [08:21] because I I love Dave's answer and I [08:22] think one of the things that we've [08:23] noticed at Privy and some of my previous [08:26] roles is that having very specific roles [08:28] for responders can help a lot with [08:30] triage. If everybody who's involved is [08:33] actively researching a different thread [08:36] and there's nobody who's coordinating [08:37] and nobody who is kind of, you know, [08:40] recontextualizing the information that's [08:42] coming in, it's really easy to have the [08:45] entire team heads down for maybe longer [08:48] than your entire budget to get things [08:51] back online. And you know, engineers who [08:54] go heads down on triage, on an [08:56] investigation, it's really good to be [08:58] able to leave them in that mode and have [09:00] somebody else who's operating at like an [09:02] incident commander level or a scribe [09:04] level or something else. Uh, and I think [09:07] that's something that a lot of bigger [09:08] companies have established really great [09:10] culture around. But if you're operating [09:12] with a smaller team, even though you may [09:14] only have, you know, three or five or 10 [09:17] engineers, being able to separate those [09:19] roles and identify who is responsible [09:21] for what during an incident can, you [09:23] know, greatly speed up the overall [09:25] response. [09:28] >> Yeah, absolutely. Like I would [09:29] definitely also like concer exactly to [09:32] what Spencer was saying. So for example [09:34] for us also internally we try to have [09:36] like let's say the incident commander [09:38] role itself between different parts of [09:40] the earth. So you know your fintech [09:42] incident commander is speaking to your [09:44] logistics incident commander. So that [09:46] way essentially you reduce the number of [09:48] nodes of different people talking to [09:50] each other. So at least there are [09:52] specific directed roles and like a a lot [09:54] more streamlined uh flow for these [09:57] incidents rather than a lot of noise. [10:03] Yeah, thanks Arjette. And um yeah, let's [10:05] hear more from you. So the last part of [10:09] this is after an outage so or an [10:12] incident like can you talk about why we [10:16] see the same problems come up over and [10:18] over again and how you move from just [10:21] like reacting to the same problem to [10:24] actually learning and actually changing [10:27] what's happening. [10:29] Yeah, I mean I think one part of the [10:31] problem is like this we start we see the [10:33] same problems because sometimes we treat [10:36] post incident reviews more like a [10:38] confession space rather than experiments [10:41] and and what I mean by that is like [10:42] teams genuinely write like good [10:44] postmortms. They're like there's clear [10:46] action items. We have owners but then [10:48] those action items sort of start [10:50] competing with your road map and then [10:52] they get dep prioritized and then a [10:54] couple of months later we see the same [10:56] incident or like a variation of that [10:58] incident with different people and then [11:00] we we are left wondering why didn't we [11:03] learn so one thing that sort of worked [11:06] with our part of like the or that where [11:08] we are where we started like we were [11:09] like okay let's treat these post [11:11] incident learnings like actual features [11:13] and not always as tech deaths. So we we [11:16] changed our whole flow from what should [11:18] we fix to [11:20] >> is there like the smallest thing that I [11:22] can do to make this less painful next [11:23] time. So like if an action item in your [11:26] incident report is [11:27] >> move this database X to Y. It's a great [11:30] action item but it's not going to get [11:32] prioritized because it requires you [11:34] getting sponsorship from your leadership [11:36] from product and all of that. So like [11:39] what's the smallest thing that you can [11:40] sort of do to make it less painful. So I [11:43] think that helped and then uh figuring [11:46] out if there is repetition of certain [11:48] incidents. So we like how do you do like [11:52] not just another postmortm but like more [11:53] like a deeper architectural review that [11:55] hey we are seeing a lot more incidents [11:57] from this part of the system. Is there [12:00] something that we need to think about a [12:02] little more in depth or is is that [12:05] something which is something else needs [12:07] attention there? And I think one other [12:09] thing that has also worked out well uh [12:12] for us is just thinking about how many [12:15] or how often is someone getting paged [12:17] outside business hours because we we [12:20] need to like figure out like active [12:22] capacity so that there is remediation [12:24] happening right because if our action [12:26] items like keep on getting stalled in [12:28] Jira then our on call also gets stuck in [12:31] slack and page of duty and everywhere [12:34] else. So I think that was one of the [12:35] biggest breakthrough for us to stop [12:38] treating some of these things as nice to [12:39] haves and more of more along along the [12:42] lines of this is how we want to prevent [12:44] the next outage that essentially breaks [12:47] the users trust in us and and that sort [12:51] of helped [12:54] >> that's great Arjett and because I'm [12:56] passionate about this I'll also add that [12:58] like yeah if you treat like if you have [13:01] access to pager duty you have a road map [13:03] of work that you can now pursue to fix [13:06] underlying issues. It's just like when [13:07] you have an app and you go inside crash [13:09] litics and you say, "What's my top crash [13:10] caused by?" Okay, great. If you if you [13:13] fix that one thing, you've made [13:14] progress. And um you can treat pages the [13:17] same way. Like what is the what is which [13:19] alert went off the most in the last 3 [13:21] months in the last quarter? And if I [13:24] tackle that one thing, how much burden [13:25] am I going to reduce off the the entire [13:27] team operationally and also in the tech [13:30] and for customers? Uh Spencer, go ahead. [13:35] >> Yeah. So, one thing that I think is is [13:37] helpful, and this is coming kind of from [13:39] the the leadership side of things, but I [13:41] think one of the responsibilities of a a [13:44] senior or or more senior engineer is to [13:46] learn how to advocate for the work that [13:49] needs to be done. It's not enough just [13:52] to understand that it needs to be done. [13:53] It's not enough to create a ticket for [13:55] it. You kind of have to learn how to [13:56] talk the business language to be able to [13:58] explain why something is important. And [14:01] maybe you're selling that to your [14:02] manager who's then going to sell it to a [14:05] product team or to a leadership team. [14:07] But learning how to be really articulate [14:09] about both the I guess the pain or the [14:12] the impact of leaving a situation as it [14:15] is and that can be cost to the road map [14:18] that can be distraction of engineers [14:20] that can be burnout that can be uh you [14:23] know product experience there's all [14:25] kinds of different uh areas where [14:27] there's impact in this. Uh but it's it's [14:30] not the manager's job. It's not you know [14:32] it is my you know job as a CTO but I [14:34] also push that back on to the engineers [14:37] who really have the context about why [14:40] something is happening over and over [14:41] again or you know the the reason that we [14:43] need to fix it. And it's really your job [14:45] as an engineer to be able to make the [14:48] case for why why we should work on [14:49] something. [14:54] >> Dave, [14:55] >> I could spend days talking about this [14:57] subject because it's really interesting. [14:58] But I I a few years ago we went through [15:01] a revamp internally of how we handle [15:04] post incident actions which I think was [15:06] really successful. But one thing that we [15:08] discovered was it was a local [15:09] optimization meaning within the incident [15:12] itself we were actually pretty good [15:14] about tackling the the high priority [15:16] items afterwards. But what we didn't [15:18] what was missing and what I think I you [15:20] know I'm also open to ideas on this is [15:23] what about patterns? What about things [15:24] across the major incidents? Like who's [15:26] looking at like systemic things that are [15:28] coming across that may be not within the [15:31] local of that one incident but may be [15:33] recurring across these incidents as [15:35] well. And I think that's something that [15:36] sometimes [15:38] those are the big ones, right? Like [15:39] because you can look at the big you can [15:40] look at the incident itself and often [15:42] yeah maybe it was a major outage but [15:44] usually there's something it's tying [15:46] back to something else. And that's the [15:48] that's the other thing. You have to not [15:51] just look for the you know the small [15:53] ticket like and which is really [15:55] important but like what is the big [15:57] pattern that we need to tackle [15:58] architecturally that may prevent some of [16:01] these incidents from occurring in the [16:02] future. [16:06] >> Yeah, that's great. [16:08] And I want to move to a question from [16:10] the audience. Uh [16:12] I I think it was you Dave that mentioned [16:15] dashboards and like all this [16:17] information. So, Florian, hi Florian [16:20] asks, "One key factor in dealing with [16:21] incidents is live business metrics. Any [16:24] learnings you'd like to share on how you [16:26] came up with or used a business metric [16:28] in a novel way? [16:33] That's a I I think ideally and I I don't [16:36] know if we're there yet, but ideally you [16:41] if it if it's a business problem, then [16:43] you should have some easy way to discern [16:46] like a tie ar tie your outage to ARR, [16:49] right? Like see like what's the revenue [16:51] impact of the thing that that's [16:53] happening right now and who's being [16:55] impacted is it like our crown jewel [16:57] customers like is there depending on if [16:59] you have sharding or if you have like [17:00] some sort of dis how how where is the [17:04] impact and who is it impacting and and [17:05] I'm not saying [17:08] practically speaking like all customers [17:10] are great but some customers are more [17:11] valuable than others to your business [17:13] right so like I think that's the end [17:15] goal would be to and it's hard to get [17:17] right because you need distributed [17:19] tracing you need to have some sort of [17:21] like account linking and it's very it's [17:24] very complicated but I think ultimately [17:26] that is the goal and in in in the [17:28] meantime I think you can probably use [17:31] like some sort substitute [17:33] uh kind of metrics in in between. So if [17:36] it's not the customer ID, it's it's it's [17:38] it's something that's related to [17:40] customers, but ultimately it's about the [17:42] end users. It's about what is the [17:44] customer impact. So that is the thing [17:46] I'll be focusing on. So because if [17:48] you're like if your CPU on one of your [17:50] clusters is at 100% but there's like it [17:54] doesn't it's there's no customer impact [17:56] then it's not I'm not saying it's not a [17:58] problem but it's not an urgent problem [17:59] to solve. So I think that that that's [18:02] the that's the thing I've seen change [18:04] over time is like from the Nagios days [18:06] of like I got I got a PEG CPU I got to [18:09] fix it to now people are thinking more [18:10] in terms of customer impact [18:16] >> hardship. Yeah, I mean completely agreed [18:18] with Dave like like if you see something [18:20] like a 5% error rate you're like okay [18:22] sure not great but also not catastrophic [18:25] but if you see like oh 100 customers are [18:28] impacted now suddenly that is a [18:30] different story right so it's it's more [18:32] along the lines of figuring out what is [18:34] your like what's the easiest way for you [18:36] to figure out your blast radius like [18:38] some sort of a blast radius dashboard in [18:41] terms of your actual impact in terms of [18:43] business or customers because like yeah [18:45] 100% CPU to uh like shouting at you, [18:49] you're like okay sure there is something [18:51] I need to do but is it actually [18:53] impacting you probably don't know so [18:55] yeah [18:56] >> and just to add on one last thing to [18:58] this but uh there are tools that allow [19:01] you to instrument this so that the tool [19:04] itself will tell you how many different [19:06] customers have experienced a particular [19:09] error or particular bug and you can set [19:11] thresholds for alerting if you know the [19:15] bug is experienced or if the errors [19:17] experienced by you know whether it's [19:19] number of users whether it's number you [19:21] know if you tenant by customer or [19:22] something like that but as long as you [19:24] have the ability to identify you know [19:27] which end user was affected by an error [19:30] or you know what is the the meta [19:32] information that caused that. You can [19:34] build a lot of instrumentation that will [19:36] allow you to notify on this very [19:38] specifically and to you know cause a [19:40] page if a a threshold was breached. [19:46] >> Thank you all. I'm going to take another [19:48] audience question. This one I'm This one [19:50] is very fun. How do you get your [19:53] engineers excited about S sur and [19:55] putting the effort into aspects like [19:57] monitoring? [20:01] >> Can I jump in here on this one for a [20:03] second? Yes, please. [20:05] >> I don't I don't know if I have an answer [20:07] for the first part because not everybody [20:10] loves being on the front line. But for [20:13] the second part, my answer is that [20:15] everybody is responsible for [20:18] keeping the the app online. Everybody [20:20] needs to be responsible for monitoring, [20:23] for operations, for those kinds of [20:24] things. And the way that I have done [20:26] that in my last several organizations is [20:28] everybody's on call and you share an on [20:30] call rotation. And when you're on call, [20:32] well, you start to learn to care about [20:34] things like monitoring and logs and [20:37] alerting because it affects your life on [20:39] a day-to-day basis. And so those things [20:41] do start to get better when everybody [20:42] has a responsibility. On the excitement [20:45] side, harder to assign that. So, I don't [20:47] know if Arjett or Dave have ideas on on [20:50] how to drive that. [20:51] >> I agree with everything you said because [20:53] I think it's more self-preservation than [20:54] excitement. Like, how many times do you [20:57] want to be woken up at 3:00 in the [20:58] morning, right? like it's not healthy [21:00] for you and it's not frankly it's not [21:02] healthy for the business because if [21:03] you're getting woken up presumably some [21:05] customers being impacted. [21:08] >> Yeah, for sure. I think I've honestly [21:10] stopped trying to get them excited about [21:12] monitoring and I'm just I've moved from [21:15] excitement to more like like just making [21:17] them angry about not having it like and [21:20] like Dave said, right? If you wake up at [21:21] 3:00 a.m. and you're like, "Oh, I wish [21:23] there was that dashboard which would [21:25] have told me how many customers are [21:26] getting impacted." Now you are angry. [21:28] you want to go and figure that out [21:30] because you spent 1 hour waking up at [21:33] 3:00 a.m. just to find something which [21:35] if that if that would have existed would [21:36] have taken you 5 minutes to figure out [21:38] if this is a real issue or not. So I [21:40] think at least for me I've moved a [21:42] little away from excitement after [21:43] failing around it and now more along the [21:45] lines of how to get some emotion out of [21:48] it. [21:49] >> You you'll also build better tools when [21:52] you're building them out of need rather [21:53] than building them out of theory. You [21:56] know, if you have a team that's just [21:57] responsible for building all of your [22:00] internal tools, you'll end up with some [22:02] really great tools and a bunch of tools [22:04] that nobody ever uses because they were [22:05] built with some theoretical backing [22:07] rather than as a response to a [22:09] particular painoint. [22:15] >> I guess the perennial question is how do [22:17] you get an engineer excited about [22:19] anything? But we'll save that for [22:22] another panel. [22:23] Um, [22:26] okay. So, let's see. [22:30] Arjett, [22:32] when does on call fatigue stop being an [22:35] individual problem [22:37] and become a systemic one and then how [22:40] do you work with that? [22:43] Uh I mean I think like on call fatigue [22:46] sort of becomes systemic the moment like [22:48] you hear one person or like a bunch of [22:51] engineers talking about that being a [22:54] factor in their decisions in their [22:55] career decisions in the projects they're [22:57] thinking about or just their fatigue [22:59] itself right I think a lot of people get [23:01] caught up in oh we would do like a [23:03] full-fledged burnout survey or a [23:05] retention metric or some of that but I [23:07] mean your signals can be soft your [23:09] signals can be small that hey people are [23:11] talking about it they're thinking about [23:13] not taking up a project, not being on a [23:15] team or just being tired of it. And I [23:16] think I mean that is where the question [23:19] around like just the fairness of how do [23:22] you improve some of this on call right [23:24] and like your second question how do we [23:27] improve the systemic part of it I'd say [23:30] one we need to figure out like are your [23:32] engineers getting paged multiple times [23:34] in a single on call shift and I think if [23:36] if they're getting multiple times paged [23:39] in a single night itself you know people [23:42] are already starting to dread their on [23:43] call week and like some sort of a rec- [23:46] recovery time and then are your [23:48] incidents even predictable that you know [23:50] it's a deployment change or it's a [23:52] config change or is it random where you [23:54] will have to bring in 10 teams to figure [23:56] something out because those are the ones [23:58] that take a lot more out of you and like [24:01] that that sort of starts compounding [24:03] your fatigue and creates that [24:05] helplessness right so I think for us [24:08] what has sort of worked is like of [24:10] course we improve on tooling but like [24:13] also just acknowledging [24:15] sustainable on call means that like [24:18] sometimes [24:19] it will it will impact of course your [24:23] sleep and everything else but is that is [24:25] that like distributed well enough or is [24:28] there like good enough cognitive load [24:29] distribution and if that's not the case [24:32] and like I think Spencer was saying out [24:33] you need 20 people in an incident to [24:36] resolve something you have a lot more [24:38] deeper systemic issues rather than the [24:40] lack of a particular tool or one person [24:42] not being involved [24:49] Yeah. And I'm actually curious, this is [24:52] something that we discussed prior to the [24:54] call, how in places like Germany, you [24:57] actually cannot mandate that everybody [24:59] is on call like Spencer is doing in his [25:01] or I I'm American so I've now worked in [25:04] both ecosystems and it's fascinating. So [25:07] I'm curious like if anybody has insights [25:09] on, you know, on [25:13] how to work with that constraint. [25:19] >> I mean I'm in Germany right now so I can [25:22] probably at least some some part of it [25:25] and I think which is where exactly your [25:27] question becomes even more important [25:28] Asha because you have to have better [25:31] systems in place. You cannot be [25:32] firefighting all the time if you have a [25:34] limited pool of engineers also there. So [25:36] like you have to at some point in time [25:38] you have to sit down and like hey do we [25:40] need to invest in SLOs's is a 2% CPU [25:43] higher than CPU threshold worth bringing [25:46] an engineer into this and like let [25:48] letting them spend time on this or is [25:49] this going to auto resolve in 15 [25:51] minutes. So I think you start thinking [25:52] about things in a lot more longer [25:55] fashion because you cannot use all of [25:57] your engineers as a defense mechanism to [26:00] all the different alerts that are [26:01] happening. And I think that like as as [26:04] an example last year at some point in [26:06] time we had only three people in our [26:08] rotation and this was still covering [26:09] like a bunch of services. Now within [26:12] three of us if we have to take care of [26:14] seven or eight different services we [26:16] cannot be reacting to everything. So you [26:18] pick your battles carefully and then you [26:21] navigate that and then you the other [26:22] part of it is you figure out what is [26:25] causing people to not be a part of on [26:27] call because these are services that [26:28] they are maintaining and that is the [26:30] part which is your systemic issue which [26:32] you tend to like help them resolve build [26:34] SLOs's build confidence tell them that [26:36] this work is appreciated it's not extra [26:39] burden work that you or your manager are [26:42] just piling on these people right let [26:44] people let compensate people if possible [26:47] right? If monetarily or by compost or [26:50] whatever things are applicable in your [26:51] company. So at least for us there is [26:53] monetary compensation and then if you're [26:55] on call for a longer night or for a [26:58] weekend then you can of course talk to [26:59] your manager get a day off like figure [27:01] those things out because if it's not [27:03] appreciated you're not going to have a [27:06] healthy set of people who are going to [27:07] be interested in doing this. [27:12] >> Yeah, that's that's great advice. And I [27:14] guess on a similar note like when we [27:15] don't have that constraint like Spencer [27:17] you have everyone on call but even [27:19] though everyone's on call you might [27:20] still implicitly like overrely on [27:22] certain people to be the fixers not just [27:25] rather than the triagers. So as your [27:28] team is scaling since you work at a [27:29] startup how are you preventing that [27:32] reliance on like a a certain number of [27:34] engineers while maintaining strong [27:36] ownership? [27:39] It's a tough question to answer because [27:41] I think the reality is that certain [27:43] engineers do tend to establish [27:45] themselves as the experts for systems, [27:49] for databases, for the types of things [27:51] that tend to break when you have a an [27:54] incident. You know, it's rarely a [27:56] front-end component that's acting in a [27:58] really mysterious way. It's you know [28:00] usually something to do with the [28:02] connection between distributed systems [28:04] or you know a load that uh is higher [28:08] than you were anticipating which tends [28:10] to tends to be you know a few engineers. [28:12] I think the the main thing for me is for [28:14] those engineers that tend to find [28:16] themselves in let's call it like [28:18] lynchpin [28:19] situations where they are the one [28:21] engineer who knows I try to work with [28:24] that engineer to make one of their [28:26] particular goals to to be to spread that [28:29] knowledge. So it's me for me it's like a [28:31] three-step system. You you start off as [28:33] the expert. You're the lynch bin. You're [28:34] the person who really has to be looked [28:36] at to fix. And I think it's inevitable [28:38] that you will have situations like that [28:40] on your team. I don't think you can just [28:41] permanently avoid that because creating [28:44] new software, creating new systems tends [28:46] to involve somebody doing something [28:49] either novel or something that the rest [28:51] of the team doesn't know, you know, from [28:52] day one. But then the job is to then go [28:55] from being a lynchpin to let's say a [28:58] knowledge sharer. You bring other people [29:00] in into that. So, you know, as a leader, [29:02] I try to assign different people to the [29:05] same systems. So maybe I have somebody [29:07] who's the, you know, the Stripe or the [29:09] billing or the subscriptions expert. The [29:11] next time we have a small project there, [29:13] I'm going to assign somebody who is not [29:14] that person. Not because it will get [29:16] done faster, but because we will start [29:17] to spread that knowledge. And I think [29:19] being willing to take a slight penalty [29:21] in velocity in order to avoid having the [29:24] exact same people responsible for a [29:27] single system can be helpful. And then [29:29] you can move from being, let's say, a a [29:32] knowledge sharer to being a multiplier. [29:34] And I think that's the really the end [29:36] goal for a lot of these really generally [29:38] more senior engineers is how do you move [29:41] from lynchpin to ultimately knowledge [29:44] sharer to then multiplier where you're [29:46] not just you know spreading the [29:49] knowledge of what you've done but you're [29:50] enabling other people to create systems [29:52] in a similar way. [29:58] >> Thank you. I'm going to take another [30:00] question from the audience that I love. [30:03] So this I'm gonna paraphrase this one. [30:06] Uh this person is asking how do you [30:09] treat the first occurrence of an [30:12] incident because in their experience and [30:15] this has also been my experience. Um [30:18] they take it to like leadership or [30:19] product manager and and the person [30:21] replies well how likely is this to [30:24] reoccur? [30:25] Um and it's usually hard to answer [30:28] because in incidents are inherently [30:30] unpredictable. So, how do you how do you [30:33] usually answer that? How do you treat [30:35] that first occurrence and like maybe [30:36] provide the right level of risk to a [30:39] stakeholder that doesn't really [30:40] understand how systems inherently work? [30:42] And usually they're looking for the [30:44] answer of like, oh, it's low [30:45] probability, so I don't have to fix it. [30:47] That's what they want to hear. Uh, how [30:50] do you work with that? [30:54] >> The the first thing that comes to my [30:56] mind here is this is really about [30:57] trade-offs. It's about risk analysis. [31:00] And I I don't even know if the frequency [31:03] or the likelihood of something recurring [31:05] is the most important [31:07] uh metric or the the most important [31:10] factor here. But if it does recur, what [31:14] is the impact to the business? Is it [31:15] going to mean is it system has a major [31:18] outage and is down for many customers or [31:21] for the entire customer base? If so, [31:23] even a very small chance of recurrence [31:26] is probably worth addressing. [31:28] if it's a minor nuisance and you don't [31:31] think it's going to recur, you know, I [31:33] think there can be a business case to [31:35] either deferring or I I mean, honestly, [31:38] I like a wait and see approach [31:40] sometimes. You know, maybe you don't [31:41] know how likely it is that something's [31:43] going to recur and it's not that big of [31:45] a deal and so you do say, "We're going [31:46] to defer this, but if it happens again [31:48] next week, then we're going to put some [31:49] time on the calendar and make sure that [31:51] we fix this." [31:54] Yeah, I think I absolutely agree with [31:57] Spencer and I think I I also feel like [31:59] the probability of reoccurrence is sort [32:01] of a misleading question. Like it's one [32:04] of those things that should I get a fire [32:06] insurance but my house has never caught [32:07] fire, right? Like I mean you're not [32:09] going to get a fire insurance after the [32:11] house catches fire. So I feel the [32:14] conversation needs to be like what did [32:16] this incident cost you? And sometimes [32:18] the cost of the fix versus the cost of [32:21] if this happens again, right? That is a [32:23] simpler trade-off like because the fix [32:25] could mostly be a lot more cheaper than [32:27] bringing 20 engineers into an incident [32:29] call managers, directors, whatnot. And [32:32] if you literally like start figuring out [32:34] the cost of managing that incident, you [32:36] realize, you know what, the fix is [32:37] probably much cheaper rather than doing [32:40] the whole shebang. [32:46] Yeah, that's great. I think in general [32:48] sometimes we end up in a state of [32:50] paralysis overanalyzing when just doing [32:53] the thing would be the faster thing and [32:55] make everybody feel better also. So [32:57] that's a great point. Um I want to ask a [33:02] little bit about leadership. So Spencer [33:05] already mentioned a little bit of like [33:06] what needs to happen after an incident [33:07] to make space. Dave, I'm curious to hear [33:10] from you. Like what else do you think [33:13] engineering leaders should be doing [33:14] differently to make on call sustainable? [33:16] How do they protect remediation time? [33:19] How do they stand up for their teams and [33:21] the people that are actually trying to [33:23] improve these systems and the on call [33:25] process? Um especially like when we all [33:28] have a bunch of planned roadmap work [33:30] that we have to negotiate. [33:32] >> Um this is tough because I think this is [33:35] as leaders it's always a problem, right? [33:37] because you have a people issue, right? [33:40] If especially when you're close to [33:41] burnout for a team or when you're seeing [33:43] a lot of on call, but and and even [33:46] though it is a human problem, I think [33:47] you need to frame it as a business [33:49] problem. So because that's essentially [33:52] and it's not to say that leadership is [33:55] unempathetic or not human, but you need [33:58] to when you're talking about trade-offs [33:59] and you're talking about investments, [34:01] you need to understand what the cost [34:03] could be to the business. So to me if [34:05] half my team is burned out I can [34:07] actually quantify that with metrics in [34:09] terms of like hey what's the if the [34:10] attrition happen if this kind of [34:12] attrition happens this is the impact of [34:14] the projects that and deliverables that [34:15] we're going to have so I'd like to [34:17] prevent that or if my team is getting [34:19] paged a lot that's not just a human [34:21] problem that's that also implies that [34:24] our systems are unstable so what is the [34:26] customer impact of that so what you can [34:28] do is actually put an argument together [34:30] that's based on business value and and [34:34] business outcomes and that then feeds [34:37] into actually helping fix the human [34:39] problem because if you just if you just [34:41] go and say like my engineers are burned [34:43] out it's not I mean even if it if [34:45] someone was coming to me I would be [34:47] really empa sympathetic to that but it's [34:49] it's hard because then you know you're a [34:51] CTO you're probably talking to your [34:53] product counterparts all the time you're [34:54] probably talking to the CEO and they're [34:56] giving you pressure for like hey I [34:58] wanted you know we got this product [34:59] stuff to release over here and so if [35:01] you're not able to make the case at that [35:03] then I don't think you're going to get [35:04] the outcomes that you need. But I think [35:06] you can make the case at that level [35:08] because the burnout actually does lead [35:10] to business issues and instability [35:13] actually does lead to business issues. [35:15] But that's the level you need to be [35:16] talking at when you're making those [35:17] arguments in my opinion. [35:25] >> Yeah. Anyone else have thoughts on [35:27] leadership's role in all of this? [35:33] I think you know and I'll look at myself [35:34] here but certainly if you're in an [35:36] executive seat you do share [35:39] responsibility for making sure that you [35:41] are advocating for things that will help [35:43] your team work in a sustainable way and [35:47] you know while I do look to my engineers [35:50] senior engineers staff engineers etc to [35:52] help me make that case ultimately I [35:55] think it is you know the leadership's [35:57] responsibility to you know avoid putting [35:59] engineers in a situ situation where [36:01] they're going to burn out. Uh or avoid [36:03] putting people, you know, Arjett talked [36:04] about multiple pages within the same on [36:07] call rotation, you know, like do you [36:09] have a contingency plan for what happens [36:12] if you have an incident that is [36:14] recurring several times in a row? Can [36:16] you run rotation early? You know, is [36:18] there a way to do that so that you don't [36:20] have the exact same people responding [36:23] day after day? Uh, you know, one of the [36:25] things that we'll do is for our our [36:28] known busy periods, we work in [36:30] e-commerce and and marketing. So, Black [36:32] Friday, Cyber Monday, some big holidays, [36:35] they're just busier. They're higher [36:37] stress. We have the highest load we'll [36:39] have for the entire year. So, we have a [36:41] different on call rotation. It's more [36:43] like I think it's a 24-hour rotation [36:45] rather than a week-long rotation for [36:47] those periods because it's much more [36:49] stressful and if you just draw the wrong [36:50] straw and you happen to end up with the [36:52] worst on call rotation, that's not [36:53] great. So I think it's really business [36:56] to business, but can you analyze the [36:58] different patterns for your business and [37:00] come up with something that [37:02] doesn't, you know, randomly put people [37:05] in a situation that they're much more [37:07] likely to to be stressed or to, you [37:09] know, have a lot of higher higher stress [37:11] incidents. [37:17] >> Okay, I'm going to take this might be [37:19] our last question. Let's see uh another [37:21] one from the audience. So, incidents [37:24] affect reputation and trust. How do you [37:26] maintain that trust with your customers [37:28] after an incident? Um, [37:32] do you call out your vendors in the I'm [37:35] not sure what RCA stands for. Sorry. [37:37] Maybe someone else does. [37:39] >> Maybe. [37:40] >> Oh, yeah. Okay. Acronyms are hard. Okay. [37:42] Do you call out your vendors? What other [37:44] te techniques do you use to maintain [37:46] trust? [37:48] >> I have I have opinions about this. I [37:51] first off never call out your vendors. [37:53] You you made the choice to select that [37:55] vendor, right? That's your [37:56] responsibility. So ultimately, yes, they [38:00] may have contributed to whatever it is [38:02] that you did, but you made that choice, [38:05] right? So, you know, you don't abdicate [38:07] responsibility by handing it off to [38:09] someone else. That's I feel pretty [38:10] strongly about that. And secondly, I [38:12] think be transparent. You know, I think [38:16] that I was pretty proud. We had a really [38:18] significant incident back in uh end of [38:21] August, beginning of September and I was [38:23] really proud of how we and the team came [38:25] together to publish probably our best [38:27] postmortem doc that we have in like the [38:29] last four years and we just decided to [38:31] approach it from a very transparent and [38:33] technically rigorous perspective. So we [38:35] had like five, you know, I was in a room [38:37] with like principal and staff engineers [38:40] saying like how much do we want to [38:41] describe about the way our systems look [38:43] and we we just said let's do everything [38:45] we can like let's be honest about what [38:47] we what broke how our architecture is [38:50] like where the failure points were here [38:52] and what we're going to do to fix it [38:53] next time. And I think that transparency [38:55] and honesty actually helps build trust [38:57] because I I I know that I feel the same [38:59] as someone who manages a lot of vendors. [39:02] I that's how I feel when I want to see a [39:04] a postmortem doc or an incident review [39:06] from one of my vendors. I want to see I [39:07] don't want to see just like two [39:08] sentences thing going like oh you know [39:11] kind of papering over it. I want to see [39:13] here's a here's a real technical issue [39:15] we ran into and here's how we're going [39:17] to prevent this from happening again. [39:21] >> Thanks Arjette. [39:23] >> Yeah I mean huge plus one to Dave. I I I [39:26] mean from a customer point of view I [39:28] don't think the customers don't they [39:30] don't actually care whose fault it was. [39:32] They care whether if you are going to [39:33] let that happen again. If you go ahead [39:36] and write in your like incident that hey [39:38] you know what AWS went down and that is [39:40] what happened. It's the same problem [39:43] that from the customer's perspective [39:45] they have a contract with you not AWS. [39:47] So AWS going down sure it's your problem [39:50] not their problem. So I I do think like [39:52] completely huge plus one to Dave and [39:55] that is exactly how I feel. So they have [39:58] basically put in your trust to have [40:00] resilient systems and vendors are always [40:03] going to be imperfect. So that's not [40:05] something you can probably escape. [40:08] >> I'll add not to Dave and Arjett's answer [40:13] which was perfect. Don't call out your [40:15] vendors, but it's really important to [40:17] build relationships if you're on the [40:19] engineering side with the go-to market [40:21] counterparts. And I think one of the [40:23] most important things you can do to [40:25] build trust is to learn how you [40:28] communicate incidents, downtime, other [40:32] unexpected scenarios directly to the [40:35] affected customers. And so for for [40:37] Privy, we have what we call support on [40:39] call or support alert. I can't remember [40:41] what it's called. It's in our Slack [40:42] channel. And we will tag our support [40:44] alert team anytime there's something [40:46] that affects customers. And the goal is [40:49] not that we have paged your duty and [40:50] that you know support is in this on call [40:52] rotation, but more that as we get into a [40:54] spot where we're actually trying to [40:55] communicate the impact or the expected [40:58] duration or or something like that to [41:00] somebody who is affected by the [41:02] incident. The engineering can describe [41:05] it however we will. you know, we're [41:06] going to use engineering terms and then [41:08] it can be translated by a customer [41:09] success rep who has a relationship with [41:12] that customer already. We also have a [41:15] status page and those kinds of things. [41:16] But we found that generally, you know, [41:19] especially more complex or ongoing [41:21] incidents, it's much better to have just [41:23] that onetoone communication either from [41:25] our support team and you know, we'll [41:27] measure an incident by support volume as [41:29] well. You know, so when we have an [41:30] incident, it's like, okay, well, how [41:31] many tickets have we had about this? And [41:34] for any real incident, you see that [41:36] support volume start to go up. So really [41:37] important to be able to enable your [41:38] support team or empower your support [41:40] team to give good answers that don't [41:43] beat around the bush, but also, you [41:45] know, you don't reveal confidential [41:47] information. And and the more you can do [41:49] to to help that uh that group out, the [41:51] better. [41:55] >> Yeah. And I think just to round that [41:57] out, I'll say that what Spencer just [41:58] said is where, you know, showing [42:00] accountability and ownership is where [42:02] you can shine as a leader. That really [42:04] separ separates like managers from true [42:06] leadership. Um, we have one more amazing [42:10] question, but we're not going to have [42:11] time to answer it. And so, please take [42:14] it to the the chat. Um, oh, wait, [42:17] actually, we have two more minutes, so [42:18] I'll I'll throw it out for someone to [42:19] give a lightning round answer. What do [42:23] you consider a good ratio of on call [42:25] engineers to application [42:28] domain etc. [42:36] I mean I'd say six to eight engineers is [42:38] probably okay but it's more about the [42:40] like if you are an engineer who's [42:42] touching that system every single day [42:43] and you're on call once in every two [42:45] months it's probably fine but if you're [42:47] an engineer who touches that particular [42:48] system once in six months and you're on [42:51] call every two months that basically [42:52] means you're literally relearning the [42:54] system again during your on call shift [42:56] right so I think it's also a little more [42:58] tied to how frequently you work on that [43:00] system and how frequently are you on [43:03] call for that [43:05] And I think it's also has to do with how [43:08] operationally excellent your team is and [43:10] how wellmaintained the services are [43:12] because I gota have a team with eight [43:14] people on it with 40 services and on [43:16] call is quiet because we you know you're [43:19] proactively taking care of those issues [43:21] ahead of time. So I think it's really [43:22] hard to say in the abstract that there's [43:23] a magic number. I think you could have [43:26] like a one team of eight people with one [43:29] service and that service is paging you [43:30] every day like seven times a day. then [43:32] that's bad, right? So, I think it it [43:34] really depends. [43:36] >> I'll take this from the [43:38] >> Sorry, go ahead. [43:39] >> I was gonna wrap up, but if you have any [43:41] last thoughts, Spencer, [43:42] >> I was just going to take this from the [43:44] other side, which is I think it's [43:45] important to have enough people on call [43:46] that the rotation isn't every other [43:48] week, you know, or something that's [43:49] unsustainable. So I would rather have a [43:53] rotation that has responsibility for say [43:55] a wider surface area of of services and [43:59] has enough people that you're not on [44:02] call more than let's say one every six [44:04] one out of every six weeks than one [44:06] where everybody has a really deep [44:08] knowledge of everything that they are [44:10] operating and you're on call every other [44:12] week. [44:15] >> Yeah, thank you for that. and and I [44:16] think our friend uh Simon who asked this [44:18] question. If you have like a more [44:19] specific question about your scenario, [44:21] great thing to bring up in the Slack [44:23] channel as well and you can get like [44:25] real advice from this group. So, thank [44:28] you all so much for the good insights. I [44:30] definitely learned some things. So, uh [44:32] we're going to head over to the panel [44:34] Q&A channel in Slack. We just messaged [44:36] it in the chat in case you didn't catch [44:38] that. And uh yeah, please head over [44:41] there to continue the discussion. Thank [44:42] you for joining. Thank you for our [44:44] panelists. and have a great day. Bye.