Rooting out sneaky sources of toil - CourtneyR.dev's 2nd Brain

Rooting out sneaky sources of toil 🔗 **Source:** [Rooting out sneaky sources of toil](https://www.youtube.com/watch?v=M3ntUs4hEg0) > [!note]+ Document Notes > tags: [toil-reduction, incident-response, platform-engineering] > author: Lead Dev > source: YouTube > reference: https://www.youtube.com/watch?v=M3ntUs4hEg0 > date created: 2025-01-07 13:22 > > # 🏷️ Title: Rooting out sneaky sources of toil >  ## 🧰 Action Inbox - [H1] Identify sources of data quality toil in service ownership — track time lost to ownership escalation #platform-engineering - [H2] Implement automated dependency upgrades to prevent security fire drills — establish regular cadence #security - [H3] Measure incident recovery time alongside downtime — build business case for investment #incident-response ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Lead Dev Online Panel](https://www.youtube.com/watch?v=M3ntUs4hEg0) - [ChainGuard](https://chainguard.dev/) ## 📝 Summary - Expert panel discusses identifying and eliminating hidden sources of toil in engineering organizations - Why reactive incident work compounds over time when teams scale - Key stakeholders: engineering leaders, SRE teams, platform engineers, security teams - Risks: trust erosion, decision paralysis, accumulated technical debt - Bottom line: Proactive measurement and automation prevent toil from becoming organizational emergencies ## 📖 Main Content This panel brings together engineering leaders from Netflix, New York Times, Google, and ChainGuard to discuss systematic approaches to toil reduction. The speakers define toil as unplanned, repetitive work that doesn't meaningfully improve systems - distinguishing it from planned operational work. Key themes include the importance of measuring recovery time (not just downtime), implementing guardrails that make safe practices the default path, and building business cases for toil reduction through concrete data. The discussion emphasizes that most toil becomes visible only when organizations are willing to track it systematically. The panel shares real-world examples from major incidents, including supply chain attacks in npm ecosystem, infrastructure outages during major news events, and the log4shell vulnerability response. These cases demonstrate how accumulated toil creates organizational paralysis during critical moments. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/M3ntUs4hEg0?t=0) — [H1] Panel introduction and toil definition - Key idea: Toil shows up as context switching, reactive incidents, and repeated process layers - Evidence: "It shows up as one-offs, but over time they compound" #### ⏱️ [02:30](https://youtu.be/M3ntUs4hEg0?t=150) — [H2] Malikica Rao introduces distributed systems toil - Key idea: Toil reduction focuses on operational and cognitive load for engineers - Evidence: "Toil is often treated as inevitable when in my experience it's usually a design signal" #### ⏱️ [03:45](https://youtu.be/M3ntUs4hEg0?t=225) — [H2] Chang Leong on human-centric toil perspective - Key idea: Toil exists on the human side of sociotechnical systems - Evidence: "Computers don't feel toil as far as I know. They don't have feelings or emotions about it" #### ⏱️ [05:15](https://youtu.be/M3ntUs4hEg0?t=315) — [H2] Leslie Cordero's NYT reliability experience - Key idea: Data quality creates significant platform engineering toil - Evidence: "Some of the biggest sources of toil have to do with the quality of our data actually" #### ⏱️ [07:30](https://youtu.be/M3ntUs4hEg0?t=450) — [H2] Manfred Moser on supply chain toil - Key idea: Unknown dependencies create emergency response situations - Evidence: "Understanding what versions of what libraries are used where is often completely unknown" #### ⏱️ [09:45](https://youtu.be/M3ntUs4hEg0?t=585) — [H1] Malikica defines toil characteristics - Key idea: Toil is unplanned repeatable effort that doesn't improve systems - Evidence: "The gap between when a system is technically backup and when it's actually safe, trusted and usable" #### ⏱️ [12:15](https://youtu.be/M3ntUs4hEg0?t=735) — [H2] Chang on reframing toil value - Key idea: Planned work can change perception of toil through business value framing - Evidence: "If it's valuable for the business and maybe it's hard to do, well, maybe that's worth investing in" #### ⏱️ [16:00](https://youtu.be/M3ntUs4hEg0?t=960) — [H1] Supply chain attack examples from Manfred - Key idea: npm ecosystem attacks spread across multiple technology stacks - Evidence: "JavaScript components were then found in fullbaked applications that were deployed to Maven Central" #### ⏱️ [19:30](https://youtu.be/M3ntUs4hEg0?t=1170) — [H2] Chang's winter storm data center story - Key idea: External events can force accelerated infrastructure investment - Evidence: "Winter is coming... did it in one year which was several years faster than we had planned" #### ⏱️ [22:45](https://youtu.be/M3ntUs4hEg0?t=1365) — [H1] Malikica's Twitter search infrastructure case - Key idea: Trust erosion from slow recovery creates organizational drag - Evidence: "Data backfill basically took us 7 days... recovery time had become a bigger product problem than the incident itself" #### ⏱️ [26:15](https://youtu.be/M3ntUs4hEg0?t=1575) — [H2] Leslie on proactive toil tracking - Key idea: Historical patterns make business case for toil reduction investment - Evidence: "Keeping track of things over of toil over time is important... showing that historical pattern" #### ⏱️ [29:30](https://youtu.be/M3ntUs4hEg0?t=1770) — [H2] Chang on incident response practice - Key idea: Practice incident response during calm periods using error budgets - Evidence: "Spend that unspent budget. If things are up, great... use that investment to have practice sessions" #### ⏱️ [33:00](https://youtu.be/M3ntUs4hEg0?t=1980) — [H1] Malikica on making safe paths default - Key idea: Automation should change behavior under pressure, not create more alerts - Evidence: "We stopped relying on heroics and just made the safe path boring and the default path" #### ⏱️ [36:45](https://youtu.be/M3ntUs4hEg0?t=2205) — [H2] Manfred on standardization benefits - Key idea: Technical standardization reduces cognitive toil and improves security response - Evidence: "Don't allow your developers to make a separate monitoring system and choose a different database" #### ⏱️ [40:15](https://youtu.be/M3ntUs4hEg0?t=2415) — [H1] Final automation principles - Key idea: Best automation makes problems disappear before engineers think about them - Evidence: "Automation is worth investing in when it changes behavior under pressure, not when it just produces more signals" ## 📚 References - [Lead Dev Panel Discussion](https://www.youtube.com/watch?v=M3ntUs4hEg0) - [ChainGuard Supply Chain Security](https://chainguard.dev/) - [SRE Best Practices](https://sre.google/) ## 🗣️ Paraphrase - Engineering organizations often accept toil as inevitable when it's actually a symptom of design problems that can be systematically addressed through measurement and automation - The most damaging aspect of toil isn't the immediate work but the cumulative trust erosion and decision paralysis it creates across teams over time - Effective toil reduction requires making safe practices the default path through guardrails and automation rather than relying on individual discipline under pressure ## 📰 Newsletter **Tackling Hidden Engineering Toil Before It Becomes Crisis** Engineering leaders from Netflix, NYT, and Google shared battle-tested strategies for identifying and eliminating sneaky sources of toil that compound over time. Key insight: measure recovery time, not just downtime, to build business cases for proactive investment. **Practical takeaways:** - Track historical patterns of repeated manual work to demonstrate organizational impact - Use error budgets for incident response practice during calm periods - Implement guardrails that make safe deployment practices the default, boring path - Standardize technology choices to reduce cognitive load and improve security response The panel emphasized that effective toil reduction changes behavior under pressure rather than creating more monitoring noise. Resources: [SRE Practices](https://sre.google/), [Incident Response](https://response.pagerduty.com/), [Platform Engineering](https://platformengineering.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Present toil reduction case studies at engineering conferences, create demo environments for resilience testing - Education: Workshop on measuring recovery time vs downtime, incident response practice sessions - Developer experience: Document standardized deployment patterns, create self-service toil identification tools - Developer marketing: Campaign around proactive vs reactive engineering culture, supply chain security awareness - Community management: Facilitate experience sharing between platform teams, moderate discussions on automation ROI #### OSPO - Policy & Governance: Contribution policies for dependency management, standardized security response procedures - License Compliance: Automated scanning for supply chain vulnerabilities, training on ecosystem attack vectors - Security Management: Vulnerability intake for npm/Maven/PyPI dependencies, security attestations for critical components - Software Inventory: SBOM strategy for cross-ecosystem dependencies, ownership mapping for security incidents - Developer Support: Training on secure dependency practices, tooling for automated upgrades - Community Engagement: Upstream funding for security maintenance, agreements with package maintainers - Strategic Alignment: KPIs for toil reduction, OKRs for incident recovery time improvements #### Community - Maintainers: Roadmap support for security-focused releases, governance templates for incident response - Contributors: Onboarding programs for security-aware development, first-issue programs in tooling repos - Sponsors: Transparent budgets for security infrastructure, impact reports on toil reduction investments - Ecosystem: Cross-project coordination for supply chain security, shared standards for dependency management #### WordPress - Core: Performance budget enforcement, automated security scanning for contrib themes/plugins - Core Performance: Profiling automated deployment processes, metrics for plugin activation impact - Core AI: Guardrails for AI-assisted dependency updates, evaluation sets for security risk assessment - Mobile: Telemetry for app deployment failures, UX improvements for security notifications - Playground: Labs for testing plugin compatibility, reproducible environments for security testing - Community: Meetup kits on security best practices, WordCamp support for incident response talks - Support: SLAs for security incident escalation, knowledge base for common toil scenarios - Design: UX audits for admin security interfaces, design tokens for alert/warning states - Documentation: DevHub security guidance gaps, style guides for incident communication - Plugins: Security liaison programs, provenance checks for automated submissions - Hosting: Cross-host security checklists, telemetry for common deployment failures #### The WPCC - Programs: Fellowships for security-focused contributors, microgrants for toil reduction tooling - Projects: Governance clinics for incident response, documentation sprints for security procedures - Funding: Matched grants for dependency maintenance, milestone payouts for security improvements ## 🧪 Hypotheses - Organizations that measure recovery time alongside downtime will invest more proactively in toil reduction than those measuring only availability metrics - Teams practicing incident response during calm periods will demonstrate faster mean time to recovery than reactive teams - Standardizing technology choices across engineering teams reduces both security response time and cognitive load for individual contributors - Automation that changes behavior under pressure provides higher ROI than automation that generates additional monitoring signals ### 📊 Methodology - Comparative study of incident response times across organizations with different measurement practices - Longitudinal analysis of teams before/after implementing regular incident response drills - Survey of engineering cognitive load across organizations with varying levels of technology standardization ### ❓ Questions - How can organizations quantify the business impact of trust erosion from slow incident recovery? - What specific guardrails most effectively prevent deployment process bypassing under pressure? - Which automation investments provide the highest ROI for reducing cross-ecosystem supply chain risk? ### 🔮 Future Research - Correlation analysis between technical standardization levels and security incident response times - Behavioral studies on engineering decision-making under different levels of operational pressure - Economic modeling of toil reduction investment vs reactive incident costs ### 💡 Implications - Practice: Engineering teams should allocate dedicated time for toil pattern identification and historical tracking - Policy: Organizations need governance frameworks that make secure practices the default deployment path - Security: Supply chain attacks increasingly cross ecosystem boundaries, requiring holistic dependency management - Education: Engineering curricula should include sociotechnical system design and human factors in reliability - Funding: Business cases for toil reduction should emphasize recovery time and organizational trust metrics over pure availability ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical audience appropriate) #### 👀 Visual learners - Present: Panel discussion format with speaker identification - Improve: Incident timeline diagrams, toil impact visualization, dependency relationship maps #### 🔊 Aural learners - Present: Full 45-minute audio discussion with multiple perspectives - Improve: Chapter markers for key topics, downloadable audio segments #### 🗣️ Verbal learners - Present: Rich metaphors (sandworm attacks, winter is coming), clear definitions - Improve: Glossary of SRE terms, analogy comparisons between different toil types #### 🧩 Kinesthetic learners - Present: Specific practice recommendations (fire drills, resilience testing) - Improve: Step-by-step incident response checklists, hands-on automation workshops #### 🧠 Logical learners - Present: Cause-effect relationships between toil and organizational outcomes - Improve: Decision trees for toil prioritization, ROI calculators for automation investment #### 🧑‍🤝‍🧑 Social learners - Present: Panel format encourages multiple perspectives and experience sharing - Improve: Discussion prompts for team retrospectives, peer review templates for toil identification #### 🧘‍♀️ Solitary learners - Present: Individual reflection opportunities on organizational toil patterns - Improve: Self-assessment worksheets, personal action planning templates #### 🌿 Naturalistic learners - Present: Ecosystem metaphors for supply chain dependencies - Improve: Lifecycle thinking about software dependency health, environmental impact considerations ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear speaker identification, structured content flow - Improve: Time-stamped transcript with speaker labels, ARIA landmarks for navigation #### Keyboard navigation - Present: Standard video player controls - Improve: Skip links for major topics, keyboard shortcuts for timestamp navigation #### Contrast and readability - Present: Good text contrast in player interface - Improve: High contrast mode support, customizable text sizing #### Text resizing and reflow - Present: Responsive player interface - Improve: Reflow support for transcript content, zoom compatibility #### Media and captions - Present: Audio content available - Improve: Professional captions with technical term accuracy, downloadable transcript #### Forms and interactions - Present: Basic video controls accessible - Improve: Accessible feedback forms for panel Q&A, keyboard-navigable timestamps #### Standards reference - WCAG 2.2 AA compliance needed for professional captions and transcript navigation ### 🌍 DEIB Assessment --- ## Full Transcript [00:12] Hi everyone, thanks for joining us today [00:14] for a discussion on rooting out sneaky [00:16] sources of toil. This is a lead dev [00:18] online panel discussion created in [00:19] partnership with Chain Guard. The [00:21] discussion is going to last about 45 [00:23] minutes after which both myself and the [00:24] panelists will head over to the lead dev [00:26] Slack channel to answer your questions [00:28] in the panel Q&A channel. We might also [00:30] have some time to answer questions [00:32] throughout. Um, if you want to take a [00:34] minute right now to actually open the [00:35] Q&A section. You can go and you can type [00:37] in questions. I'm going to do my best [00:38] throughout to maybe weave in some of [00:40] your questions to get the real answers [00:42] that you want and then at the end to [00:44] save some time. Anything that we don't [00:46] answer, we'll head over and we can put [00:48] it into the Slack afterwards to make [00:49] sure that you get the questions that you [00:50] want answered. [00:52] So with that in mind, let's get started [00:54] with some introduction. My name is [00:56] Dominique Sino Richie. I'm the CTO at [00:58] Inspiron and I've led engineering teams [00:59] at scaleups like Lever, Shopify, [01:02] Affinity. Today I'm joined by Malikica [01:04] Rao, Chang Leong, Leslie Cordderero, and [01:06] Manfred Moser to share their experiences [01:08] and insights. I'm going to ask each of [01:10] you to introduce yourselves. If we can [01:12] kick it off with Malikica, go ahead. [01:15] Hi everyone. Thank you Dominic. Uh I'm [01:18] Malika. I'm an engineering leader. Spent [01:21] the last several years building and [01:23] operating highly scale distributed [01:26] systems. That's it at the center of [01:27] products, people, engineering and [01:30] products that people use every day. [01:32] Things like search recommendations [01:34] personalization systems and really at [01:36] the core of platform infrastructure. And [01:39] across uh Twitter, Walmart and most [01:42] recently Netflix. I've led teams that [01:44] are responsible not just for building [01:47] new capabilities but for keeping complex [01:50] systems reliable, scalable, especially [01:53] when they are shared by many downstream [01:55] teams and that high blast radius. And a [01:57] lot of my work has uh focused on [02:00] reducing operational and cognitive load [02:03] uh for our engineers for our IC's by [02:06] designing safer production workflows, [02:09] better recovery paths and clear um [02:11] ownership models. Um I'm excited for [02:15] this conversation because toil is often [02:17] treated as inevitable [02:20] uh when in my experience it's usually a [02:22] design signal. uh I care a lot about [02:24] building these systems and teams that [02:26] are uh resilient enough to learn under [02:29] pressure instead of burning out. So [02:31] definitely looking forward to the [02:33] discussion. [02:35] Hey everyone, I'm Chang. Uh I'm a [02:37] software safety nerd. Um and I've been [02:40] working on software for the first 25 [02:42] years. Uh I was constructing software [02:44] really with an eye on the computers and [02:46] systems. And for the past 5 years I've [02:49] been uh turned on to the human side of [02:51] things. uh it's a sociotechnical system [02:53] and so we're sort of missing half the [02:55] equation here the people side of things [02:57] and so uh I joined um an incident [03:01] analysis team 5 years ago and that was [03:04] really eye opening as to the ways we can [03:07] um look to monitor signals from the [03:09] system about the humans and uh give us a [03:12] whole set of interventions that are on [03:14] the human side not necessarily on the [03:16] computer side focused for a long time [03:18] most of my career on the computer side [03:20] the code side the system side but sort [03:23] of ignored the humans that were part of [03:25] operating the whole thing. And so I'm [03:27] excited today to talk about toil which [03:29] is very much on the human side. Uh [03:31] computers don't I think feel toil as far [03:33] as I know. They don't have feelings or [03:35] emotions about it but uh certainly the [03:37] people who need to contribute to the [03:38] system do. And so I'm excited to talk [03:41] about uh that today. [03:45] Leslie. [03:48] >> Uh hi everyone. Uh my name is Leslie. [03:50] Um, oh, can you hear me? Okay. Okay. It [03:53] fl it like flinched for a second. Sorry [03:55] about that. Um, my name is Leslie. Uh, [03:57] I'm currently a staff engineer and tech [03:59] lead um at the New York Times. Um, I'm [04:01] the tech lead for both the uh site [04:04] reliability engineering team and the [04:05] reliability platforms team. Um, and so [04:08] I've been here for about three and a [04:10] half years. Um, always working on [04:12] reliability related problems under our [04:14] platform engineering organization. Um, [04:16] so some of the things I've worked on [04:18] were election readiness for uh the pre [04:21] uh the general election that just [04:22] passed. Um, and then also in general [04:24] just preparing for high-skll um events. [04:27] So uh we have New York Times cooking for [04:30] example. So Thanksgiving is a big time [04:31] for us. Um wire cutter um Amazon Prime [04:35] Day um Black Friday are huge days for [04:37] us. Um so yeah that's kind of been my [04:40] pre most recent experience. And then [04:42] before that I was um in education [04:44] technology. Um also was there was in the [04:48] industry for a big high uh high skill [04:50] moment which was co and switching to [04:52] remote learning. Um so at the time I was [04:54] actually at on the Google for education [04:56] team. Um and uh we uh Google classroom [04:59] scaled from about 40 million users to [05:02] over 110 million users in the matter of [05:04] weeks. Um so needless to say I grew an [05:07] appreciation for um production and [05:10] debugging in production. um and also saw [05:12] the pains of toil um in real time. Those [05:15] things that we had kind of just been [05:17] putting off for because it seemed low [05:19] value um really came to uh kind of bite [05:23] us in the butt during that time. So, uh [05:26] excited to talk about all that stuff. [05:29] >> Uh thanks everyone. Uh my name is [05:31] Manfred Boza. I'm a European background [05:35] turned Canadian that uh has been in the [05:37] open source ecosystems well over 20 well [05:40] over two decades. Um I with a bit of a [05:43] focus in the Java ecosystem but also [05:46] lots of work in the CI/CD space DevOps [05:48] and that kind of stuff. I'd been doing [05:50] that stuff before it was called DevOps. [05:52] Um and um I have a long history in the [05:56] build tools and software supply chain [05:59] technologies. I'm an Apache Maven [06:01] committer, a Trino contributor and uh [06:04] involved in various other effects. But [06:06] these days I've been pretty much turned [06:08] into an educator. So I work as a [06:10] developer relations engineer at Chain [06:12] Guard where we look at the components [06:15] that build that are used to build your [06:17] applications and hopefully provide you [06:19] safe and secure ones. So well looking [06:21] forward to talk about toil because it's [06:24] always interesting to see when people [06:25] talk about toil. I'm like well that's [06:27] just like the normal work of a software [06:29] engineer. [06:31] >> Awesome. Yeah to your point a lot of the [06:34] teams accept some level of toil as [06:36] normal. They might not even realize that [06:38] it's toil. It's can be self-inflicted. [06:41] It might be invisible. I find that it [06:44] shows up as either constant context [06:46] switching, surprise security [06:48] vulnerabilities, or hey, some of our [06:50] core dependencies are not available when [06:51] we need them to be available, reactive [06:54] incident work, and even layers of [06:56] process where like you're doing [06:57] something over and over even though it [06:59] doesn't provide the value that it once [07:00] did or maybe there's a new way, a better [07:03] way to do it. Now, what I find in [07:06] talking to individual engineers, they [07:07] don't often see the patterns. It doesn't [07:09] feel like a system and systemic problem [07:11] in the moment. because they show up as [07:13] one-offs, but over time to your point a [07:16] few of you mentioned this, they [07:17] compound. Whether that's because you're [07:19] you have more traffic, more users, more [07:22] people. Sometimes it can be for other [07:24] reasons as well. You have more engineers [07:26] contributing to the same codebase. But [07:27] then all of a sudden, I do find that [07:29] it's worth to start investing and [07:31] thinking about what proactive looks [07:32] like. And so today's conversation is [07:34] going to focus on naming the real [07:36] sources of toil that exist, showing how [07:39] strong teams create urgency without [07:42] waiting for that urgency to be created [07:44] for them by major external factors. And [07:47] then I'd love to get as many great [07:49] examples from this panel as possible so [07:51] that you can all walk away with actual [07:54] things and things that you can try on [07:55] your individual teams or in your [07:57] organizations. [07:59] And so with that in mind, let's start [08:00] with what you consider toil and what do [08:03] you see show up the most in your [08:05] organizations? Uh maybe Leslie, we could [08:08] actually start with you. What do you see [08:09] as the main sources of toil in your [08:11] organization? Where do you see unplanned [08:13] engineering work come from? [08:15] >> Uh yeah, so well the times has been [08:19] around for quite a while. It's our 175th [08:21] birthday this year. I know. Um, and so [08:24] we, you know, I always say we've been [08:26] doing technology and software longer [08:27] than Google has been alive, right? Um, [08:30] so, uh, we have a lot of toil in pretty [08:33] much any type of manifestation. Um, but [08:36] I think some of the biggest sources of [08:38] toil, um, I tend to see have to do with, [08:41] um, the quality of our data actually. [08:44] Um, not having like strong sources of [08:46] truth with respect to like service [08:48] ownership data, um, service data in [08:50] general. Um, and from a platform [08:52] engineering perspective, that's been [08:53] that's really difficult when you're [08:56] talking about like adoption for example [08:58] and tracking down like, okay, this [08:59] issue, who do I escalate it to? And so [09:02] there's just like a lot of sort of [09:03] wasted time just from that exercise, [09:05] which again, it's like a platform or [09:08] reliability management team. Um, that's [09:10] a lot of what you're doing is like [09:12] nudging other people to improve their [09:13] systems. So um, that's been one like [09:16] that's been one source of toil that's [09:18] been on my mind quite a lot lately. So [09:20] that's why I'm talking about it now. Um, [09:22] and then there are also other sources [09:23] like classic sources of truth where you [09:25] know like most companies are monitors [09:28] and alerts are a little bit too noisy or [09:30] maybe a lot of too noisy. Um, and then [09:33] uh in general just I think like things [09:35] like spinning up a new service there's [09:37] traditionally been a lot of toil around [09:39] that. Um, just kind of the like classic [09:41] workflows of a software engineer. um [09:43] we've been working the last five plus [09:46] years to really platformize that um by [09:48] making having there be like centralized [09:50] tooling and whatnot. So um yeah, those [09:53] are some of the sources that of truth of [09:54] toil that uh come up for me. I'm sure I [09:57] could keep going, but I'll let someone [09:59] else chime in. [10:03] >> I like how [10:03] >> Yeah. Uh [10:04] >> no, you go. [10:06] >> Uh yeah, thank you. So when I think [10:08] about toil, I I really don't think about [10:11] uh hard work. uh I think about that [10:14] unplanned repeatable effort that doesn't [10:16] meaningfully improve our systems. The [10:19] kind of work that keeps smart engineers [10:21] busy but it doesn't really make things [10:23] safer or simpler uh for the next time. [10:26] So when I see toil show up uh most uh [10:30] today is in actually three places where [10:33] recovery paths aren't engineered, safety [10:35] mechanisms that are optional or easy to [10:38] bypass and really that operational [10:40] signals that create noise uh to Leslie's [10:43] point instead of that clarity. So a lot [10:45] of that unplanned oil comes from the gap [10:48] between when a system is technically [10:50] backup and when it's actually safe, [10:53] trusted and usable. Again those are two [10:55] different things and that gap is where I [10:58] think engineers lose time, teams lose [11:00] confidence and innovation quietly [11:02] stalls. So yeah I'm curious on your all [11:05] thoughts as well. [11:12] >> It was interesting to say see earlier [11:14] when Lissy was mentioning that it's like [11:16] often like people just not understanding [11:19] what what components are used [11:21] internally. I think that's a very common [11:23] scenario especially in the software [11:25] engineering side when we talk about like [11:28] different libraries being used and [11:30] different language ecosystems being used [11:32] people are like what we're still using R [11:34] or well there's someone using PHP in our [11:36] organization or like there's some I [11:38] don't know even some WordPress or [11:39] whatever running somewhere in the [11:41] organization in large companies and [11:43] that's already like unknown and then [11:45] when it comes to security issues in [11:47] those individual libraries understanding [11:49] what versions of what libraries are used [11:52] where is often like completely unknown [11:55] and that's often creates these like all [11:58] hands on deck. There's some craziness [11:59] going on in the ecosystem, some security [12:01] attack and then people just don't know [12:04] where things are even happening and [12:05] that's like I think that there's those [12:07] two differences of toil. has the like [12:09] the completely unplanned crazy [12:12] everything is like on fire let's have we [12:14] have to fix this and then the regular [12:16] one where it's just like let's do like [12:17] the dependency updates and like make [12:19] sure everything is secure and [12:21] unfortunately [12:23] the less you do the regular sort of toil [12:25] work the bigger the fire is when it then [12:28] happens and item always happens there's [12:30] always something coming up right like [12:31] some software end of life so I'm sure [12:34] Chang has lots of experience with those [12:36] kind of fun events [12:37] >> yeah Actually, Chen, you one of the [12:40] things you said to me that really struck [12:41] me is like people assume that S sur work [12:43] is all toil, but you actually draw the [12:46] line between what you consider toil and [12:49] what is like planned operational work. [12:51] What's your kind of what's your take on [12:52] what toil is? [12:54] >> Yeah, it's interesting uh that you bring [12:56] it to that because I as I was [12:58] mentioning, I really think about the [12:59] human experience of it and um when you [13:02] go look up the definition of toilet, [13:03] it's just work [13:05] >> and it's work. So presumably the work [13:07] needs to be done. Um and obviously if it [13:10] it doesn't need to be done, we can not [13:12] do it. Uh so that's a good conversation [13:14] to have to eliminate what you think of [13:15] as toil or like burdensome work. But [13:18] what um what I was thinking was that uh [13:24] the the planned work you could you could [13:26] move it all to planned. It's not like [13:27] you didn't see your software upgrades [13:30] coming library upgrades coming. um you [13:34] could uh foresee that as work that is [13:36] necessary. And one of the things that I [13:38] think um people think of toil as bad or [13:42] undesirable is because it doesn't use [13:44] their skills or it's not valued by the [13:46] business. And so a way you can change [13:48] toil is perhaps by reframing it by the [13:50] people who are asking for the work to be [13:52] done. Why does this work need to be [13:53] done? Is it valuable for the business? [13:55] If it's valuable for the business and [13:57] maybe it's hard to do, well, maybe [13:58] that's worth investing in. If it has a [14:00] key value driver, a key value uh revenue [14:02] generator or something for you, like we [14:04] need to do this, but it's hard. Okay. [14:06] Well, maybe if it makes us money, we [14:08] should spend effort in making it good [14:10] and easy for us to make money. Um, and [14:13] if it's uh not valuable, why is it not [14:16] valuable? Is it because there's [14:17] expertise that's hidden? Um, one way you [14:20] can maybe reveal that is by asking AI to [14:22] do it for you. uh if you think it's so [14:25] easy to do and doesn't require uh [14:28] doesn't generate a lot of value and [14:29] doesn't require skill, perhaps you can [14:32] ask an AI to do it, but obviously [14:34] monitor your AI, make sure it's doing [14:36] the thing you want. And if it's not [14:37] doing the thing you want, every time it [14:39] does the thing you didn't want, that's [14:41] an that's an expression of expertise [14:43] that the person who was doing it before [14:44] you gave it to the AI has. It's [14:47] something that's not encoded in the [14:48] system. And so I think that's a way to [14:52] reveal that expertise is valuable or the [14:54] work you're doing is actually valuable [14:57] and maybe worth investing more in if [14:59] it's so valuable or if it's not [15:01] valuable, don't do it. [15:06] Yeah, there's definitely a different [15:08] like perception of value that's very [15:11] important and I think that's also a bit [15:13] of a sign of how mature an organization [15:15] is, right? Like if if an organization [15:17] has mature leadership that understands [15:18] how important it is to be on like you [15:21] know not be stuck on Java 8 but use 25 [15:24] or whatever or like like use a later [15:26] version of npm and like like be on the [15:29] on the sort of more on the bleeding edge [15:31] than on the very old stuff just because [15:34] there's less security issues like that [15:36] requires mature leadership and [15:37] unfortunately the whole like oh we need [15:39] to ship this new feature and that's much [15:41] more important and write a press release [15:42] about it is always seen as more [15:46] important and I think it's a bit of a [15:47] sign of a mature leadership in the [15:49] engineering organization to understand [15:52] that this stuff is very critical [15:56] >> definitely and often teams won't take it [15:59] seriously especially when they're early [16:01] in the curve of development and maybe [16:03] they're trying to there is a careful [16:05] balance when you're growing a business [16:07] where you're like you have to actually [16:09] deliver what your customers need [16:10] otherwise you can have the most stable [16:12] system that you build, but actually they [16:14] won't be using it. And what I found is [16:17] that teams are forced to confront this [16:20] when a major dependency vulnerability [16:22] hits or maybe a major system outage. [16:24] We've had quite a few of these over the [16:26] last few years, like really significant [16:28] ones that I think none of us would have [16:30] expected as engineering leaders. And [16:32] then all of a sudden, people outside of [16:34] engineering are like, "Wait a minute, [16:36] what are these dependencies you have? [16:38] What should you be doing to invest in [16:39] the resilience of your system, etc." Um, [16:42] Manfred, maybe you have a story around [16:44] this is like when a new when a specific [16:46] open source supply chain attack makes [16:48] the news, why do you find that teams [16:50] actually slow everything down right away [16:52] or that it causes that urgency? [16:55] >> It's been, let's just say, very exciting [16:58] in the last couple of months to look [16:59] from the outside. I'm glad I wasn't [17:01] involved in any of these companies that [17:03] dealt with that, but I we have customers [17:05] that are in like these situations and [17:07] then come for us for help obviously. Um, [17:10] so there was the the like I'll talk [17:12] about the npm ecosystem. So the [17:14] JavaScript components that come out of [17:15] the npm registry. Um, I'm not saying the [17:19] other ecosystems don't have these [17:20] problems, but the npm ecosystem just was [17:22] in the in the in the news recently, [17:25] right? Like further back, you might [17:27] remember the log for shell kind of stuff [17:29] in the Java ecosystem. There's various [17:31] ones in the Python ecosystem, but in the [17:32] npm ecosystem specifically, last year we [17:36] had that chalk debug attack and then the [17:38] shy halloot and then the shahalude ones [17:40] which um were identity theft. So [17:43] basically people just sent a spam email [17:46] to a maintainer. the maintainer fell for [17:48] the trap, reset their two-factor [17:50] authentication, their credentials were [17:53] stolen, and then the open source [17:56] projects uh publishing rights were used [17:58] to publish malware into the npm [18:01] registry. What was so interesting and [18:03] why it was called the shahalude worm [18:05] which is like the worm's name from the [18:06] dune you know in the desert sand [18:09] sandworm kind of thing is these malware [18:13] that were infected they spread to other [18:16] components and over time it made like a [18:18] wave of attacks and um there was the the [18:21] first one the shy halloot and then shaw [18:23] one halude later a couple of weeks later [18:25] and they spread really wide and that was [18:27] the shocking thing that lots of [18:29] organizations completely misjudged [18:31] because things happened such as a [18:34] JavaScript component that was for a Java [18:36] server side application like a NodeJS [18:38] application was infected. Well, the same [18:41] JavaScript also were on the on the [18:43] client side. So, in the browser it went [18:46] further. Some of these components are [18:48] used in desktop applications especially [18:51] also developer applications. So, Visual [18:53] Studio Code for example is a very common [18:55] editor text editor for developers. It [18:58] has an extension ecosystem. It's written [19:00] in JavaScript. This is actually kind of [19:02] a browser application. So developer [19:04] workstations were infected. Secrets were [19:07] stolen from the developer workstations [19:08] and then were used to publish more [19:10] GitHub stuff and cause a whole bunch of [19:13] more more headaches, right? And then [19:15] even it spread even ecosystem. So some [19:18] of those JavaScript components were then [19:20] found in fullbaked applications that [19:22] were deployed to Maven Central. So the [19:25] idea that you're just safe if you're [19:27] using one ecosystem but not the other, [19:29] that's gone. Like everything is so [19:31] complex and I'm sure like Malik or [19:33] Leslie have an idea of like what they're [19:35] dealing with. Like there's no such thing [19:37] as a company that just uses one stack [19:39] and unfortunately the security issues [19:41] also spread across ecosystems now. And [19:43] that's and you don't know that. And so [19:46] that's why it's often like all hands on [19:48] deck. We need to find out what's going [19:49] on. shut everything down, reset all [19:51] credentials and stuff like you need to [19:53] have processes for that because [19:55] otherwise ideally even have shortlived [19:57] credentials. There's a lot of things you [19:59] can do. [20:01] >> Chang, you have a couple of great [20:02] examples of like things completely [20:04] shutting down and forcing the [20:05] prioritization. Do you want to share? [20:08] >> Uh, sure. Uh, Manford referenced log for [20:11] shell back in the day uh in the Java [20:13] ecosystem. um we had uh thousands and [20:18] thousands of uh service microservices [20:21] and systems using Java and um in this [20:25] one we needed to upgrade basically all [20:27] of our libraries uh to at the same time [20:30] and we didn't even have a really good [20:31] way internally not just of kicking off [20:34] all those builds and even if we did do [20:37] the upgrades but even discovering them [20:39] uh we didn't have the tooling internally [20:41] for that. So that required a bunch of uh [20:44] last minute scramble um to basically do [20:48] what Manford describes as something [20:50] routine that you should be doing all the [20:51] time. It should be very easy and we [20:53] should have tools for but we neglected. [20:56] Um so that was one of them. And another [20:58] one uh I guess is is less of a uh a last [21:02] minute scramble due to security stuff [21:04] but a few years ago there was a winter [21:08] storm uh similar I guess in the US to [21:10] what we had last week and an entire [21:12] region's uh energy provider um system [21:16] was unstable. It was under threat of [21:19] being overloaded and at some points um [21:22] they thought they may have to shut the [21:23] whole thing down preemptively. We had at [21:25] the time uh in-house data centers housed [21:29] at several different cities but all [21:31] within one region. So we thought oh [21:33] geographically it would be a pretty big [21:35] asteroid. Well it you know the weather [21:36] is a pretty big asteroid and so we were [21:39] in the process of uh reducing that risk. [21:42] Uh it was a year uh a multi-year effort [21:46] to get into the cloud out of our own [21:48] data centers. But due to this winter [21:50] storm um we accelerated that and were [21:53] able and the company executives [21:55] recognized this was an existential [21:56] business threat to the business. The way [21:58] to get money is to have your website up. [22:00] No website, no money. Well, they [22:02] recognized that this was a major risk [22:04] even with their backup data center in a [22:06] different uh city several hundred miles [22:07] away. Not enough. Uh so they accelerated [22:10] that. Several years was not soon enough. [22:12] So they said in one year we're going to [22:15] move to the cloud. [22:17] and um basically dropped half a dozen [22:20] other companywide annual objectives and [22:22] said this is the one thing and the [22:24] nickname for it was winter is coming and [22:27] uh you know Game of Thrones style. It we [22:30] did it in one year which was several [22:31] years faster than we had planned and [22:33] turned out to actually save quite a [22:35] bunch of money um in the short term. [22:38] >> I mean that's incredible. I hope that [22:40] most of the engineering leaders that [22:42] watch this don't have to go through that [22:44] in order to prioritize toil. I think [22:46] Malikica, you've got some really great [22:48] examples where it's more around seeing [22:50] the cumulative effect, which I think is [22:52] really applicable to PE people starting [22:54] to think about how can I start [22:55] investing? What can I do? Do you want to [22:58] share a little bit about that and then [22:59] we let's maybe shift after that into [23:01] some real practical techniques that [23:03] people can use. [23:05] >> Yeah, I think this um discussion around [23:09] how does it really become a forcing [23:11] function is an important one. Uh I I'll [23:13] tell you the story from uh Twitter [23:16] search infra. Uh for us the forcing [23:19] function wasn't really a dramatic [23:20] outage. It was a slow trust erosion. Uh [23:24] we own core search infrastructure used [23:26] by a lot of downstream product and ML [23:28] teams and incidents weren't typically [23:31] constant but recovery was very slow [23:34] manual and unpredictable. Those were the [23:36] signals that we were uh seeing uh time [23:39] and again. And I think the real breaking [23:41] point was when a data backfill after you [23:45] know the index came down because of an [23:47] upstream trigger. Um there was an [23:50] incident in incident uh brought down the [23:52] index and the data backfill basically [23:54] took us 7 days. The system technically [23:58] came back up within hours right but [24:00] teams uh that depended on us were [24:03] blocked for a full week and that's when [24:06] ignoring toil was no longer an option. [24:09] So because recovery time had become a [24:11] bigger product problem than the incident [24:13] itself. So that was a breaking point for [24:16] us and what surprised me the most was [24:19] how much damage showed up as invisible [24:21] product drag right like teams delayed [24:24] launches product experiments in [24:26] downstream teams froze and people [24:28] literally stopped trusting our timelines [24:30] and SLAs and even how we defined our [24:33] SLOs's and SLI. So in real terms, I [24:36] think it cost us that senior engineering [24:39] time, roadmap momentum and platform [24:41] credibility. Um that's what pushed us to [24:45] start measuring recovery and backfill [24:47] time explicitly and not just outages or [24:50] how do we triage um and to really treat [24:54] these incidents as a systems design [24:56] problem and not just as an ops problem [24:58] to uh uh to you know Leslie's point uh [25:02] that she was making earlier. Right? So I [25:05] think the biggest cause wasn't downtime [25:07] right it was that decision paralysis [25:09] that happens across the organization and [25:12] to your all point I do believe that this [25:14] is so much of an organizational problem [25:16] and not just a technical one [25:19] >> I definitely think that one of the [25:20] things that's helped me has been when [25:22] I've started measuring the number of [25:24] incidents and the recovery time people [25:26] start to wonder why it's taking so long [25:29] to get back up and that recovery time [25:31] you mentioned that being able to use [25:33] that as a way to really demonstrate to [25:35] the business this is the thing worth [25:37] investing in. A while ago, I was at a [25:39] company where there was one particular [25:40] system that was so archaic. It took 50 [25:44] hours to get back up if it went down. [25:47] And that's just not acceptable. And it [25:49] was for real live like extremely [25:52] critical use case. And so luckily not [25:55] the entire system obviously would become [25:56] available, but like that was what made [25:59] us be able to invest within that one [26:01] year. And so I do think measuring plays [26:02] a really really big role. And so maybe [26:05] shifting to then now what's worked and [26:07] practiced for you. Leslie, let's start [26:10] with you. What's something that you've [26:11] done to maybe integrate toil reduction [26:14] or prioritize specific sources of toil [26:16] in your planning so that you can make [26:18] sure that you're actually being [26:20] proactive with it? [26:21] >> Uh yeah. So one thing that's important [26:24] for us is that we do distinguish um like [26:27] business as usual work um from toil. Um [26:30] so we always allocate some time for like [26:32] business as usual work which can be [26:34] which can be toil but ne not necessarily [26:37] is toil. Um, in terms of being more [26:39] proactive about eliminating toil, um, I [26:42] think, you know, a lot of it is about [26:45] juggling priorities, right? Um, and so [26:47] the way I think is just like kind of [26:48] making a case for the impact that it can [26:51] have in terms of like the work streams [26:53] that would reduce toil. Um, and [26:54] connecting them to like other other [26:57] times that it could have enabled that [26:59] like high important um, work that uh, [27:02] you know, the business actually needs. [27:03] Um so kind of identifying like patterns [27:06] like oh this issue keeps coming up and [27:09] it's related to toil like maybe now that [27:11] we've seen it um hindered our delivery [27:14] timelines multiple times maybe we should [27:17] consider it as part of our actual like [27:19] prioritized work. Um, so I think I try [27:22] to do those things sort of continuously [27:24] um and point those out continuously so [27:25] that way when it is planning time it's [27:28] not a surprise that I'm like bringing up [27:29] these um bringing up these uh work [27:32] streams um to kind of eliminate uh toil. [27:34] So definitely like keeping track of [27:37] things over of toil over time is [27:39] important. Um like showing that [27:41] historical pattern is going to be super [27:42] crucial. Um because otherwise it's hard [27:45] for leadership to really uh like [27:47] understand how much is actually [27:48] impacting you. Um [27:52] yes. Uh and some other things to keep in [27:54] mind. Um [27:56] oh I guess looking for like technical [27:58] solutions that uh address might address [28:01] like multiple forms of toil um is [28:04] another way. Um and also connecting it [28:06] to like re um incident management um uh [28:11] statistics like was just mentioned. Um [28:13] so for example we had a lot of toil [28:15] around like our observability platforms. [28:17] Um we had uh our data across multiple [28:20] platforms. We still do but it's better [28:22] now. Um and we showed that there was a [28:25] lot of toil in it just by doing like the [28:27] context switch of switching between um [28:29] platforms, right? Like our logs were [28:31] being collected in one vendor, our [28:32] traces were being collected another. Um [28:34] and just like that move that transfer [28:36] between the two like cost us time. Um [28:38] and that's assuming people even knew [28:41] where to find each of these signals, [28:43] right? Um and so we had a huge [28:45] initiative to um centralize our tra to [28:49] centralize our data and also to [28:50] implement um slinstrument uh our apps [28:53] with uh distributed tracing. Um and one [28:56] way that we showed uh the importance of [28:58] that work uh was by tying it back to [29:00] like actual um time to recovery and [29:03] things like that. Um and we could see [29:04] like an actual correlation between the [29:06] two um like the number um of services [29:09] instrumented and the time it took to [29:10] recover. Um, now in terms of what sort [29:13] of spearheaded the initiative, um, I was [29:16] proactive and highlighted this as like a [29:19] risk for election readiness actually. [29:21] Um, and but it was still getting people [29:24] to really value the work was an uphill [29:26] battle, right? Um, and it really did [29:29] take a huge incident um, for us to [29:31] really like prioritize it. Um, there was [29:34] a major news story um, which one was it? [29:38] um oh I think it was when when Trump uh [29:42] got shot um that brought down the times [29:46] like horribly and um that was and again [29:50] we were preparing for the general [29:51] election. So um that big issue was known [29:55] to everyone and it was even escalated [29:57] you know to our board of directors like [29:59] everyone knew about it and that was [30:00] really like the in the no pun incident [30:03] um that got us to prioritize a lot of [30:05] this work. Um and so ideally, you know, [30:07] you want to be uh ahead of those things [30:09] happening, right? Um and luckily we at [30:11] least did have a plan. It was just like [30:13] the priority and like became all the [30:16] more uh clear. So um but yeah, ideally, [30:19] you know, you're being more proactive [30:21] and um avoiding those sort of scary [30:24] times in general. [30:26] >> Yeah, we keep going down the thread of [30:28] incident response. So let's maybe go one [30:30] level deeper on the incident response [30:32] piece before we talk more about [30:33] planning. Chang, you I think you might [30:36] have some specific tips for how to [30:38] reduce incident response and to [30:40] proactively really think about how how [30:42] you can use it to be able to invest. [30:45] >> Yeah, perfect. Um yeah, thanks Leslie [30:48] for a great tea up uh of incident [30:50] response. Um Dominic referenced having [30:53] things to measure and show the business [30:55] that the the work we're doing to reduce [30:57] toil is valuable. And for me, the way to [31:00] do that is to set service level [31:01] objectives, businessoriented ones, not [31:03] about the systems, but about the user [31:04] experiences. Uh, can people log in and [31:07] do the things that they need to do to [31:09] get their business done for whatever [31:11] reason it is that they're using your [31:12] services? And if you specify that in [31:14] business terms, then executives will be [31:16] able to say, okay, I'm willing to pay [31:18] this much or invest this much effort in [31:20] order to achieve this level of [31:22] reliability or delivery. And as Manford [31:25] referenced, we need to practice our [31:28] incident response. And so the thing that [31:31] uh I would recommend is for teams to [31:33] come up with a system, a pattern, a [31:35] process, tooling, and practice [31:37] responding to incidents. If you don't [31:39] have them organically, then make sure [31:41] everyone gets practice in sort of spend [31:43] that uh that unspent budget. If things [31:46] are up, great. Um spend that budget that [31:49] the executives have given you. They said [31:50] it could be down for this much. We can [31:52] afford this much. use that investment to [31:55] have practice sessions and practice [31:58] responding, have a plan for how to [32:00] respond. As Leslie said, they didn't [32:02] necessarily make the investment, but [32:04] they had a plan. And so having a plan is [32:07] sort of half the battle. The other half [32:08] is being ready to execute that plan, [32:10] making sure everyone's on the same page. [32:12] And so I would recommend [32:15] uh setting business speak budgets for [32:19] how much the business is willing to [32:20] invest in it and spending that budget in [32:23] response time as Dominique said uh [32:25] measure stuff and a thing is recovery [32:28] and your recovery can be improved if you [32:30] are practiced at recovering. I think [32:32] I've seen businesses try and chase the [32:35] fool's errand of prevent everything. [32:38] Okay, if we prevent everything, we won't [32:39] ship any software. And so, we know [32:41] they're going to happen. Things are [32:42] going to happen, but we need to humans [32:45] provide that flexibility and adaptive [32:47] capacity to respond in a way that [32:50] writing code just doesn't. Um, so if [32:52] people are ready to follow a plan to [32:54] respond to a disaster, that's much more [32:57] adaptable to any kind of disaster, uh, [32:59] whether it's zombies attacking or [33:00] asteroids hitting the earth, um, than [33:03] any software you could have written it [33:05] in advance. [33:07] >> Yeah. So I still want to believe that [33:09] teams can really invest in like [33:11] proactively reducing toil and build a [33:14] plan. Sorry Leslie if you want to throw [33:16] throw in something and then then I can [33:18] go next. [33:19] >> Um yeah I was just going to say that [33:20] practice part I think is super [33:22] important. One of the things we did was [33:24] um you know resilience engineering and [33:26] res resilience testing. So um you know [33:28] we uh originally started mostly with [33:31] just like load testing right um and that [33:33] was how that's a great way of sort of [33:35] identifying those things proactively um [33:38] because I mean to a certain extent the [33:39] most interesting and impactful problems [33:41] that you can address are ones that you [33:43] don't know right um or at least you [33:45] don't know until they like happen so you [33:47] can simulate the scenarios in which that [33:49] might happen um so that's sort of like [33:51] one way I think about doing it more [33:53] proactively in a way that's not like [33:55] being too preventative Right. So much so [33:58] that you have to um give up the high [34:00] value work that you do need to deliver [34:02] to make your business sustainable. [34:05] >> Yeah. I also find these exercises of [34:07] resiliency testing where you just like [34:09] what happens if that cloud goes offline? [34:11] Let's simulate that. Those are actually [34:13] super fun for engineers to deal with [34:15] because like you're like you there's [34:18] always some sort of thing where you're [34:19] like oh we just switch this off it'll [34:21] fall over. Clean it to that it'll be [34:22] just fine. And then you're like no it's [34:24] not. that's falling over there and there [34:26] and like five other surprises and it's [34:28] actually like really fun detective work [34:30] to chase down what's going on. And if [34:32] you can do that in a pro in a proactive [34:34] exercise then the the pressure and the [34:38] panic of oh there financial damage and [34:40] like reputation blah all that falls [34:42] away. So you can like I I totally [34:44] believe in the like do a hackathon day [34:47] or like a reli resilience day where you [34:49] just like let's see which plugs we can [34:51] pull and how the thing falls over kind [34:53] of thing in a test scenario. And that's [34:55] I think it's totally fun for engineering [34:57] teams to do. [34:58] >> I've been able to connect it to business [35:00] objectives just by looking at like [35:02] what's the projected customer growth or [35:04] the traffic that we expect. And if you [35:06] model just a little bit of that, then [35:08] you're like, well, if our goal is to [35:09] have this many customers and therefore [35:11] this much traffic, then it's easy to [35:12] work backwards from there to do the load [35:14] testing for the right number of users [35:16] and to demonstrate we actually need to [35:18] fix these issues that we've identified [35:20] or we won't work by for uh by these big [35:23] events. And I've done that in places [35:25] where we didn't have a Black Friday or [35:27] we didn't have like a major event that [35:29] we expected. It was more just we know [35:31] that we're slowly growing and as part of [35:34] that growth there's a certain point [35:35] where some of our services are going to [35:37] fall over and it's been a great way to [35:39] not have to rely on urgency but instead [35:41] to be like here's the capacity that we [35:43] actually know we need to support based [35:45] on where we're going. Um yeah Malikica [35:48] you want to go ahead. [35:50] >> Yeah. No great points there by the [35:52] panelists. Uh I think one thing that uh [35:56] we learned the hard way and uh uh you [35:59] know kind of put it in practice uh [36:02] retrospectively was we stopped relying [36:04] on heroics and just made the safe path [36:08] boring and the default path and that [36:10] gave us a lot of u uh value. uh it [36:14] really came after an incident where a [36:16] critical production change bypassed our [36:18] shadow canary path and the system came [36:21] back up quickly but validating and [36:23] repairing those downstream implications [36:25] took much longer and that was creating a [36:27] hidden toil. So we created these truly [36:30] blameless reviews right where we [36:32] realized that the problem wasn't a human [36:35] error. if one person could do it, it was [36:37] basically like a blind spot in the [36:39] system itself and it was that our [36:42] deployment workflow made it possible to [36:44] uh skip those safety checks under [36:46] pressure. So we hardened the system [36:48] instead of asking people to be more [36:50] careful looked at our abstractions, our [36:52] guard rails, our contracts and we made [36:55] shadow canaries non-optional for certain [36:57] classes of changes and added automated [37:00] validation around it as well. So that [37:02] really invested in that post deploy [37:04] verification and recovery tooling. So [37:06] the result wasn't just like fewer [37:08] incidents right that's not the point but [37:11] it was really that faster calmer [37:13] recovery and much less operational noise [37:15] and I think Cheng mentioned this systems [37:18] are going to fail but can we build those [37:21] systems that will fail well teams that [37:23] can respond well and make learning [37:25] really fast and you know investing in [37:28] those fire drills operational reviews [37:30] premort has really paid uh a lot of [37:33] value so it really infors that belief [37:35] that I care deeply which is most teams [37:38] are back up fast but not back to safe [37:41] and that gap is where I think toil [37:43] really explode sometimes and worth [37:45] keeping an eye out for. [37:47] >> You made such an important point. So [37:50] when it comes to reducing toil, you can [37:52] use process and planning and all of [37:55] these great things, but actually you can [37:57] also automate and set guard rails to [38:00] make it such that especially the entire [38:02] organization, regardless of if they're a [38:05] feature delivery team, a project team [38:07] that doesn't have that SR or that [38:09] platform desire to do that, they can do [38:11] it. Leslie, you've probably had to put [38:14] in place some of those guardrails in [38:16] your roles. Is there any are there any [38:17] good examples of things that have worked [38:19] really well to kind of help force some [38:20] of those teams to either, you know, [38:22] upgrade their dependencies proactively [38:24] or to address some of the incident [38:26] response learnings? What has worked for [38:28] you? [38:29] >> Uh yeah, so I mean automation is [38:31] definitely a huge part of the job. Um so [38:33] you know basic things even just like [38:35] automatic upgrades so that you don't [38:37] even have to do it or think about it, [38:38] right? Um are pretty like table stakes [38:41] things to start off with. Um other [38:44] things that you might want to have in [38:45] place for uh in terms of automation is [38:47] like automatic rollbacks, right? Um if [38:50] an incident is uh if an issue is [38:53] detected, right? You can set that up um [38:55] with your observability and monitoring [38:57] platform. So that way, you know, it's h [38:58] it's happening in an automated way where [39:00] you don't have to like be watching your [39:02] dashboards um as it's like being [39:04] deployed to production and whatnot. Um [39:07] so things like that are, you know, [39:08] pretty like table stakes things I think [39:10] to have in place. Um and then there's [39:12] also, you know, going to be more [39:13] creative options that you can encounter [39:15] too. Um maybe what I said was already [39:18] considered um creative, but um you know, [39:21] having it like roll out based on having [39:24] um canary deployments, things like that. [39:27] Um there was something else I had in [39:30] mind, but I it left my mind. Um [39:34] >> I'm sure it'll come back. I'm I bet that [39:36] Manfred has a pretty strong opinion [39:38] here. [39:41] Well, I think I think one of the aspects [39:43] also that are worth thinking about and [39:45] that that came to mind when we're [39:47] talking earlier about the monitoring [39:49] system unless it was rolling out. I [39:51] think um standardization is sometimes [39:54] very useful like don't allow your [39:57] developers to make a separate monitoring [39:59] system and choose a different database [40:01] and a different version of a different [40:03] library and a different container with a [40:05] different operating system in it. So you [40:07] have to like don't allow all this like [40:10] uh like diversity and and like like [40:13] different choices just for the sake of [40:15] it like be bold enough to standardize on [40:18] like we are only using like I don't know [40:21] some specific containers that are super [40:23] lightweight don't have stuff in it um [40:26] for that standardize on one version of [40:29] whatever the spring framework or [40:31] whatever and like stick to docker don't [40:33] use vms and docker and some other [40:35] mechanism and docker compose and [40:37] whatever else and like desktops running [40:39] under like you know servers running [40:40] under your desk or something like [40:42] standardize and then have those all feed [40:45] into the same observability system like [40:48] Lesie was saying right so those those [40:50] help a lot and that then also allows you [40:52] to better understand where any security [40:54] issues come in play and then ideally you [40:57] have process in place to make sure you [41:00] use the the latest and greatest versions [41:02] of those dependencies or you have some [41:04] vendor that can provide you security [41:05] backports if you're really stuck on some [41:07] old one. So, but you need to understand [41:09] what's going on, right? And you need to [41:11] allocate time for doing that work. [41:13] >> Yeah, that brings up a really good topic [41:15] of uh governance, technical or [41:18] operational governance. Um, which I [41:19] think is super important, especially for [41:21] like large organizations or enterprises. [41:23] Um, where a lot there's a lot of toil in [41:26] the cognitive like work of of figuring [41:28] out how to do something, right? Um, when [41:30] you have standards in place, um, that [41:33] kind of work is already done for you, [41:35] right? um like to a certain extent you [41:37] know a lot of the technical choices that [41:39] we make um obviously require tradeoffs [41:42] right but it's almost more important to [41:44] have a like faster way to deliver it [41:46] than to have the right choice right um [41:50] and also the right choice is very [41:51] subjective right so um definitely so I [41:55] think having like strong governance in [41:57] place makes all these things a lot [41:58] easier um because then centralized teams [42:00] if they know if they know that they can [42:02] rely on you know assumption in the [42:05] technical systems they can build [42:07] solutions for those technical systems. [42:08] So having like a a minimized amount of d [42:11] of drift I think is like always going to [42:13] be a good thing um for those things. Um [42:15] so like in observability for example um [42:18] you know we have some centralized [42:19] instrumentation libraries. Um there's [42:21] every pretty much every language exists [42:23] at the times because again we'reund and [42:25] something years old. Um but um it was [42:29] it's now that we're standardizing on [42:31] like four central languages, we were [42:32] able to just kind of prioritize those [42:34] languages, right? Um and that also [42:36] allowed us to show the difference and [42:38] like this is how long it would have [42:39] taken you to um instrument your [42:42] application if you had used one of our [42:43] standardized languages versus if you're [42:45] using our non-standardized languages. [42:47] Now you have to figure it on your own. [42:48] It's like a difference in weeks of the [42:50] matter. So that also kind of builds the [42:52] case for addressing these sources of toy [42:55] or technical debt. [42:57] >> Awesome. I think I'm going to wrap us up [42:59] here unless Malikica, you want to throw [43:01] in one quick one last maybe good example [43:04] or practice for the team and then I'm [43:07] going to wrap us up just because we're [43:08] up at the hour or at the 45 minutes. [43:11] >> No. Awesome. Makes sense. Dominic, I was [43:13] just going to quickly say that [43:14] automation is worth investing uh in when [43:18] it changes behavior under pressure, [43:20] right? Not when it just produces more [43:22] signals. Preventive tooling should [43:24] remove like entire classes of failure, [43:27] not just create more alerts for humans [43:29] to try edge. So the best automation in [43:31] my mind is what makes quiet [43:35] disappearance of problems before [43:37] engineers ever have to think about them. [43:40] So let it target those classes of [43:42] failures and not just those one-offs. [43:45] Those probably don't need the um cost of [43:48] putting in that automation in place. [43:50] Yeah, [43:51] >> what an amazing point to end on so that [43:53] people can actually think about what [43:55] they can start by automating that won't [43:57] create more noise but actually help [43:59] people. That's really really awesome. So [44:02] thank you everybody for all of your [44:04] insights today on today's panel. [44:06] Obviously, the most damaging toil [44:08] becomes visible when teams are actually [44:10] willing to track it. I think a lot of [44:12] you shared good examples of incident [44:14] response and data being able to be use [44:16] that in planning as a really really [44:19] great way and you don't have to wait for [44:22] failure to justify prioritization. [44:24] Um, I think maybe the the thing we [44:26] didn't touch on, but good for wrapup is [44:28] that you do need sponsorship and you did [44:31] all kind of touch upon it at these [44:33] different pieces. And sometimes that [44:34] sponsorship comes from an urgent piece [44:36] of it. It comes from having identified [44:38] proactively the risks and the work to do [44:40] in order to do it. But ultimately seeing [44:43] your role as an individual engineer, [44:45] engineering leader as raising that risk [44:47] that you can proactively get that [44:49] sponsorship will be really really [44:50] impactful for all of your success as you [44:52] want to go take away and and draw this [44:54] as as a next step for your you and your [44:56] team. So thanks for all the panelists [44:59] for sharing your thoughtful real world [45:00] insights. We are going to head over now [45:02] to the lead dev slack channel in the [45:04] panel Q&A channel and you can continue [45:06] the discussion or ask any questions that [45:08] you have there. Thanks for joining and [45:09] see you next time.