Rooting Out Sneaky Sources of Toil  Hi everyone, thanks for joining us today for a discussion on rooting out sneaky sources of toil. This is a lead dev online panel discussion created in partnership with Chain Guard. The discussion is going to last about 45 minutes after which both myself and the panelists will head over to the lead dev Slack channel to answer your questions in the panel Q&A channel. We might also have some time to answer questions throughout. Um, if you want to take a minute right now to actually open the Q&A section. You can go and you can type in questions. I'm going to do my best throughout to maybe weave in some of your questions to get the real answers that you want and then at the end to save some time. Anything that we don't answer, we'll head over and we can put it into the Slack afterwards to make sure that you get the questions that you want answered. So with that in mind, let's get started with some introduction. My name is Dominique Sino Richie. I'm the CTO at Inspiron and I've led engineering teams at scaleups like Lever, Shopify, Affinity. Today I'm joined by Malikica Rao, Chang Leong, Leslie Cordderero, and Manfred Moser to share their experiences and insights. I'm going to ask each of you to introduce yourselves. If we can kick it off with Malikica, go ahead. Hi everyone. Thank you Dominic. Uh I'm Malika. I'm an engineering leader. Spent the last several years building and operating highly scale distributed systems. That's it at the center of products, people, engineering and products that people use every day. Things like search recommendations personalization systems and really at the core of platform infrastructure. And across uh Twitter, Walmart and most recently Netflix. I've led teams that are responsible not just for building new capabilities but for keeping complex systems reliable, scalable, especially when they are shared by many downstream teams and that high blast radius. And a lot of my work has uh focused on reducing operational and cognitive load uh for our engineers for our IC's by designing safer production workflows, better recovery paths and clear um ownership models. Um I'm excited for this conversation because toil is often treated as inevitable uh when in my experience it's usually a design signal. uh I care a lot about building these systems and teams that are uh resilient enough to learn under pressure instead of burning out. So definitely looking forward to the discussion. Hey everyone, I'm Chang. Uh I'm a software safety nerd. Um and I've been working on software for the first 25 years. Uh I was constructing software really with an eye on the computers and systems. And for the past 5 years I've been uh turned on to the human side of things. uh it's a sociotechnical system and so we're sort of missing half the equation here the people side of things and so uh I joined um an incident analysis team 5 years ago and that was really eye opening as to the ways we can um look to monitor signals from the system about the humans and uh give us a whole set of interventions that are on the human side not necessarily on the computer side focused for a long time most of my career on the computer side the code side the system side but sort of ignored the humans that were part of operating the whole thing. And so I'm excited today to talk about toil which is very much on the human side. Uh computers don't I think feel toil as far as I know. They don't have feelings or emotions about it but uh certainly the people who need to contribute to the system do. And so I'm excited to talk about uh that today. Leslie. >> Uh hi everyone. Uh my name is Leslie. Um, oh, can you hear me? Okay. Okay. It fl it like flinched for a second. Sorry about that. Um, my name is Leslie. Uh, I'm currently a staff engineer and tech lead um at the New York Times. Um, I'm the tech lead for both the uh site reliability engineering team and the reliability platforms team. Um, and so I've been here for about three and a half years. Um, always working on reliability related problems under our platform engineering organization. Um, so some of the things I've worked on were election readiness for uh the pre uh the general election that just passed. Um, and then also in general just preparing for high-skll um events. So uh we have New York Times cooking for example. So Thanksgiving is a big time for us. Um wire cutter um Amazon Prime Day um Black Friday are huge days for us. Um so yeah that's kind of been my pre most recent experience. And then before that I was um in education technology. Um also was there was in the industry for a big high uh high skill moment which was co and switching to remote learning. Um so at the time I was actually at on the Google for education team. Um and uh we uh Google classroom scaled from about 40 million users to over 110 million users in the matter of weeks. Um so needless to say I grew an appreciation for um production and debugging in production. um and also saw the pains of toil um in real time. Those things that we had kind of just been putting off for because it seemed low value um really came to uh kind of bite us in the butt during that time. So, uh excited to talk about all that stuff. >> Uh thanks everyone. Uh my name is Manfred Boza. I'm a European background turned Canadian that uh has been in the open source ecosystems well over 20 well over two decades. Um I with a bit of a focus in the Java ecosystem but also lots of work in the CI/CD space DevOps and that kind of stuff. I'd been doing that stuff before it was called DevOps. Um and um I have a long history in the build tools and software supply chain technologies. I'm an Apache Maven committer, a Trino contributor and uh involved in various other effects. But these days I've been pretty much turned into an educator. So I work as a developer relations engineer at Chain Guard where we look at the components that build that are used to build your applications and hopefully provide you safe and secure ones. So well looking forward to talk about toil because it's always interesting to see when people talk about toil. I'm like well that's just like the normal work of a software engineer. >> Awesome. Yeah to your point a lot of the teams accept some level of toil as normal. They might not even realize that it's toil. It's can be self-inflicted. It might be invisible. I find that it shows up as either constant context switching, surprise security vulnerabilities, or hey, some of our core dependencies are not available when we need them to be available, reactive incident work, and even layers of process where like you're doing something over and over even though it doesn't provide the value that it once did or maybe there's a new way, a better way to do it. Now, what I find in talking to individual engineers, they don't often see the patterns. It doesn't feel like a system and systemic problem in the moment. because they show up as one-offs, but over time to your point a few of you mentioned this, they compound. Whether that's because you're you have more traffic, more users, more people. Sometimes it can be for other reasons as well. You have more engineers contributing to the same codebase. But then all of a sudden, I do find that it's worth to start investing and thinking about what proactive looks like. And so today's conversation is going to focus on naming the real sources of toil that exist, showing how strong teams create urgency without waiting for that urgency to be created for them by major external factors. And then I'd love to get as many great examples from this panel as possible so that you can all walk away with actual things and things that you can try on your individual teams or in your organizations. And so with that in mind, let's start with what you consider toil and what do you see show up the most in your organizations? Uh maybe Leslie, we could actually start with you. What do you see as the main sources of toil in your organization? Where do you see unplanned engineering work come from? >> Uh yeah, so well the times has been around for quite a while. It's our 175th birthday this year. I know. Um, and so we, you know, I always say we've been doing technology and software longer than Google has been alive, right? Um, so, uh, we have a lot of toil in pretty much any type of manifestation. Um, but I think some of the biggest sources of toil, um, I tend to see have to do with, um, the quality of our data actually. Um, not having like strong sources of truth with respect to like service ownership data, um, service data in general. Um, and from a platform engineering perspective, that's been that's really difficult when you're talking about like adoption for example and tracking down like, okay, this issue, who do I escalate it to? And so there's just like a lot of sort of wasted time just from that exercise, which again, it's like a platform or reliability management team. Um, that's a lot of what you're doing is like nudging other people to improve their systems. So um, that's been one like that's been one source of toil that's been on my mind quite a lot lately. So that's why I'm talking about it now. Um, and then there are also other sources like classic sources of truth where you know like most companies are monitors and alerts are a little bit too noisy or maybe a lot of too noisy. Um, and then uh in general just I think like things like spinning up a new service there's traditionally been a lot of toil around that. Um, just kind of the like classic workflows of a software engineer. um we've been working the last five plus years to really platformize that um by making having there be like centralized tooling and whatnot. So um yeah, those are some of the sources that of truth of toil that uh come up for me. I'm sure I could keep going, but I'll let someone else chime in. >> I like how >> Yeah. Uh >> no, you go. >> Uh yeah, thank you. So when I think about toil, I I really don't think about uh hard work. uh I think about that unplanned repeatable effort that doesn't meaningfully improve our systems. The kind of work that keeps smart engineers busy but it doesn't really make things safer or simpler uh for the next time. So when I see toil show up uh most uh today is in actually three places where recovery paths aren't engineered, safety mechanisms that are optional or easy to bypass and really that operational signals that create noise uh to Leslie's point instead of that clarity. So a lot of that unplanned oil comes from the gap between when a system is technically backup and when it's actually safe, trusted and usable. Again those are two different things and that gap is where I think engineers lose time, teams lose confidence and innovation quietly stalls. So yeah I'm curious on your all thoughts as well. >> It was interesting to say see earlier when Lissy was mentioning that it's like often like people just not understanding what what components are used internally. I think that's a very common scenario especially in the software engineering side when we talk about like different libraries being used and different language ecosystems being used people are like what we're still using R or well there's someone using PHP in our organization or like there's some I don't know even some WordPress or whatever running somewhere in the organization in large companies and that's already like unknown and then when it comes to security issues in those individual libraries understanding what versions of what libraries are used where is often like completely unknown and that's often creates these like all hands on deck. There's some craziness going on in the ecosystem, some security attack and then people just don't know where things are even happening and that's like I think that there's those two differences of toil. has the like the completely unplanned crazy everything is like on fire let's have we have to fix this and then the regular one where it's just like let's do like the dependency updates and like make sure everything is secure and unfortunately the less you do the regular sort of toil work the bigger the fire is when it then happens and item always happens there's always something coming up right like some software end of life so I'm sure Chang has lots of experience with those kind of fun events >> yeah Actually, Chen, you one of the things you said to me that really struck me is like people assume that S sur work is all toil, but you actually draw the line between what you consider toil and what is like planned operational work. What's your kind of what's your take on what toil is? >> Yeah, it's interesting uh that you bring it to that because I as I was mentioning, I really think about the human experience of it and um when you go look up the definition of toilet, it's just work >> and it's work. So presumably the work needs to be done. Um and obviously if it it doesn't need to be done, we can not do it. Uh so that's a good conversation to have to eliminate what you think of as toil or like burdensome work. But what um what I was thinking was that uh the the planned work you could you could move it all to planned. It's not like you didn't see your software upgrades coming library upgrades coming. um you could uh foresee that as work that is necessary. And one of the things that I think um people think of toil as bad or undesirable is because it doesn't use their skills or it's not valued by the business. And so a way you can change toil is perhaps by reframing it by the people who are asking for the work to be done. Why does this work need to be done? Is it valuable for the business? If it's valuable for the business and maybe it's hard to do, well, maybe that's worth investing in. If it has a key value driver, a key value uh revenue generator or something for you, like we need to do this, but it's hard. Okay. Well, maybe if it makes us money, we should spend effort in making it good and easy for us to make money. Um, and if it's uh not valuable, why is it not valuable? Is it because there's expertise that's hidden? Um, one way you can maybe reveal that is by asking AI to do it for you. uh if you think it's so easy to do and doesn't require uh doesn't generate a lot of value and doesn't require skill, perhaps you can ask an AI to do it, but obviously monitor your AI, make sure it's doing the thing you want. And if it's not doing the thing you want, every time it does the thing you didn't want, that's an that's an expression of expertise that the person who was doing it before you gave it to the AI has. It's something that's not encoded in the system. And so I think that's a way to reveal that expertise is valuable or the work you're doing is actually valuable and maybe worth investing more in if it's so valuable or if it's not valuable, don't do it. Yeah, there's definitely a different like perception of value that's very important and I think that's also a bit of a sign of how mature an organization is, right? Like if if an organization has mature leadership that understands how important it is to be on like you know not be stuck on Java 8 but use 25 or whatever or like like use a later version of npm and like like be on the on the sort of more on the bleeding edge than on the very old stuff just because there's less security issues like that requires mature leadership and unfortunately the whole like oh we need to ship this new feature and that's much more important and write a press release about it is always seen as more important and I think it's a bit of a sign of a mature leadership in the engineering organization to understand that this stuff is very critical >> definitely and often teams won't take it seriously especially when they're early in the curve of development and maybe they're trying to there is a careful balance when you're growing a business where you're like you have to actually deliver what your customers need otherwise you can have the most stable system that you build, but actually they won't be using it. And what I found is that teams are forced to confront this when a major dependency vulnerability hits or maybe a major system outage. We've had quite a few of these over the last few years, like really significant ones that I think none of us would have expected as engineering leaders. And then all of a sudden, people outside of engineering are like, "Wait a minute, what are these dependencies you have? What should you be doing to invest in the resilience of your system, etc." Um, Manfred, maybe you have a story around this is like when a new when a specific open source supply chain attack makes the news, why do you find that teams actually slow everything down right away or that it causes that urgency? >> It's been, let's just say, very exciting in the last couple of months to look from the outside. I'm glad I wasn't involved in any of these companies that dealt with that, but I we have customers that are in like these situations and then come for us for help obviously. Um, so there was the the like I'll talk about the npm ecosystem. So the JavaScript components that come out of the npm registry. Um, I'm not saying the other ecosystems don't have these problems, but the npm ecosystem just was in the in the in the news recently, right? Like further back, you might remember the log for shell kind of stuff in the Java ecosystem. There's various ones in the Python ecosystem, but in the npm ecosystem specifically, last year we had that chalk debug attack and then the shy halloot and then the shahalude ones which um were identity theft. So basically people just sent a spam email to a maintainer. the maintainer fell for the trap, reset their two-factor authentication, their credentials were stolen, and then the open source projects uh publishing rights were used to publish malware into the npm registry. What was so interesting and why it was called the shahalude worm which is like the worm's name from the dune you know in the desert sand sandworm kind of thing is these malware that were infected they spread to other components and over time it made like a wave of attacks and um there was the the first one the shy halloot and then shaw one halude later a couple of weeks later and they spread really wide and that was the shocking thing that lots of organizations completely misjudged because things happened such as a JavaScript component that was for a Java server side application like a NodeJS application was infected. Well, the same JavaScript also were on the on the client side. So, in the browser it went further. Some of these components are used in desktop applications especially also developer applications. So, Visual Studio Code for example is a very common editor text editor for developers. It has an extension ecosystem. It's written in JavaScript. This is actually kind of a browser application. So developer workstations were infected. Secrets were stolen from the developer workstations and then were used to publish more GitHub stuff and cause a whole bunch of more more headaches, right? And then even it spread even ecosystem. So some of those JavaScript components were then found in fullbaked applications that were deployed to Maven Central. So the idea that you're just safe if you're using one ecosystem but not the other, that's gone. Like everything is so complex and I'm sure like Malik or Leslie have an idea of like what they're dealing with. Like there's no such thing as a company that just uses one stack and unfortunately the security issues also spread across ecosystems now. And that's and you don't know that. And so that's why it's often like all hands on deck. We need to find out what's going on. shut everything down, reset all credentials and stuff like you need to have processes for that because otherwise ideally even have shortlived credentials. There's a lot of things you can do. >> Chang, you have a couple of great examples of like things completely shutting down and forcing the prioritization. Do you want to share? >> Uh, sure. Uh, Manford referenced log for shell back in the day uh in the Java ecosystem. um we had uh thousands and thousands of uh service microservices and systems using Java and um in this one we needed to upgrade basically all of our libraries uh to at the same time and we didn't even have a really good way internally not just of kicking off all those builds and even if we did do the upgrades but even discovering them uh we didn't have the tooling internally for that. So that required a bunch of uh last minute scramble um to basically do what Manford describes as something routine that you should be doing all the time. It should be very easy and we should have tools for but we neglected. Um so that was one of them. And another one uh I guess is is less of a uh a last minute scramble due to security stuff but a few years ago there was a winter storm uh similar I guess in the US to what we had last week and an entire region's uh energy provider um system was unstable. It was under threat of being overloaded and at some points um they thought they may have to shut the whole thing down preemptively. We had at the time uh in-house data centers housed at several different cities but all within one region. So we thought oh geographically it would be a pretty big asteroid. Well it you know the weather is a pretty big asteroid and so we were in the process of uh reducing that risk. Uh it was a year uh a multi-year effort to get into the cloud out of our own data centers. But due to this winter storm um we accelerated that and were able and the company executives recognized this was an existential business threat to the business. The way to get money is to have your website up. No website, no money. Well, they recognized that this was a major risk even with their backup data center in a different uh city several hundred miles away. Not enough. Uh so they accelerated that. Several years was not soon enough. So they said in one year we're going to move to the cloud. and um basically dropped half a dozen other companywide annual objectives and said this is the one thing and the nickname for it was winter is coming and uh you know Game of Thrones style. It we did it in one year which was several years faster than we had planned and turned out to actually save quite a bunch of money um in the short term. >> I mean that's incredible. I hope that most of the engineering leaders that watch this don't have to go through that in order to prioritize toil. I think Malikica, you've got some really great examples where it's more around seeing the cumulative effect, which I think is really applicable to PE people starting to think about how can I start investing? What can I do? Do you want to share a little bit about that and then we let's maybe shift after that into some real practical techniques that people can use. >> Yeah, I think this um discussion around how does it really become a forcing function is an important one. Uh I I'll tell you the story from uh Twitter search infra. Uh for us the forcing function wasn't really a dramatic outage. It was a slow trust erosion. Uh we own core search infrastructure used by a lot of downstream product and ML teams and incidents weren't typically constant but recovery was very slow manual and unpredictable. Those were the signals that we were uh seeing uh time and again. And I think the real breaking point was when a data backfill after you know the index came down because of an upstream trigger. Um there was an incident in incident uh brought down the index and the data backfill basically took us 7 days. The system technically came back up within hours right but teams uh that depended on us were blocked for a full week and that's when ignoring toil was no longer an option. So because recovery time had become a bigger product problem than the incident itself. So that was a breaking point for us and what surprised me the most was how much damage showed up as invisible product drag right like teams delayed launches product experiments in downstream teams froze and people literally stopped trusting our timelines and SLAs and even how we defined our SLOs's and SLI. So in real terms, I think it cost us that senior engineering time, roadmap momentum and platform credibility. Um that's what pushed us to start measuring recovery and backfill time explicitly and not just outages or how do we triage um and to really treat these incidents as a systems design problem and not just as an ops problem to uh uh to you know Leslie's point uh that she was making earlier. Right? So I think the biggest cause wasn't downtime right it was that decision paralysis that happens across the organization and to your all point I do believe that this is so much of an organizational problem and not just a technical one >> I definitely think that one of the things that's helped me has been when I've started measuring the number of incidents and the recovery time people start to wonder why it's taking so long to get back up and that recovery time you mentioned that being able to use that as a way to really demonstrate to the business this is the thing worth investing in. A while ago, I was at a company where there was one particular system that was so archaic. It took 50 hours to get back up if it went down. And that's just not acceptable. And it was for real live like extremely critical use case. And so luckily not the entire system obviously would become available, but like that was what made us be able to invest within that one year. And so I do think measuring plays a really really big role. And so maybe shifting to then now what's worked and practiced for you. Leslie, let's start with you. What's something that you've done to maybe integrate toil reduction or prioritize specific sources of toil in your planning so that you can make sure that you're actually being proactive with it? >> Uh yeah. So one thing that's important for us is that we do distinguish um like business as usual work um from toil. Um so we always allocate some time for like business as usual work which can be which can be toil but ne not necessarily is toil. Um, in terms of being more proactive about eliminating toil, um, I think, you know, a lot of it is about juggling priorities, right? Um, and so the way I think is just like kind of making a case for the impact that it can have in terms of like the work streams that would reduce toil. Um, and connecting them to like other other times that it could have enabled that like high important um, work that uh, you know, the business actually needs. Um so kind of identifying like patterns like oh this issue keeps coming up and it's related to toil like maybe now that we've seen it um hindered our delivery timelines multiple times maybe we should consider it as part of our actual like prioritized work. Um, so I think I try to do those things sort of continuously um and point those out continuously so that way when it is planning time it's not a surprise that I'm like bringing up these um bringing up these uh work streams um to kind of eliminate uh toil. So definitely like keeping track of things over of toil over time is important. Um like showing that historical pattern is going to be super crucial. Um because otherwise it's hard for leadership to really uh like understand how much is actually impacting you. Um yes. Uh and some other things to keep in mind. Um oh I guess looking for like technical solutions that uh address might address like multiple forms of toil um is another way. Um and also connecting it to like re um incident management um uh statistics like was just mentioned. Um so for example we had a lot of toil around like our observability platforms. Um we had uh our data across multiple platforms. We still do but it's better now. Um and we showed that there was a lot of toil in it just by doing like the context switch of switching between um platforms, right? Like our logs were being collected in one vendor, our traces were being collected another. Um and just like that move that transfer between the two like cost us time. Um and that's assuming people even knew where to find each of these signals, right? Um and so we had a huge initiative to um centralize our tra to centralize our data and also to implement um slinstrument uh our apps with uh distributed tracing. Um and one way that we showed uh the importance of that work uh was by tying it back to like actual um time to recovery and things like that. Um and we could see like an actual correlation between the two um like the number um of services instrumented and the time it took to recover. Um, now in terms of what sort of spearheaded the initiative, um, I was proactive and highlighted this as like a risk for election readiness actually. Um, and but it was still getting people to really value the work was an uphill battle, right? Um, and it really did take a huge incident um, for us to really like prioritize it. Um, there was a major news story um, which one was it? um oh I think it was when when Trump uh got shot um that brought down the times like horribly and um that was and again we were preparing for the general election. So um that big issue was known to everyone and it was even escalated you know to our board of directors like everyone knew about it and that was really like the in the no pun incident um that got us to prioritize a lot of this work. Um and so ideally, you know, you want to be uh ahead of those things happening, right? Um and luckily we at least did have a plan. It was just like the priority and like became all the more uh clear. So um but yeah, ideally, you know, you're being more proactive and um avoiding those sort of scary times in general. >> Yeah, we keep going down the thread of incident response. So let's maybe go one level deeper on the incident response piece before we talk more about planning. Chang, you I think you might have some specific tips for how to reduce incident response and to proactively really think about how how you can use it to be able to invest. >> Yeah, perfect. Um yeah, thanks Leslie for a great tea up uh of incident response. Um Dominic referenced having things to measure and show the business that the the work we're doing to reduce toil is valuable. And for me, the way to do that is to set service level objectives, businessoriented ones, not about the systems, but about the user experiences. Uh, can people log in and do the things that they need to do to get their business done for whatever reason it is that they're using your services? And if you specify that in business terms, then executives will be able to say, okay, I'm willing to pay this much or invest this much effort in order to achieve this level of reliability or delivery. And as Manford referenced, we need to practice our incident response. And so the thing that uh I would recommend is for teams to come up with a system, a pattern, a process, tooling, and practice responding to incidents. If you don't have them organically, then make sure everyone gets practice in sort of spend that uh that unspent budget. If things are up, great. Um spend that budget that the executives have given you. They said it could be down for this much. We can afford this much. use that investment to have practice sessions and practice responding, have a plan for how to respond. As Leslie said, they didn't necessarily make the investment, but they had a plan. And so having a plan is sort of half the battle. The other half is being ready to execute that plan, making sure everyone's on the same page. And so I would recommend uh setting business speak budgets for how much the business is willing to invest in it and spending that budget in response time as Dominique said uh measure stuff and a thing is recovery and your recovery can be improved if you are practiced at recovering. I think I've seen businesses try and chase the fool's errand of prevent everything. Okay, if we prevent everything, we won't ship any software. And so, we know they're going to happen. Things are going to happen, but we need to humans provide that flexibility and adaptive capacity to respond in a way that writing code just doesn't. Um, so if people are ready to follow a plan to respond to a disaster, that's much more adaptable to any kind of disaster, uh, whether it's zombies attacking or asteroids hitting the earth, um, than any software you could have written it in advance. >> Yeah. So I still want to believe that teams can really invest in like proactively reducing toil and build a plan. Sorry Leslie if you want to throw throw in something and then then I can go next. >> Um yeah I was just going to say that practice part I think is super important. One of the things we did was um you know resilience engineering and res resilience testing. So um you know we uh originally started mostly with just like load testing right um and that was how that's a great way of sort of identifying those things proactively um because I mean to a certain extent the most interesting and impactful problems that you can address are ones that you don't know right um or at least you don't know until they like happen so you can simulate the scenarios in which that might happen um so that's sort of like one way I think about doing it more proactively in a way that's not like being too preventative Right. So much so that you have to um give up the high value work that you do need to deliver to make your business sustainable. >> Yeah. I also find these exercises of resiliency testing where you just like what happens if that cloud goes offline? Let's simulate that. Those are actually super fun for engineers to deal with because like you're like you there's always some sort of thing where you're like oh we just switch this off it'll fall over. Clean it to that it'll be just fine. And then you're like no it's not. that's falling over there and there and like five other surprises and it's actually like really fun detective work to chase down what's going on. And if you can do that in a pro in a proactive exercise then the the pressure and the panic of oh there financial damage and like reputation blah all that falls away. So you can like I I totally believe in the like do a hackathon day or like a reli resilience day where you just like let's see which plugs we can pull and how the thing falls over kind of thing in a test scenario. And that's I think it's totally fun for engineering teams to do. >> I've been able to connect it to business objectives just by looking at like what's the projected customer growth or the traffic that we expect. And if you model just a little bit of that, then you're like, well, if our goal is to have this many customers and therefore this much traffic, then it's easy to work backwards from there to do the load testing for the right number of users and to demonstrate we actually need to fix these issues that we've identified or we won't work by for uh by these big events. And I've done that in places where we didn't have a Black Friday or we didn't have like a major event that we expected. It was more just we know that we're slowly growing and as part of that growth there's a certain point where some of our services are going to fall over and it's been a great way to not have to rely on urgency but instead to be like here's the capacity that we actually know we need to support based on where we're going. Um yeah Malikica you want to go ahead. >> Yeah. No great points there by the panelists. Uh I think one thing that uh we learned the hard way and uh uh you know kind of put it in practice uh retrospectively was we stopped relying on heroics and just made the safe path boring and the default path and that gave us a lot of u uh value. uh it really came after an incident where a critical production change bypassed our shadow canary path and the system came back up quickly but validating and repairing those downstream implications took much longer and that was creating a hidden toil. So we created these truly blameless reviews right where we realized that the problem wasn't a human error. if one person could do it, it was basically like a blind spot in the system itself and it was that our deployment workflow made it possible to uh skip those safety checks under pressure. So we hardened the system instead of asking people to be more careful looked at our abstractions, our guard rails, our contracts and we made shadow canaries non-optional for certain classes of changes and added automated validation around it as well. So that really invested in that post deploy verification and recovery tooling. So the result wasn't just like fewer incidents right that's not the point but it was really that faster calmer recovery and much less operational noise and I think Cheng mentioned this systems are going to fail but can we build those systems that will fail well teams that can respond well and make learning really fast and you know investing in those fire drills operational reviews premort has really paid uh a lot of value so it really infors that belief that I care deeply which is most teams are back up fast but not back to safe and that gap is where I think toil really explode sometimes and worth keeping an eye out for. >> You made such an important point. So when it comes to reducing toil, you can use process and planning and all of these great things, but actually you can also automate and set guard rails to make it such that especially the entire organization, regardless of if they're a feature delivery team, a project team that doesn't have that SR or that platform desire to do that, they can do it. Leslie, you've probably had to put in place some of those guardrails in your roles. Is there any are there any good examples of things that have worked really well to kind of help force some of those teams to either, you know, upgrade their dependencies proactively or to address some of the incident response learnings? What has worked for you? >> Uh yeah, so I mean automation is definitely a huge part of the job. Um so you know basic things even just like automatic upgrades so that you don't even have to do it or think about it, right? Um are pretty like table stakes things to start off with. Um other things that you might want to have in place for uh in terms of automation is like automatic rollbacks, right? Um if an incident is uh if an issue is detected, right? You can set that up um with your observability and monitoring platform. So that way, you know, it's h it's happening in an automated way where you don't have to like be watching your dashboards um as it's like being deployed to production and whatnot. Um so things like that are, you know, pretty like table stakes things I think to have in place. Um and then there's also, you know, going to be more creative options that you can encounter too. Um maybe what I said was already considered um creative, but um you know, having it like roll out based on having um canary deployments, things like that. Um there was something else I had in mind, but I it left my mind. Um >> I'm sure it'll come back. I'm I bet that Manfred has a pretty strong opinion here. Well, I think I think one of the aspects also that are worth thinking about and that that came to mind when we're talking earlier about the monitoring system unless it was rolling out. I think um standardization is sometimes very useful like don't allow your developers to make a separate monitoring system and choose a different database and a different version of a different library and a different container with a different operating system in it. So you have to like don't allow all this like uh like diversity and and like like different choices just for the sake of it like be bold enough to standardize on like we are only using like I don't know some specific containers that are super lightweight don't have stuff in it um for that standardize on one version of whatever the spring framework or whatever and like stick to docker don't use vms and docker and some other mechanism and docker compose and whatever else and like desktops running under like you know servers running under your desk or something like standardize and then have those all feed into the same observability system like Lesie was saying right so those those help a lot and that then also allows you to better understand where any security issues come in play and then ideally you have process in place to make sure you use the the latest and greatest versions of those dependencies or you have some vendor that can provide you security backports if you're really stuck on some old one. So, but you need to understand what's going on, right? And you need to allocate time for doing that work. >> Yeah, that brings up a really good topic of uh governance, technical or operational governance. Um, which I think is super important, especially for like large organizations or enterprises. Um, where a lot there's a lot of toil in the cognitive like work of of figuring out how to do something, right? Um, when you have standards in place, um, that kind of work is already done for you, right? um like to a certain extent you know a lot of the technical choices that we make um obviously require tradeoffs right but it's almost more important to have a like faster way to deliver it than to have the right choice right um and also the right choice is very subjective right so um definitely so I think having like strong governance in place makes all these things a lot easier um because then centralized teams if they know if they know that they can rely on you know assumption in the technical systems they can build solutions for those technical systems. So having like a a minimized amount of d of drift I think is like always going to be a good thing um for those things. Um so like in observability for example um you know we have some centralized instrumentation libraries. Um there's every pretty much every language exists at the times because again we'reund and something years old. Um but um it was it's now that we're standardizing on like four central languages, we were able to just kind of prioritize those languages, right? Um and that also allowed us to show the difference and like this is how long it would have taken you to um instrument your application if you had used one of our standardized languages versus if you're using our non-standardized languages. Now you have to figure it on your own. It's like a difference in weeks of the matter. So that also kind of builds the case for addressing these sources of toy or technical debt. >> Awesome. I think I'm going to wrap us up here unless Malikica, you want to throw in one quick one last maybe good example or practice for the team and then I'm going to wrap us up just because we're up at the hour or at the 45 minutes. >> No. Awesome. Makes sense. Dominic, I was just going to quickly say that automation is worth investing uh in when it changes behavior under pressure, right? Not when it just produces more signals. Preventive tooling should remove like entire classes of failure, not just create more alerts for humans to try edge. So the best automation in my mind is what makes quiet disappearance of problems before engineers ever have to think about them. So let it target those classes of failures and not just those one-offs. Those probably don't need the um cost of putting in that automation in place. Yeah, >> what an amazing point to end on so that people can actually think about what they can start by automating that won't create more noise but actually help people. That's really really awesome. So thank you everybody for all of your insights today on today's panel. Obviously, the most damaging toil becomes visible when teams are actually willing to track it. I think a lot of you shared good examples of incident response and data being able to be use that in planning as a really really great way and you don't have to wait for failure to justify prioritization. Um, I think maybe the the thing we didn't touch on, but good for wrapup is that you do need sponsorship and you did all kind of touch upon it at these different pieces. And sometimes that sponsorship comes from an urgent piece of it. It comes from having identified proactively the risks and the work to do in order to do it. But ultimately seeing your role as an individual engineer, engineering leader as raising that risk that you can proactively get that sponsorship will be really really impactful for all of your success as you want to go take away and and draw this as as a next step for your you and your team. So thanks for all the panelists for sharing your thoughtful real world insights. We are going to head over now to the lead dev slack channel in the panel Q&A channel and you can continue the discussion or ask any questions that you have there. Thanks for joining and see you next time., Rooting Out Sneaky Sources of Toil, YouTube, 01kg84jdgy9b5xff03aywhbpm8.